Kubernetes on lo0 — Tech Blog

deepagents on a cluster of your own: the SDK is free, the server is not, and what you have to build in between

Tue, 15 Sep 2026 07:00:00 +0200

Opening of the agentic platform vertical. The previous articles dealt with the pieces separately: the inference gateway, its second door, what a fleet of agents costs and isolation. This one deals with what goes on top. Verified against deepagents 0.7.14 and the LangGraph repository, commits of 14 September 2026.

TL;DR

deepagents is not a runtime. It is a factory that assembles middleware and delegates to LangChain’s agent factory. It returns a compiled LangGraph graph and nothing else. The repository’s own documentation says so without decoration: it introduces no new runtime. That is good news for whoever deploys it, because it runs wherever Python runs.

The planning tool no longer exists. It disappeared in 0.7.0 as a breaking change, along with the state channel that backed it. Almost everything written about this harness still describes it as one of its three legs. Today you have to ask for it explicitly, and you have to ask for it in each subagent too.

With the default configuration, the agent’s files live inside the checkpoint. The state backend stores the full content, binaries in base64, in the same place the conversation is persisted. Changing that is one line and it is the first architecture decision of the deployment.

The database grows per loop step, not per turn. Four tables, one row per superstep, one per channel and version, another per task write. And of the four cleanup operations the base class declares, the Postgres implementation has only one: delete an entire thread. The other three raise not-implemented.

Pruning by hand can empty the history silently. The code itself warns that a partial delete breaks the delta chain and leaves the channels rebuilding empty without raising any error. The only safe pruning is by whole thread.

Tenant isolation is the thread identifier, and there is no owner check. Whoever can send a thread identifier reads that thread. The on-disk filesystem backend fixes its root when the agent is built, so it separates nobody. And the documentation’s example for computing the per-user namespace does not work outside the managed platform.

There is a configuration trap that leaves a short-window model uncompacted. If the model does not expose its window, the compaction threshold falls back to 170,000 tokens. Served on a vLLM with a 32,768 window, the agent blows up on context length long before it compacts. The failure is silent and is fixed by passing the profile by hand.

Observability without the platform does exist, and it goes through a package that is not obvious. LangChain’s core does not emit OpenTelemetry: zero references. The exporter lives inside the LangSmith SDK, and there is a mode that emits over OTLP only, without talking to the service. It is the road to a Langfuse of your own, and it is worth knowing where it passes.

And here is where the free part ends: the server. The SDK, the graph engine and the checkpointers are MIT. The server that publishes them over HTTP carries an Elastic 2.0 licence, asks for a commercial licence key for production, adds Redis to the stack and reports metadata to an external endpoint unless there is an isolation agreement. The development tool is explicitly for development.

What is left is a list of things you have to write. Per-thread authorisation, retention, a run queue, cancellation, clean draining and migrations. None of them is hard. All of them are work, and none of them appears in the twenty-line front-page example.

You are here: the harness arrives after the model

The sequence repeats on every project. First you set up inference, which is the visible part. Then the gateway, because you need keys and budgets. Then observability, because nobody knows what is going on. And when all of that works, somebody asks why the assistant cannot do multi-step tasks, and the word agent enters the scene.

At that point the temptation is to write the loop by hand, and for two weeks it looks like a good idea. The loop is easy. What is not easy is everything you discover afterwards: what happens when the context fills up, where the intermediate files are stored, how you resume a task that was left half done, how you ask a human for confirmation without losing state.

An agent harness is exactly that: the packaged answer to those questions. And choosing one looks like a library decision, about the size of choosing an HTTP client. It is not. It decides where your users’ state lives, which database grows, what can be isolated and what cannot, and in the case at hand, whether there is a commercial licence waiting at the end of the road.

This article looks at deepagents with those eyes. It is not a usage guide, and I do not assess whether its agents are any good. I look at what it forces you to build.

The analogy: the scaffolding and the crane

A building site needs scaffolding and it needs a crane, and they are not the same thing even though both are metal structures surrounding the building.

You put up the scaffolding yourself with parts you bought. It adapts to the façade, it can be extended, and the day the work finishes you take it down and put it away. Nobody charges you for using it once it is bought.

The crane is another matter. You hire it, it comes with an operator, it has a serial number and you have to notify somebody when it goes up. Without a crane the work still moves forward, more slowly and with more people, but it moves forward.

Here the scaffolding is the SDK and the graph engine, and it belongs to whoever buys it. The crane is the server that exposes the agent over HTTP, and that one comes with a contract. This article is about where exactly the line between the two sits, because you cannot see it in the brochure, and about how much work can be done without a crane.

What deepagents actually is

The first thing to get out of your head is that it is a system. It is a function.

The main factory assembles a list of middleware and calls LangChain’s agent factory, which is what builds the graph. What it returns is a compiled graph, with an added configuration on top that raises the recursion limit to a very high number and adds an integration tag in the metadata. The repository’s own architecture document states it: no new runtime is introduced.

Compiled with the default options, the graph has the nodes you would expect from a reasoning-and-action loop: start, a prior node that repairs incomplete tool calls, the model node, the tools node and the end. The conditional edges run from the model to tools, to the model itself or to the end. There is nothing more. Everything that looks like a system is middleware that adds nodes only when it declares hooks of its own.

The tools that ship by default are the filesystem ones, plus the subagent delegation one: list, read, write, edit, delete, search by pattern, search by content, execute, and the task tool.

The good consequence of this is portability. If something is a LangGraph graph and nothing else, it runs in any Python container, with the checkpointer you choose, against the model you choose. There is no hidden service behind it.

What is no longer there

It deserves its own section because it contradicts almost everything published. The planning tool, the one that wrote a task list and kept it in a state channel, was removed in 0.7.0 as a breaking change declared in the changelog. The tool is gone, the channel is gone, and the prompt fragment that went with it is gone.

It survives as LangChain middleware, and you have to ask for it by hand. With one detail that bites: you have to ask for it in each subagent too, because each subagent assembles its own stack. All that remains from the previous stage are archaeological leftovers, such as the state key that is still on the list of keys excluded when passing context to a subagent.

The plan, today, rests on files. Which is consistent with the rest of the design, and is another reason to look closely at where those files live.

Where state lives

This is the first architecture decision and the one that is most expensive to get wrong.

The agent’s state has two large channels. Messages, with a reducer of its own that deduplicates by identifier and treats deletions as tombstones. And files, with its own reducer where a null value means deleted. Both use a channel type that writes deltas and stores a full snapshot every fifty updates, and the code comment explains why: to take checkpoint growth from quadratic to linear with respect to the number of messages.

That channel type is marked beta in its own code, with a warning that the on-disk representation may change. Worth knowing before you rest a platform on top of it.

The backends

There is a common interface and several implementations. The ones that matter for deciding:

Backend	Where the files live	Persistence
State	Inside the graph state, that is, inside the checkpoint	Per thread, in the checkpointer’s database
Filesystem	In a real directory anchored to a root	Wherever you mount the volume
Store	In LangGraph’s store, with a namespace	Crosses threads, with a time to live
Composite	Routed by path prefix among the above	Mixed

The default backend is the state one. That is: by default, everything the agent writes ends up inside the checkpoint, and binaries go base64-encoded inside the same JSON. For a demo it is convenient, because you have nothing to set up. For a platform with real users it is a sizing decision taken by omission.

There is a detail that rounds the matter off. The filesystem middleware automatically evicts large tool results to a file when they exceed a token threshold, so that they do not occupy the context window. It is a good mechanism. But with the state backend, evicting to a file means moving the content from one place in the checkpoint to another place in the same checkpoint. It leaves the model’s window and it does not leave the database.

What grows in the database

The Postgres checkpointer creates four tables: one for migrations, one for checkpoints, one for blobs by channel and version, and one for writes by task. The primary keys all start with the thread identifier, and there is an index per table on that column.

What you have to internalise is the rhythm. One checkpoint row is written per engine superstep, not per conversation turn. A turn with four tool calls is several supersteps. None of that is recycled on its own.

And here comes the uncomfortable part. The checkpointer’s base class declares four maintenance operations: delete a thread, delete by run, copy a thread and prune with a strategy. In the Postgres implementation only the first is implemented. The other three inherit the not-implemented exception from the base class. Deleting a thread is three delete statements by identifier, and that is it.

In other words: retention is yours to build. And with a care that the code itself documents and that is worth quoting in full, because it is the kind of warning you read after the incident. A naive pruning that removes intermediate checkpoints and their writes can cut the delta chain; the surviving checkpoint is rarely a snapshot point, so its channels would rebuild empty without any error being raised.

Translated into operations: the periodic sweep deletes whole threads and only whole threads. Never loose rows. And since the checkpoints table has no date column, to know which thread is old you have to keep count outside or rely on the checkpoint identifier being a time-sortable UUID.

There is a variant of the checkpointer that stores only the last state and retains no history, which caps growth at the root. With one important reservation to verify before using it: it is not clear that it is compatible with the delta channels this harness uses by default for messages and files. Combining the two without checking is asking for a silent problem.

The store, which does have cleanup

Curiously, the sibling piece is better resolved. The store has a configurable time to live, with refresh on read, skipping of expired entries and a sweep interval, and the sweep is a delete by expiry date.

Two warnings. The sweeper is a thread inside the process, so with several replicas there are several sweepers competing for the same delete; it is idempotent, but it is free contention and you avoid it by switching it off and putting the delete in a scheduled cluster job. And the store’s semantic search requires the vector extension in the database, which constrains the managed Postgres image, which normally does not carry it.

The isolation that does not exist

If the platform is going to have more than one tenant, this is the section that decides the deployment topology.

The only thing separating one thread from another is the thread identifier. It is the first column of the primary key in all three tables. There is no owner check anywhere in the checkpointer. Whoever manages to send a thread identifier in the call configuration reads that thread. Authorisation is the responsibility of the layer you write on top, and since that layer also has to be written, it is worth noting down now.

By backend, the situation differs:

The state one inherits thread isolation, which is enough if the HTTP layer validates that the user owns the thread.
The store one computes the namespace per call, through a function that receives the runtime context. It is the correct option for multi-tenant.
The filesystem one fixes its root in the constructor, when the agent is built. There is no path in the code that recomputes it per request, per thread or per tenant. Sharing one deployment between tenants with this backend means everybody sees the same directory.

On that last point, do not confuse it with the protection that does exist. Virtual mode, which now comes enabled by default, blocks paths that escape the root. That protects against directory traversal. It does not protect against the neighbour. The project’s own threat document admits it: that mode exists to support the composite backend’s prefix routing, not as a security boundary.

There is also a documentation trap worth flagging because it costs an afternoon. The example the documentation proposes for computing the per-user namespace reads the identity from a server-information field of the runtime context. That field is annotated in the LangGraph code as metadata injected by the managed server, and null when running open-source LangGraph without managed deployments. On a cluster of your own with an HTTP layer of your own, that example fails. What you have to use is the context you fill in yourself, with the identifier that comes validated from the identity provider.

And one more boundary that gets crossed without warning: the memory and skills middleware interpolate file content into the system prompt as is, without sanitising. If two tenants share a directory that serves as the source for that, the first writes instructions that the second’s agent executes. It is prompt injection through shared storage, and it is avoided with the same measure as everything above: do not share the filesystem backend between tenants.

The topology conclusion is short. Real multi-tenant with files means one deployment per tenant, with its namespace, its volume and its quota, or else the store backend with a namespace computed per request. There is no third way.

The model, and the trap you need to know about

The default model is a specific commercial provider, and it is marked deprecated: passing an empty model warns and will stop working. You can inject any LangChain chat model, which is what anyone serving their own weights will do.

With two integration details that are not obvious.

The first. To point at an OpenAI-compatible endpoint, which is how a vLLM behind a gateway looks, you have to pass the already-built instance, not the string with the provider prefix. With the string, a provider profile is applied that forces the use of the responses API, which a vLLM normally does not implement. Passing the built object applies no profile and the agent stays clean.

The second is the trap, and in my view it is the most likely configuration failure in the whole deployment.

The compaction middleware computes its thresholds from the model profile. If the model declares its input window, it compacts on reaching 85 % of the window and keeps 10 %. If it does not declare it, it falls back to a fixed value: 170,000 tokens.

The chain that leads to not declaring it is the usual one in a deployment of your own. The OpenAI client resolves the profile by looking the model name up in a static table of the provider’s models. A model served as qwen3-30b-a3b is not in that table, so the profile comes out empty, and empty becomes null. The method that assigns it swallows any exception, so there is no warning.

Result: a vLLM with a 32,768-token window will not compact until 170,000. The server will return a context-length error long before that, and it will do so intermittently, only when conversations get long. There is a safety net that catches the error and trims, but that is reactive recovery after a failed call, not planning.

The fix is one line, because the profile is a public field and only fills itself in when it is empty: you pass it when building the model, with a value below the real window to leave room for generation. And it deserves an assert at start-up, because it is the kind of thing nobody looks at until it fails.

A related sizing note. With the default thresholds and a 32,768 window, compaction would trigger at 27,852 tokens, while eviction of large tool results is fixed at 20,000. That is, a single tool result can occupy almost the whole budget before anything evicts it. With 128,000 windows the defaults are proportionate; below 64,000 you have to lower those two thresholds by hand.

The tools, and the MCP that is not where it seems

This harness has its filesystem tools and its delegation tool, and the rest are passed as an ordinary list of LangChain tools.

The fact that matters to anyone with an MCP gateway in place: the SDK does not integrate MCP. There is no reference to the protocol in the package, nor any dependency on adapters. The integration lives in the command-line tool that accompanies the project, which does depend on the MCP adapters for LangChain, resolves configurations in the style of the desktop clients, and turns each remote tool into a LangChain tool that then goes through the ordinary list.

So the path exists and is proven, but you have to walk it yourself: load the gateway’s tools with the adapters and pass them in the list. Which, incidentally, fits well with what we already know about the catalogue: the credential-based filtering the gateway does is applied in the listing, so each agent receives the subset that belongs to it without the harness having to know anything.

The subagents

They are declared in three ways and invoked with a task tool that takes a description and a type. What you need to know to design with them:

The subagent receives the parent’s state except the messages and a few private keys, and it starts with a single human message which is the task description. Files are inherited, and it shares the same backend object, so parent and child work on the same logical filesystem. The result comes back as a tool message with the structured response or the child’s last text.

Depth is effectively one level. Verified by compiling the graph: the stack assembled for a declarative subagent does not include the subagent middleware, so the child does not have the task tool and cannot delegate in turn.

And there is no concurrency limit of its own. The parallelism is that of the engine’s parallel tool calls, and the task tool’s prompt explicitly invites launching them in parallel. Anyone with a per-tenant GPU quota will want to put the brake where it belongs, because it is not here.

Code execution

The sandbox protocol is well defined and the packaged implementations are nearly all paid services: four commercial providers, plus an embedded JavaScript interpreter and a sandbox from the platform vendor itself.

That leaves a local shell backend that is worth looking at closely before considering it. It runs the command with the system shell, and its own docstring lists what it does not do: no isolation, no process separation, no resource limits, and it lists production and multi-tenant environments as inappropriate use cases. It also has an option to inherit the process environment which, enabled, hands the model every variable, including the gateway credentials and the database connection string. It comes disabled, and it is an easy foot to shoot.

The reasonable way out on a cluster of your own is to implement the protocol yourself, and it is smaller than it looks. The sandbox base class requires four things: an identifier, execute, upload files and download files. Everything else, list, read, write, edit, delete and search, is built by the base class on top of execute. Looking at one of the commercial implementations, it is on the order of a hundred and fifty lines.

One image requirement that constrains the Dockerfile and is worth knowing beforehand: the base class helpers inject small encoded Python scripts to resolve searches and checks, so the sandbox image needs Python and a POSIX shell or half the calls fail.

On how to implement it, the shape that fits a Kubernetes of your own is an ephemeral container per session with a bounded time to live, resource limits, no service account token mounted and a network policy that only allows egress towards the inference gateway. Execute is resolved against the cluster’s exec API, and upload and download through the same channel. The alternative with less code is an isolated runtime class for the agent pod, one pod per tenant, and the shell backend inside; you lose multi-tenancy in a single pod and you have nothing to write.

Observability, and the package it goes through

Here there is a fact that surprises and that is worth being clear about before designing the trace pipeline.

LangChain’s core does not emit OpenTelemetry. Zero references in the whole package. Its native tracer talks to the vendor’s observability service and that is that.

The OTLP exporter does exist, but it lives inside that service’s SDK. And there is the good news: that SDK has a tracing mode that emits only over OTLP, without talking to the service, reading the destination and headers from OpenTelemetry’s standard environment variables. The transport is HTTP with protobuf, which is exactly what a self-hosted Langfuse accepts on its ingestion route.

That is: to take the agent’s traces to your own observability you have to install the vendor’s package and ask it not to talk to the vendor. It works, it is supported and it is documented, but it is a dependency you have to declare in the supply chain analysis, not a configuration detail. And if the OpenTelemetry packages are missing, it emits a warning and silently stops tracing, so start-up should check for it.

There is the alternative of the callback handler that Langfuse itself publishes, with fewer pieces. The practical difference is correlation: over the OTLP route the agent’s spans share a trace identifier with the gateway’s and the inference engine’s, and you see a whole request end to end. Over the callback route, you do not. For anyone who already has a collector deployed, the OTLP route is the one that pays.

The server: here is where the free part ends

Everything above is an architecture decision. This is a licence decision, and it is the one that gives the article its title.

The harness SDK is MIT. The graph engine is MIT. The memory, SQLite and Postgres checkpointers are MIT. All of that runs anywhere, with no key, no registration and without calling anybody.

The server that exposes a graph over HTTP is another matter. The package that implements it carries an Elastic 2.0 licence, whose relevant clauses forbid offering the software to third parties as a managed service and forbid modifying or circumventing the licence key functionality. The command-line tool’s own command prints the conditions: for local development it asks for a service API key, and for production use it asks for a licence key in an environment variable.

There is also usage reporting. The server code contains a beacon endpoint to which it sends metadata, with a header carrying the licence key. The self-hosting documentation confirms it from the other side, listing among the requirements network egress to that domain for licence verification and usage reporting if not running in isolated mode. Isolated mode is a contractual variant, not a box you tick.

And it drags infrastructure along: besides Postgres, it asks for Redis.

The development tool, which is the one that appears in every tutorial, describes itself as development mode with hot reload and an in-memory server. It is not a production server and it does not claim to be.

None of this is a reproach. It is a legitimate and fairly common business model: permissive core, commercial control plane. What is not legitimate is finding out late. Anyone building a platform with a sovereignty requirement, with no dependencies on external services and with the argument that the whole stack is open, has to know that the piece that turns the graph into a service does not meet that requirement.

What you have to build

The way out is to write the HTTP layer yourself, and the good news is that it is boring. A service that compiles the graph once with the checkpointer pointing at the database, and exposes an endpoint that validates the identity provider’s token, derives the thread identifier and the tenant context, and invokes the graph. Event streaming if needed.

What you additionally have to write, and which the commercial server gave you ready-made:

Per-thread authorisation. The engine does not have it. It is the most important and the easiest to forget, because nothing fails if it is missing.
Retention and deletion. A scheduled job that deletes whole threads, never loose rows, for the reasons in the state section.
Migrations. Schema setup has to be called explicitly and needs data-definition permissions, so it goes in a separate job with a different role from the runtime one. And since it creates indexes concurrently, it cannot go through a connection pooler in transaction mode; point it at the direct write service.
Health probes and draining. The engine has a cooperative draining mechanism, but wiring it to the container’s termination signal is your own work.
Run queue and cancellation, if you need long background runs. This is where the absence is most felt, and the reasonable answer is a queue on the same database rather than another system.
Durability decided deliberately. There are three modes: persist before the next step, persist in parallel, or persist only on exit. The last one is no good on Kubernetes: if the pod dies, the whole turn is lost.

On that last point it is worth being explicit, because it affects tool design. The engine stores each task’s writes as soon as it finishes, and on resume it reapplies those that were already there and does not repeat them. A tool that finished is not run again. A tool that was in flight when the pod died is, because its write never landed. The semantics are at-least-once, so every tool with an external effect has to be idempotent or carry an idempotency key. There is no exactly-once.

And the same applies to interrupts for human confirmation: on resume, the node is re-executed in full, so any side effect that sat before the interrupt inside the same node is repeated. The rule is that the interrupt goes first.

How it fits with activity logging

Two things about this setup touch compliance head on, and both come out of earlier sections.

The first is that the checkpoint is, de facto, a repository of personal data. It contains the whole conversation and, with the default backend, the files the agent has written. If the platform serves identifiable people, that database needs encryption, retention of its own and separate access control, exactly like the gateway’s spend table. The difference is that here there is no redaction switch that helps: the state is the state.

The second is traceability. An agent that delegates to subagents and executes tools produces a chain of actions that you have to be able to reconstruct, and the only place where that chain is complete is the trace. Traces going out over OTLP to your own observability stops being a technical preference and becomes the logging mechanism. It is worth designing it that way from the start, and not adding it afterwards.

As always in this series, the annex’s specific codes have to be checked against the current text before taking them into a compliance document; the approach is developed in the technical controls article.

Checklist

Decide the filesystem backend before the first demo, because the default one puts the content in the database.
Pass the model profile by hand with the real window, and check it at start-up.
Lower the eviction thresholds if the model’s window is below 64,000 tokens.
Build the model as an instance, never as a string with a provider prefix, if there is an OpenAI-compatible endpoint behind it.
Write per-thread authorisation before opening the service to more than one user.
One deployment per tenant if you use the filesystem backend; a namespace computed per request if you use the store one.
Compute that namespace from your own context, not from the server-information field, which is null outside the managed platform.
A scheduled retention job that deletes whole threads and only whole threads.
Schema migrations in a separate job, with a role of their own and against the direct write service.
Synchronous durability, and tools with external effects made idempotent.
Traces over OTLP with the mode that does not talk to the vendor’s service, and a start-up check that the packages are there.
Put a concurrency limit on subagents if there is a GPU quota involved.
If you need code execution, implement the sandbox protocol against an ephemeral container; the local shell backend is advised against by its own documentation.
Note in the supply chain analysis the hard dependencies on commercial providers that the package drags in even when unused.

Traps and things that are not what they look like

The planning tool was removed in 0.7.0. You have to add it by hand, and in each subagent too.
The default backend stores files inside the checkpoint, binaries included in base64.
Evicting a large result to a file does not get it out of the database with that backend, only out of the context window.
One checkpoint is written per superstep, not per turn.
Of the four declared cleanup operations, only delete-a-thread exists in Postgres.
A partial pruning can empty the history without raising any error.
The delta channel is in beta and its on-disk representation may change.
There is no owner check in the checkpointer. The thread identifier is not a credential.
The filesystem backend’s root is fixed when the agent is built and it separates no tenants.
The documentation’s example for the per-user namespace does not work outside the managed platform.
The filesystem middleware and the subagent middleware cannot be removed.
If the model does not declare its window, compaction does not trigger until 170,000 tokens, silently.
Passing the model as a string with a provider prefix forces an API that a vLLM does not implement.
The SDK does not integrate MCP; the integration lives in the command-line tool.
Subagents do not delegate: depth is one level.
LangChain’s core does not emit OpenTelemetry; the exporter lives in the vendor’s service SDK.
The HTTP server carries an Elastic 2.0 licence, asks for a licence key for production and reports usage unless there is an isolation agreement.
The development tool is for development, with an in-memory server.
Resumption is at-least-once for in-flight tools.

Closing

The question this article started with was whether deepagents fits into a platform of your own. The answer is that it does, and that it fits better than I expected, precisely because it is less than it appears. A middleware stack on top of a graph is easy to host, easy to understand and easy to replace. What does not fit is the piece next to it.

That asymmetry is the pattern I have been seeing all year across the ecosystem, and it deserves naming. The core is published under a permissive licence because the core is where you compete for adoption. The control plane is published under a restrictive licence because the control plane is where you charge. It happened with the gateways, it is happening with observability, and it happens here. Anyone building a sovereign platform cannot assess a whole project by the licence of its main repository: you have to look package by package, and look at which of them is the one you are going to need the day the thing has users.

The good part is that the bill, in this case, is paid in work and not in money. What you have to build to do without the commercial server is an HTTP service with no mystery to it, a retention job, a migration job and an authorisation layer. None of that is research. All of it is a week of an engineer who knows what they are doing, and in exchange you are left with a system that deploys the same way in a school as in a classified datacenter, with no internet egress and with nobody counting runs on the other side.

What I do recommend is making that list before the first demo and not after. Because the front-page example works in twenty lines, and the twenty lines have the default backend, the model without a profile and no authorisation at all.

Sources

langchain-ai/deepagents, código y documento de arquitectura: https://github.com/langchain-ai/deepagents.
langchain-ai/deepagents, registro de cambios de la rama del SDK (eliminación de la herramienta de planificación en la 0.7.0): https://github.com/langchain-ai/deepagents/blob/main/libs/deepagents/CHANGELOG.md.
langchain-ai/langgraph, motor, canales y checkpointers: https://github.com/langchain-ai/langgraph.
langgraph-checkpoint-postgres, esquema de tablas y operaciones de mantenimiento: https://github.com/langchain-ai/langgraph/tree/main/libs/checkpoint-postgres.
langgraph-checkpoint, clase base del checkpointer y aviso sobre poda con canales de deltas: https://github.com/langchain-ai/langgraph/tree/main/libs/checkpoint.
LangChain, despliegue de servidor autónomo (requisitos de licencia, Postgres, Redis y salida de red): https://docs.langchain.com/langsmith/deploy-standalone-server.
LangChain, trazado con OpenTelemetry: https://docs.langchain.com/langsmith/trace-with-opentelemetry.
Elastic License 2.0: https://www.elastic.co/licensing/elastic-license.
Langfuse, integración nativa OpenTelemetry (ruta de ingesta y autenticación): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, integración con LangChain por retrollamadas: https://langfuse.com/integrations/frameworks/langchain.
langchain-ai/helm, chart del servidor: https://github.com/langchain-ai/helm.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

Sat, 12 Sep 2026 07:30:00 +0200

This post opens the identity block of the operations track. The previous one covered the gateway side: the JWT that does not validate audience, the user who never reaches the trace. This one covers the piece on the other side. Verified against Keycloak 26.7.3, revision 2026-07-28 of the MCP specification and the text of Royal Decree 311/2022 in the BOE.

TL;DR

Keycloak 26.7.3 was released on 31 August 2026 and is still incubating at the CNCF, which it entered in April 2023. The cadence is four minor releases a year and support in practice covers a single branch: the twenty security bugs fixed in 26.7.3 were not backported to the 26.4, 26.5 or 26.6 branches at the time of release.

The Quarkus-based distribution separates build and runtime configuration. What is set in the build phase is frozen into the image, and changing it at startup forces a rebuild. In containers, the pattern is to build into the image and start with the optimised flag.

Storage is relational and only relational. The alternative engine that was in development was withdrawn in version 25.0. Since 26.0 sessions are persisted to the database by default, which changes the load profile: the dominant cost in the official benchmarks is database CPU.

Organisations solves multi-tenancy inside a realm, fully supported since 26.0. It groups members, associates email domains and links identity providers per organisation. It is the alternative to replicating realms per customer.

The measured bottleneck is password hashing. Since 25.0 the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. The official sizing guide gives one vCPU for every 15 password logins per second, against 120 per second for client credentials and for refreshes.

Keycloak does not implement two standards the MCP specification marks as mandatory. Protected resource metadata belongs to the MCP server, not the authorisation server, so there is nothing to hold against it there. Resource indicators do belong to it and are not supported. Keycloak’s own documentation declares partial conformance with the last three MCP revisions for that reason. The practical substitute is scopes plus an audience mapper.

There is a recent critical bug worth looking at today. CVE-2026-18963, scored 9.1, allowed resetting any user’s password without authenticating, administrators included. Fixed in 26.7.2, released on 19 August 2026.

You are here: the cross-cutting layer

Identity is not a stage of the pipeline, it is a layer that crosses all of them. It shows up in stack hardening as one of the seven layers of defence, in the model’s chain of trust as workload identity, in the gateway as token validation, and in the post about authenticating MCP servers as authorisation server.

What was missing was the post about the piece itself, and above all the decision of where to put it and what to expect from it.

The analogy: the civil registry and building permits

A small town hall has a population register, which says who lives there, and it has permits, which say who can do what. They are two different things and it is worth not confusing them, because the register is good for the whole town while a permit is good for one specific job on one specific street.

Keycloak is the register. It knows who each person is, identifies them reliably, knows which groups they belong to and issues a credential document that expires. What it does not do is decide whether this person can modify this particular file: that is the permit, and it lives in each service.

Confusing the two produces the most expensive design mistake in platforms of this kind, which consists of putting into the token everything that will need to be decided later. The register document grows, gets handed out everywhere, and when a permission changes you have to wait for everyone’s document to expire.

There is a third element the analogy captures well. A credential document issued for presenting at the tax office should not be good for entering the municipal depot. That is the token’s audience, and it is the central subject of the second half of this post.

Part 1. What Keycloak is today

Project status

Item	Value
Latest stable version	26.7.3, released on 31 August 2026
Previous ones	26.7.2 (19 August), 26.7.0 (9 July), 26.6.0 (8 April)
Cadence	four minor releases a year
CNCF	incubating since 10 April 2023, not graduated
Support	the current branch; earlier ones do not receive fixes reliably

That last point deserves underlining because it conditions the upgrade policy. The analysis of 26.7.3 shows that its twenty security fixes were not backported to earlier branches at release time. Operating Keycloak means keeping up to date with the current branch, not staying on a “stable” version for a year.

Since 26.0 there is a backward compatibility guarantee across minor versions for features marked as fully supported, and breaking changes are opt-in.

The internal architecture in five pieces

Two startup modes. The development one enables HTTP, does not resolve the hostname strictly and disables the theme cache. The production one requires a hostname and TLS, or it will not start. Confusing them in production is the first antipattern on the final list.

The build phase. The build command materialises the optimisations into the image. Build options, such as the database engine, the enabled features or metrics, are frozen. Runtime ones, such as the database password or the hostname, are applied at startup. Changing a build option at startup forces a rebuild with the corresponding latency cost. The correct pattern in containers is to build into the image and start with the optimised flag.

Extensions via SPI. Authenticators, protocol mappers, user storage, event listeners, hashing algorithms and secret vaults. They are packaged as JARs and resolved in the build phase.

Infinispan cache. Embedded by default, with database-based discovery as the recommended mechanism. The discovery alternatives via Kubernetes DNS, TCP and UDP are deprecated. There are local caches, with default capacities of 10,000 entries in several of them, and replicated caches for sessions, client sessions, offline sessions, authentication sessions, login failures and action tokens. Remote mode, with an external Infinispan, is needed for first-generation multi-site deployment.

Relational storage. JPA over a database. The alternative engine was removed in version 25.0. The databases supported today include PostgreSQL from 14 to 18, MariaDB and MySQL in their extended support versions, SQL Server 2019 and 2022, Oracle 19c and 23.x, and the managed variants Aurora PostgreSQL and Azure SQL.

The minimum conceptual model

Before touching anything you need to be clear on eight concepts, and the order in which they relate:

Realm: the isolation boundary. Its own users, its own signing keys, its own policies. Two realms share nothing.
Client: an application that asks for tokens. Public if it cannot keep a secret, such as a browser application; confidential if it can, such as a service; and bearer-only if it merely validates tokens without requesting them.
Client scope: a reusable bundle of scopes and mappers, assigned to clients as default or as optional. It is the unit with which you bound what a token carries.
Protocol mapper: what projects an attribute, a role or an audience into the token. The audience mapper is the one that solves the central problem of this post.
Role, realm or client, and composite role, which aggregates others.
Group, hierarchical, with inherited roles and attributes.
Identity provider: federation towards another external OIDC or SAML. User federation: LDAP, Active Directory and Kerberos, with synchronisation and attribute mapping.
Authentication flow: a sequence of executions with required, alternative, conditional or disabled requirements, and subflows. It is where you build a second factor or a check of your own.

And one feature that changes architectures: organisations, fully supported since 26.0. It solves multi-tenancy inside a single realm, grouping members, associating email domains and linking identity providers per organisation, so that provider discovery is done by the user’s email domain. The classic alternative, one realm per customer, multiplies administration and shares no configuration.

Part 2. Who can authenticate against it

This is the table you need before drawing anything. The right-hand column is usually the one that decides.

Piece	Protocol	Cost
Kubernetes API	native OIDC, or the structured authentication configuration file	Free
LiteLLM	OIDC and JWT, plus UI SSO	JWT and SSO above five users: Enterprise
vLLM	static key only	No OIDC
Langfuse	generic OIDC and Keycloak provider	SSO free; per-project access control: Enterprise
Grafana	generic OAuth yes in the open edition	Team synchronisation: Enterprise
Backstage	generic OIDC and community-maintained Keycloak provider	Free
Open WebUI	OIDC, with role management by claim	Free
LibreChat	generic OIDC, with Keycloak documented	Free
Argo CD	native OIDC or Dex, with groups into its RBAC file	Free
MinIO	OIDC with role assumption via web identity	See the note below

Two clarifications that change decisions.

Kubernetes. The modern mechanism is structured authentication configuration, which allows several JWT issuers plus validations and mappings with CEL expressions. The corresponding proposal, KEP-3331, went alpha in 1.29, beta in 1.30 and is stable since 1.34. It is the route to use today, and it replaces the loose API server flags.

vLLM has no identity. All there is is a static key via environment variable. This is not a defect to fix, it is an architectural fact: the engine sits behind the gateway and its access control is network-based, not identity-based. Any design that exposes vLLM directly to users is giving up per-person traceability.

About MinIO a note of honesty is in order: the public documentation now redirects to the commercial edition, so the detail of how it maps policies from a claim is verified there and not in the open edition. Anyone depending on that mechanism should confirm it against their own version before designing on top of it.

Part 3. Humans and machines

They are two different problems and it is worth solving them separately.

People

Authorization code with PKCE, with the S256 method. The MCP specification requires it when the client is technically capable, and it goes further: it mandates rejecting the flow if the authorisation server metadata does not publish the supported challenge methods.

Workloads with no person behind them

Here the obvious option is client credentials with a Keycloak service account, authenticated with a secret or, better, with a JWT signed by private key.

But there is a less known and better option for workloads inside the cluster: federated client authentication, introduced in January 2026 under the banner of doing away with secrets. It lets a client authenticate with a token issued by an external provider, and one of those providers is Kubernetes. The announcement placed full support for the Kubernetes service account variant and for OIDC in version 26.6; the SPIFFE variant, by contrast, expressly says it will stay in preview until the SPIFFE client authentication standard is finalised at the IETF.

That means a pod can authenticate against Keycloak with the token the kubelet already projects into it, with no additional secret to rotate. It is worth confirming the exact status and the flag name against the release notes of whichever version you have deployed, because the feature is recent and has changed status between 26.5 and 26.6. It is preferable to token exchange, whose standard version is supported but is internal-to-internal only, while the old one, which covered external-to-internal and impersonation, is deprecated and in preview.

Keycloak against SPIFFE

The question always comes up and the answer is that they solve different things with a small overlap.

SPIRE attests the node and the process, and issues short-lived identity documents with no prior secret. Keycloak does not attest workloads: it trusts a credential someone had to place there, except in the federated authentication case just mentioned, where the one attesting is Kubernetes. In exchange, Keycloak federates people, issues tokens with business data and governs consent, things SPIFFE does not do.

The practical rule: SPIFFE for service-to-service identity inside the perimeter, Keycloak for anything that has a person behind it or crosses the perimeter. The SPIFFE detail is in the close of the chain of trust series.

Part 4. MCP, and the two missing standards

This part has the most consequences and is the worst documented outside the primary sources.

What the specification requires

From revision 2026-07-28, with the normative words as they appear:

The MCP server must implement RFC 9728 protected resource metadata, and the client must use it to discover the authorisation server.
The client must implement RFC 8707 resource indicators, including the resource parameter in both the authorisation request and the token request, and must send it regardless of whether the authorisation server supports it.
The MCP server must validate that the access token was issued specifically for it as the audience, and must not accept or relay any other token.
Authorisation servers and clients should support client identifier metadata documents. RFC 7591 dynamic registration becomes optional and is declared deprecated, retained for compatibility.

The revision also adds the issuer in the authorisation response, from RFC 9207, and a stateless model with state handlers where it is expressly stated that holding a state handler is not equivalent to being authenticated.

What Keycloak supports

From its own MCP integration guide:

Standard	Keycloak
OAuth 2.1 and authorisation server metadata	Supported
RFC 9207, issuer in the response	Supported
RFC 7591, dynamic registration	Supported
Client identifier metadata document	Supported, experimental, after enabling the feature
RFC 8707, resource indicators	Not supported
RFC 9728, protected resource metadata	Out of scope, it belongs to the MCP server

The conformance declared by the project itself is “partially supported, without resource indicators” for the last three MCP revisions.

Two practical conclusions:

The protected resource metadata document is published by the MCP server or a proxy in front of it, never by Keycloak. That is correct and there is nothing to fix, but you have to remember to implement it, because the client needs it to discover where to go.
Without resource indicators, the audience is bounded with scopes. The official route is to use the scope parameter instead of resource, and to hang from that client scope an audience mapper with the MCP server URL as the included custom audience.

It is a workaround, it works, and it has to be documented in the design because anyone coming after will look for the standard parameter and will not find it.

Part 5. The confused deputy

With the above on the table, the classic mistake becomes obvious on its own. A token with a generic audience turns any compromised service into a master key: whoever obtains the token a user presented to the dashboard can present it to the gateway, and from there to the MCP server.

The MCP specification is explicit in three sentences worth quoting verbatim in any architecture document:

MCP servers must not accept tokens that were not explicitly issued for them.
If the MCP server calls upstream APIs, it must not forward the token it received from the client.
MCP proxies that use static client identifiers must obtain user consent for each dynamically registered client before forwarding to third-party authorisation servers.

Translated into the specific figure this track has been dealing with: the gateway must not forward the user’s JWT to third-party MCP servers. It has to obtain a different token with the right audience, or act as an OAuth client in its own right. By default, as already documented in the MCP gateway post, the behaviour is to use the server’s own credential, which is the correct one; there are two modes that forward the caller’s token and both have a name in the specification.

Part 6. Two hours of task against five minutes of token

The specification pushes towards short-lived tokens and mandatory refresh token rotation in public clients. A two-hour agentic task comfortably exceeds the life of an access token and may also exceed session idle time if nobody refreshes.

Three real options, with their cost:

Offline tokens. They survive session expiry, and expire through their own idle timeout or through a maximum if that limit is enabled. The risk is obvious: a near-permanent credential, hard to inventory. And the MCP specification adds that MCP servers should not include that scope either in the authentication header or in the supported scopes of their metadata.

Automatic refresh in the client. It is the right answer when there is a client that can implement it, with refresh token rotation and secure storage. For an agent with no person behind it, it is better to renew via client credentials.

Long-lived virtual keys in the gateway. It works, and it is what most people end up doing. The price has to be written down in the risk analysis: identity ends up decoupled from the provider, so deactivating someone in Keycloak does not invalidate their key, and traceability back to the real person depends on the discipline with which the keys were created. The levers that compensate are expiry, automatic rotation and a mandatory alias, covered in the virtual keys post.

A note of honesty about the default values. The numbers going around (five minutes of access token, thirty of session idle time, ten hours of maximum) do not appear published in the administration guide and I did not locate them in the code. Treat them as a console observation and confirm them in your own realm before quoting them in a document.

Part 7. Sizing and operating

On Kubernetes

The operator exposes three custom resources: the server, the realm import and the OIDC client. Probes and metrics live on management port 9000, with the health and readiness paths, and metrics in Prometheus format provided the corresponding option was enabled in the build phase. That port is not exposed publicly.

For high availability there are three shapes. A single cluster spread across zones, with no additional infrastructure. First-generation multi-cluster, with an external load balancer and an external Infinispan at each site. And second-generation multi-cluster, introduced in 26.7.0 and still in preview, which connects clusters without external Infinispan, using the synchronously replicated database as the source of truth, with less cache and more database load.

The official sizing figures

From the CPU and memory sizing concepts guide, which is still published:

1 vCPU for every 15 password logins per second, tested up to 300 per second.
1 vCPU for every 120 client credential grants per second, tested up to 2,000.
1 vCPU for every 120 token refresh requests per second, tested up to 435.
Leave a 150 % CPU headroom for spikes, startup and failover.
1,250 MB of memory per pod with realm data and 10,000 cached sessions. Keycloak allocates 70 % of the limit to the heap and around 300 MB outside it, so the limit comes from subtracting those 300 MB from the expected usage and dividing by 0.7.

The stated assumptions behind those figures include Argon2 with five iterations, sessions in the database and the default cache of 10,000 entries.

The difference between 15 and 120 is the whole story of Keycloak’s performance: the cost is in password hashing. Since 25.0, the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. Before it was PBKDF2, whose iterations were multiplied by ten in version 24. In FIPS mode PBKDF2 is still used.

From that comes an architectural consequence for an AI platform: machine workloads, which go through client credentials, cost eight times less than a human login. Sizing by number of services is cheap; sizing by morning login peaks is the expensive part.

The benchmark

From the 26.4 performance report, published on 1 October 2025: up to 2,000 logins per second and 10,000 refreshes per second with three pods, 74 vCPU in total and 8 GB per pod, over an Aurora PostgreSQL database, with almost linear vertical scaling.

Two useful findings from that report. With 20 ms of round-trip time between sites, the 99th percentile rises from 51 to 130 ms, and the project advises against deploying across different regions. And raising the cache from 10,000 to 200,000 entries brought the database CPU peak down from 77.77 % to 63.77 %, in exchange for somewhat longer garbage collection pauses.

Security

The urgent matter, if the deployment is below 26.7.2:

CVE-2026-18963, score 9.1, critical. A bypass of the email action token in the credential reset flow allowed resetting any user’s password with no prior authentication, administrators included. Fixed in 26.7.2, released on 19 August 2026. The temporary mitigation, if you cannot upgrade, is to disable password recovery in all realms. No known exploitation was recorded as of the end of August.

In 26.7.3 there are another nineteen, among them the missing hostname verification in the TLS certificate towards LDAP with a score of 8.8, two tenant restriction bypasses in token exchange with Microsoft and Google brokers, and two access control bypasses in organisations.

Official hardening comes down to four decisions:

Strict hostname, which is the default value since the second generation of that configuration. The server does not infer the URL from headers, which prevents poisoning email links and redirects.
Administration console on a different hostname, and network-restricted.
Proxy headers declared explicitly, trusting only a controlled reverse proxy.
Separate management port, not exposed.

Part 8. The mapping to the ENS

The codes and titles below are verified against the official text of Royal Decree 311/2022 published in the BOE, Annex II.

Code	Literal title	What a central IdP contributes
op.acc.1	Identificación	A unique identifier per entity, user or process that gains access
op.acc.2	Requisitos de acceso	Permissions are granted by role and group, not by loose credential
op.acc.3	Segregación de funciones y tareas	It supports it with roles, but the control is organisational
op.acc.4	Proceso de gestión de derechos de acceso	Least privilege and a specific remote access policy
op.acc.5	Mecanismo de autenticación (usuarios externos)	Flows and factors configurable per flow
op.acc.6	Mecanismo de autenticación (usuarios de la organización)	Reinforcements required already at the low level
op.exp.8	Registro de la actividad	It contributes part of the events, not all of them
op.exp.10	Protección de claves criptográficas	Management and rotation of the realm signing keys

Two nuances an auditor is going to look for.

The first, about op.acc.1: the text requires every entity that accesses the system, whether user or process, to have a unique identifier. That includes agents. A virtual key shared between several agents does not comply, and that is exactly what usually happens when JWT authentication falls outside the budget.

The second, about op.exp.8: the control requires recording the identifier of the user or entity associated with the event, the date and time, what information it is performed on, the type and the result. Keycloak contributes the authentication events. The usage events are contributed by the gateway and the MCP servers, and there the two already documented gaps reappear: adding and removing MCP servers generates no audit log and the trace user is set by the client.

The correspondence between control and product in this table is my own interpretation and must be validated by the certification body. The codes and titles are not: those are literal from the BOE.

Part 9. The configuration to leave written down

Three concrete pieces, so as not to stay at the level of concepts.

Hardened startup, as container arguments:

kc.sh start --optimized \
--hostname=https://sso.ejemplo.es \
--hostname-admin=https://sso-admin.interna.ejemplo.es \
--proxy-headers=xforwarded \
--health-enabled=true \
--metrics-enabled=true \
--http-management-port=9000 \
--cache=ispn --cache-stack=jdbc-ping \
--db=postgres

The metrics, health, cache and database options are build options: they go into the image, and passing them here forces a rebuild if they do not match.

The client scope with audience, which is the workaround forced by the lack of resource indicators. Created with the administration tool:

# one scope per protected resource
kcadm.sh create client-scopes -r plataforma \
 -s name=mcp-inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

# the mapper that sets the token audience
kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocol=openid-connect \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

The client asks for that scope and receives a token whose audience is only that MCP server. The MCP server validates the audience and rejects any other token, as the specification requires.

Realm import via GitOps, with the operator’s custom resource, so that the configuration does not live only in the console:

apiVersion: k8s.keycloak.org/v2beta1
kind: KeycloakRealmImport
metadata:
 name: plataforma
 namespace: identidad
spec:
 keycloakCRName: sso
 realm:
 realm: plataforma
 enabled: true
 # without this, the recovery email is attack surface
 resetPasswordAllowed: false
 bruteForceProtected: true
 sslRequired: all

Part 10. What Keycloak does not solve

Fine-grained authorisation over resources. The question “can this person see this particular trace” is a relation between objects, not a role. That problem is solved by Zanzibar-style engines, with OpenFGA and its object, relation and user tuples, or a policy language such as Cedar. Keycloak gives roles and scopes, and with those you do not model a graph of relations.

Machine secrets. Storage credentials, provider keys, database passwords. That is a secrets manager, covered in stack hardening.

Cryptographic workload identity with attestation. That is SPIFFE and SPIRE. In Keycloak it exists as an identity provider in preview, which is a different thing.

Authorisation inside the model. Which MCP tool an agent may invoke, and with what arguments. The specification offers scopes, the insufficient scope error and step-up elevation, but the per-tool and per-argument decision is taken by the MCP server or a policy engine, not by the identity provider.

Antipatterns

Forwarding the user’s token from the gateway to third-party MCP servers. Explicitly forbidden by the specification.
Static client identifier with dynamic registration and no per-client consent, which is the exact recipe for the confused deputy.
General scopes, of the full-access kind, and publishing the entire catalogue of supported scopes in the metadata.
Treating a state handler as authentication, which the new revision expressly forbids.
Following OAuth discovery towards any URL, which opens the door to requests towards internal addresses. It also applies to the authorisation server when it downloads a client metadata document.
Trust headers badly exposed in chat front ends. The Open WebUI documentation warns that an incorrect configuration allows authenticating as any user.
Starting in development mode in production, with the hostname not resolved strictly.
A single shared static key as the only control between the gateway and the engine.

Startup checklist

Check the version. Below 26.7.2 there is a critical password reset bug.
Build the image with the build phase done and start with the optimised flag.
Strict hostname, administration console on another hostname, separate management port and not exposed.
Decide between organisations and one realm per tenant before creating the second tenant.
One client scope per resource, with an audience mapper, because resource indicators are not supported.
In the gateway, define the two audience and issuer variables. Without them none of that is validated.
For workloads inside the cluster, look at the Kubernetes identity provider before handing out client secrets.
Size by login peaks, not by number of services: the difference is eightfold.
Export the Keycloak events to the same place the gateway’s go, because the activity logging control needs them together.
Write into the design what the system does with a two-hour task, before someone solves the problem with an offline token.

Closing

Keycloak is a mature piece that solves one bounded problem well: knowing who the caller is, and saying so verifiably in a document that expires. In a sovereign inference platform that is worth a lot, because the alternative is an inventory of static keys spread across ten services that no audit can walk through.

What is not advisable is asking more of it. Fine-grained authorisation lives in each service, workload identity with attestation lives in SPIFFE, and the decision of which tool an agent may call lives in the MCP server.

And there is one concrete gap worth noting down, because it does not close on its own: the MCP specification marks two standards as mandatory, and of the two that would fall to an authorisation server, resource indicators are not implemented. The workaround with scopes and an audience mapper works, but it is a workaround, and whoever comes after will look for the standard parameter. Leaving it written in the design saves an afternoon.

Sources

Keycloak 26.7.3, 31 de agosto de 2026, y descargas.
Configuración del servidor, caché, bases de datos y nombre de host.
Integración con Model Context Protocol, intercambio de tokens y autenticación federada de clientes, 27 de enero de 2026.
Dimensionado de CPU y memoria e introducción a alta disponibilidad.
Bancos de pruebas de rendimiento de Keycloak 26.4, 1 de octubre de 2025.
Especificación de MCP, revisión 2026-07-28, autorización y buenas prácticas de seguridad.
KEP-3331, configuración estructurada de autenticación.
Real Decreto 311/2022, Anexo II.
Registro de CVE-2026-18963 en la base de datos de Red Hat y notas de la versión 26.7.3 en GitHub.
CNCF, proyecto Keycloak.

The gateway does not live alone: the JWT that does not validate the audience, the user who never reaches the trace, and the seams you have to write by hand

Sat, 12 Sep 2026 06:30:00 +0200

Eighth article in the operational track of the control layer. The previous seven deal with the gateway from the inside: the pair with Langfuse, day 2, virtual keys, humans and agents, prefix routing, the MCP gateway and sizing for agents. This one deals with what surrounds it. Verified against LiteLLM 1.102.0 and Langfuse 4.35.0, both with a commit of 11 September 2026.

TL;DR

The gateway has five neighbours: a database, a cache, an identity provider, a trace backend and a collector. Each neighbourhood has its own physics, and these are the seven things it decides.

The proxy starts without Postgres, but then only the master key works. The DATABASE_URL variable is optional (proxy_server.py:1163); without it, any key other than the master one returns a 400 for a missing connection (user_api_key_auth.py:1886). With a database configured but down, the default behaviour is to return 503, unless general_settings.allow_requests_on_db_unavailable is switched on.

Redis is never mandatory, and that is precisely the trap. There is a circuit breaker with a threshold of five failures and a recovery of sixty seconds (constants.py:456). When it opens, the proxy fails no request: it degrades to local memory. What is lost silently is that rate limits stop being global and become per pod.

No observability backend can bring a request down. All callbacks are launched with asyncio.create_task and their exceptions are caught and logged (litellm_logging.py:3235). If Langfuse does not respond, events are discarded as the queue fills up. There is no added latency, and there is no reliable record either: the pair’s observability is a statistical figure, not an audit record.

JWT authentication validates neither the audience nor the issuer unless two variables are defined. In _build_decode_kwargs (handle_jwt.py:1003) the audience comes from JWT_AUDIENCE and the issuer from JWT_ISSUER; if they are empty, the code sets verify_aud=False and verify_iss=False and emits a warning once only. Any valid token signed by that provider, issued for another service, is accepted.

Role-based access control ships disabled. enforce_rbac has a default value of False (_types.py:4848 onwards). With that value, a token whose role resolves to nothing does not produce a 403: it carries on through the flow. Real authorisation, then, comes from team membership, if there is a team claim configured at all.

The user identifier that reaches Langfuse is not the one from Keycloak. The user_id field of the trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604), that is, with the user field of the OpenAI body, which the client sets. The subject of the token and the owner of the key do not travel. Anyone who wants per-person traceability needs to configure end_user_id_jwt_field or make the client send that field.

Langfuse provisions nothing from a claim. There is no variable that maps an OIDC claim to an organisation or to a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID and LANGFUSE_DEFAULT_PROJECT_ID, the same for everyone. On top of that, per-project access control sits behind the Enterprise licence in the self-hosted edition, as do audit logs and per-project retention.

You are here: between the four pieces

The LiteLLM and Langfuse pair walked through the seam between the gateway and the trace backend: integration routes, cost that arrives as zero, correlation and queues with discarding. This article widens the focus to the other three neighbourhoods, with identity at the centre, because that is the one that orders everything else and the one an auditor asks about.

The identity piece itself, that is, Keycloak, is dealt with in the next article. Here we deal with the gateway side.

The analogy: the building site with five trades

On a small building site, the problem is rarely inside each trade’s work, it is at the joins: where the plumber leaves the pipe sleeve and the bricklayer closes up the partition wall, where the electrician wants to run through the same gap. Each trade does its part well and the join ends up badly resolved because it belongs to nobody.

An inference platform has five trades. The gateway knows about tokens, the identity provider knows about people, the trace backend knows about events, the database knows about spend and the cache knows about counters. Each one has good documentation. The joins have nobody’s documentation, and that is where the user’s identity gets lost between the token and the trace.

There is one more detail of the analogy that is useful. On a building site, the part that causes the most problems is not the one that looks badly finished, it is the one that gets covered up. The same happens here: the five seams in this article fail silently.

Seam 1. What brings things down and what degrades them

It pays to have written down what happens when each neighbour falls over, because intuition fails in almost every case.

Neighbour	Does it start without it?	If it falls over
Postgres	Yes, master key only	503 by default; degraded with `allow_requests_on_db_unavailable`
Redis	Yes	Circuit breaker after 5 failures; limits become per pod
Langfuse	Yes	Events discarded as the queue fills; no impact on latency
OTel collector	Yes	Same as the previous one
Identity provider	Yes	Access to the interface and the JWT flow goes down; virtual keys carry on

Postgres

Without DATABASE_URL, _setup_prisma_client returns None without error (proxy_server.py:10375). The proxy serves /chat/completions with the master key. Any other key gets a 400. There is no global budget either, and start-up itself warns about it: Redis does not replace the database (proxy_server.py:9240).

With a database configured and down, the exception handler re-raises unless the degradation flag is active (db/exception_handler.py:56). There is also a DISABLE_PRISMA_HEALTH_CHECK_ON_STARTUP and a watcher that reconnects. The detail that matters for diagnosis: a transient database failure returns 503, not 401. If during an incident you see 401s, the problem is credentials; if you see 503s, it is the database or admission.

Redis

The circuit breaker is enabled by default, with a threshold of five failures, a recovery of sixty seconds and a minimum duration of five (constants.py:456, logic in caching/redis_cache.py:153). The socket has a timeout of 0.1 seconds (constants.py:453). With the circuit open, DualCache serves from memory without propagating an exception (dual_cache.py:215).

The consequence has to be measured, not assumed: with four workers and three replicas, a limit of one hundred requests per minute becomes one thousand two hundred. It is a failure mode that generates no errors and that can last for days without anyone noticing, until the bill arrives or the engine saturates.

The right alert is not about Redis, it is about litellm_service_latency labelled by service (integrations/prometheus_services.py:119), which exposes Redis and database latency separately.

The callbacks

Everything behind, that is, Langfuse and the collector, is launched outside the response path and its failures are caught. The classic integration uses the v2 SDK with its own background thread and a flush interval of one second, adjustable with LANGFUSE_FLUSH_INTERVAL (langfuse.py:198). There is a cap of fifty instantiated Langfuse clients (constants.py:554), because each client is a thread.

The OTLP integration uses the batch processor of the OpenTelemetry SDK (opentelemetry.py:3060) with its default values. In both cases, saturation means discarding, and discarding generates no error towards the client.

Seam 2. Identity: the three doors and what each one validates

Everything goes through a single authentication constructor (user_api_key_auth.py:1245), in this order:

Custom authentication from the enterprise edition.
Public routes.
Opaque OAuth2, if enabled, with a licence check.
JWT, if general_settings.enable_jwt_auth is active and the token has three parts.
Master key, with constant-time comparison.
Virtual key, with SHA-256 hash and lookup in the verification table.

A JWT works for /chat/completions, not just for administration: the routes allowed by default for a team include openai_routes (_types.py:4886), which contains the chat endpoint. Administration, on the other hand, is only reachable by a token with an administrator role.

The first thing to know: it is paid for

The JWT branch carries an explicit licence check, with the literal message that JWT authentication is an enterprise-only feature (user_api_key_auth.py:1416). SSO for the administration interface is free up to five billable users and requires a licence above that (ui_sso.py:976).

This conditions the architecture of any sovereign deployment that does not want to pay for a licence: the Keycloak integration is left for the administration interface and for provisioning, while inference traffic authenticates with virtual keys. Virtual keys are not worse, but they are a different thing, and it has to be said in the risk analysis: they are credentials decoupled from the identity provider, so removing a person from Keycloak does not invalidate their key.

The `litellm_jwtauth` fields that decide security

From the LiteLLM_JWTAuth class (_types.py:4848 onwards), these are the ones that matter, with their default values:

Field	Default	What it does
`enforce_rbac`	`False`	If false, an unresolved role does not produce a 403
`enforce_scope_based_access`	`False`	Without this, `scope_mappings` is not applied
`enforce_team_based_model_access`	`False`	Model access by team
`team_id_jwt_field`	`None`	Claim the team comes from
`team_id_upsert`	`False`	If false, a team that does not exist gives a 404
`user_id_upsert`	`False`	Same with the user
`public_key_ttl`	`600`	Cache of the public keys
`public_key_stale_ttl`	`3600`	Extra margin if the provider is down
`admin_jwt_scope`	`litellm_proxy_admin`	Scope that grants administration
`user_allowed_email_domain`	`None`	Restriction by email domain

The accepted algorithms are RS256/384/512, PS256/384/512, ES256/384/512 and EdDSA (handle_jwt.py:163). There is no symmetric one, which eliminates the algorithm confusion class of attack. It is a good design decision and it deserves recognition.

Key download supports several comma-separated URLs in JWT_PUBLIC_KEY_URL, resolves discovery if the URL points to the well-known OpenID configuration, retries three times with increasing waits and memoises the failure for thirty seconds (handle_jwt.py:680, :751, :88). If the provider is down, it serves the stale copy up to the sum of the two TTLs, that is, up to seventy minutes. For availability that is fine; for revocation it is not, and it has to be written down.

One detail of key rotation: with more than one key in the set, an exact match of the key identifier is required, but with a single key and no identifier it is accepted without checking (handle_jwt.py:913).

The validation that does not happen

This is the part to fix on day one. In _build_decode_kwargs (handle_jwt.py:1003), the audience is taken from the JWT_AUDIENCE environment variable and the issuer from JWT_ISSUER. If they are not defined, the code sets verify_aud=False and verify_iss=False, and emits a warning once only.

What that means in a real deployment with Keycloak: a token issued for the Grafana client, or for the Backstage one, signed by the same realm, is accepted by the gateway as a valid credential. Expiry is always validated, with zero margin. So is the signature. The audience is not.

There is a second route, issuers, with per-issuer configuration, which does validate issuer and audience unless explicitly disabled (handle_jwt.py:1186). That is the one to use when there is more than one provider.

The minimum fix is two lines of environment:

JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma

And on the Keycloak side, an audience mapper in a dedicated client scope that injects that audience. The detail is in the Keycloak article, because the standard that would resolve this cleanly, resource indicators, is not supported.

From claim to permission

The administrator role comes from admin_jwt_scope appearing in the scope claim, and it returns a result without going through a team (handle_jwt.py:309, :1373). The rest is resolved by team.

If the team in the claim does not exist in the database, with team_id_upsert false a 404 is returned with the message that the team has to be created (auth_checks.py:3035). With the flag active, it is created by invoking team creation with a synthetic administrator identity (auth_checks.py:2874). That is the only automatic provisioning route that exists from Keycloak towards the gateway, and it works well. It pays to know that it creates teams with no budget and no limits, so a default value is needed in the configuration.

The vulnerability worth knowing about

CVE-2026-35030, scoring 9.1 on CVSS 3.1 and 9.4 on CVSS 4.0, advisory GHSA-jjhc-v7c2-5hh6. With JWT authentication active, the cache of the OIDC user info endpoint used the first twenty characters of the token as the key, so that two tokens with the same prefix were confused with each other. Fixed in 1.83.0.

In 1.102.0 the code uses the full SHA-256 hash (handle_jwt.py:962). A search for similar patterns across the tree finds no authentication cache indexed by token prefix: the truncations that remain are masked for logging. And it is worth recalling what has already been published about the MCP gateway: below 1.83.14 there is unauthenticated remote execution or SQL injection.

Seam 3. Identity does not reach the trace

Here lies the most expensive design flaw in the arrangement, because it is discovered at the first audit.

The user_id field of a Langfuse trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604 and :724), which comes from the user field of the OpenAI request body or from end_user_id_jwt_field (litellm_pre_call_utils.py:1584). It is not the subject of the token nor the owner of the virtual key.

The knock-on consequences:

If the client does not send user, the trace has no user.
If the client does send user, the trace has whatever value the client cares to put there, unverified.
An agent calling with a service key leaves traces with no person behind them.

There are three ways to fix it, in order of robustness:

Configure end_user_id_jwt_field so that the end user comes from a token claim and not from the body. It is the only option in which the value is not controlled by the client, and it requires JWT authentication, that is, a licence.
Force the trace_user_id metadata field, which overwrites the field (langfuse.py:726; on the OTLP route, langfuse_otel.py:96). It can be set per team.
Contractually require the client to send user and validate it in a guardrail. It works, but it is a client declaration, not a proof.

Something similar happens towards the engine. The caller’s identity does not travel to vLLM unless litellm.add_user_information_to_llm_headers is switched on, whose default value is None (litellm/__init__.py:223). With it, x-litellm-* headers are injected with user identifier, team identifier and key hash (litellm_pre_call_utils.py:1356). Forwarding client headers requires general_settings.forward_client_headers_to_llm_api, and the authorization header is always filtered out, which is correct.

Towards MCP servers, by default the server’s own credential is used, not the user’s token. Only two modes forward the caller’s token, and doing so has a name in the MCP specification: token passthrough, which is explicitly forbidden. It comes up again in the Keycloak article.

Seam 4. Langfuse as a neighbour

What can be automated and what cannot

The exact list of variables for SSO in Langfuse 4.x, taken from web/src/env.mjs:

With the Keycloak provider: AUTH_KEYCLOAK_CLIENT_ID, AUTH_KEYCLOAK_CLIENT_SECRET, AUTH_KEYCLOAK_ISSUER, AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING, AUTH_KEYCLOAK_CLIENT_AUTH_METHOD, AUTH_KEYCLOAK_CHECKS, AUTH_KEYCLOAK_ID_TOKEN_SIGNED_RESPONSE_ALG, AUTH_KEYCLOAK_SCOPE, AUTH_KEYCLOAK_ID_TOKEN, AUTH_KEYCLOAK_NAME.

With generic OIDC: the same set with the AUTH_CUSTOM_ prefix, plus AUTH_CUSTOM_FETCH_USERINFO and claim mapping with LANGFUSE_CUSTOM_SSO_SUB_CLAIM, LANGFUSE_CUSTOM_SSO_EMAIL_CLAIM, LANGFUSE_CUSTOM_SSO_NAME_CLAIM and LANGFUSE_CUSTOM_SSO_IMAGE_CLAIM.

Global ones: AUTH_DISABLE_USERNAME_PASSWORD, AUTH_DISABLE_SIGNUP, AUTH_DOMAINS_WITH_SSO_ENFORCEMENT, AUTH_SESSION_MAX_AGE.

And now what is missing. There is no variable that maps a claim to an organisation or a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID, LANGFUSE_DEFAULT_ORG_ROLE with a default value of VIEWER, LANGFUSE_DEFAULT_PROJECT_ID and LANGFUSE_DEFAULT_PROJECT_ROLE (features/auth/lib/createProjectMembershipsOnSignup.ts:61). The documentation itself confirms it: enterprise SSO does not provision roles automatically at sign-up.

On top of that, in the per-edition permissions table (features/entitlements/constants/entitlements.ts:139), the open and self-hosted professional editions do not include per-project access control or the administration API. They belong to the enterprise edition, as do audit logs and per-project retention. In the open edition, every user inherits the role of their organisation.

The declarative bootstrap, which does exist

For on-premise without a licence, the workable route is bootstrapping through variables (web/src/initialize.ts), which is idempotent: LANGFUSE_INIT_ORG_ID (mandatory, if missing the rest is ignored with a warning), LANGFUSE_INIT_ORG_NAME, LANGFUSE_INIT_PROJECT_ID, LANGFUSE_INIT_PROJECT_NAME, LANGFUSE_INIT_PROJECT_RETENTION, LANGFUSE_INIT_PROJECT_PUBLIC_KEY, LANGFUSE_INIT_PROJECT_SECRET_KEY, LANGFUSE_INIT_USER_EMAIL, LANGFUSE_INIT_USER_NAME, LANGFUSE_INIT_USER_PASSWORD.

With that, one project per tenant is created from GitOps with one initialisation container per tenant and the keys in Kubernetes secrets.

Langfuse credentials per team

This part is well resolved on the gateway side and few people use it. TeamCallbackMetadata.callback_vars (_types.py:2152) accepts langfuse_public_key, langfuse_secret_key, langfuse_host and langfuse_environment, with an allow list in initialize_dynamic_callback_params.py:67. The gateway caches one logger per credential set (langfuse_handler.py:24) and, on the OTLP route, builds a full exporter per key (langfuse_otel.py:400).

That is to say: one Langfuse project per LiteLLM team is possible today, without a licence, and it is the correct way to isolate traces between tenants. What does not exist is automatic project creation, nor the correspondence between the team and the project. That is glue you have to write.

The ingestion route

Langfuse accepts only OTLP over HTTP, in http/protobuf or http/json. gRPC is not supported. The base route is /api/public/otel and the signal lives at /api/public/otel/v1/traces, verified in the repository. Authentication is Basic with the project’s key pair, plus the x-langfuse-ingestion-version: 4 header for the version 4 route.

The gateway builds it exactly like that (langfuse_otel.py:328, :347, :359) and then normalises the endpoint by adding the signal suffix (opentelemetry.py:3237). The result matches the real route, so the integration works without touching anything. The detail matters when a collector is put in the middle and somebody copies the endpoint by hand.

And the calendar warning that already appeared in the operational pair still stands: the classic callback is tied to the v2 SDK and the date marked is 16 November 2026. With a nuance worth reading before scheduling an emergency migration, which is detailed in the data model article: that date applies to Langfuse Cloud, and in the code of the self-hosted version the old route is not switched off, it changes behaviour according to the write mode.

When to put a collector in

Direct if Langfuse is the only trace destination. Via a collector if there is more than one consumer, or if any of these three things is needed that the proxy’s batch processor does not provide:

Retries with increasing waits and an on-disk queue, with the sending_queue and file_storage processors. With that, a Langfuse maintenance window stops losing traces.
Tail sampling, which lets you keep 100 % of errors and 1 % of the rest. The proxy cannot sample by outcome, because at the point it decides it does not yet know how things ended.
Attribute redaction before they leave the namespace. It is the only layer where you can delete the MCP tool arguments which, as has already been documented, are written in the clear, bypassing the message redaction switch.

Seam 5. Postgres, Redis and the memory policy

Separate Postgres, always. Each piece applies its own Prisma migrations over the public schema of the database it is given. Two Prisma clients migrating the same schema collide over table names and over migration history. On top of that, Langfuse 4.x requires Postgres 15 as a minimum, recommends 16 and wants UTC by default. The load profile is also the opposite: the gateway writes spend records at high frequency, while Langfuse does light transactional work because trace data goes to ClickHouse.

Separate Redis, and the argument is not the one it looks like. There is no key collision by design: the gateway uses hash tags of the form {api_key:...}:requests and an optional namespace, and Langfuse uses the BullMQ prefix plus its own prefix for its key cache. The decisive argument is another one: Langfuse requires maxmemory-policy=noeviction because its queues are data, not cache, while the gateway assumes a disposable cache. Sharing an instance means that a spike of rate limit keys can fill the memory and cause write errors in the ingestion queues instead of a harmless eviction.

If it is shared anyway, two minimum measures: REDIS_KEY_PREFIX in Langfuse, namespace in the gateway, and different logical databases.

And a reminder about surface area: Langfuse 4.x does not start without ClickHouse or without an S3-compatible bucket. CLICKHOUSE_URL and LANGFUSE_S3_EVENT_UPLOAD_BUCKET are mandatory and have no default value (packages/shared/src/env.ts:120 and :271). Its failure surface is far larger than the gateway’s, and that changes where to put the high availability effort.

Ports, for the NetworkPolicy

Piece	Port
LiteLLM (API, interface and metrics)	4000
Langfuse web	3000
Langfuse worker	3030
ClickHouse	8123 HTTP, 9000 native
S3-compatible storage	9000
Redis	6379
Postgres	5432
OTel collector	4317 gRPC, 4318 HTTP

The pairs to allow under a default-deny policy, following the method from hardening the stack:

litellm → postgres:5432, redis:6379
litellm → collector:4318 (or langfuse-web:3000 if going direct)
litellm → vllm:8000
litellm → keycloak:8443 (interface and JWT flow only)
collector → langfuse-web:3000
langfuse-web → postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-worker→ postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-web → keycloak:8443
ingress → litellm:4000, langfuse-web:3000
prometheus → litellm:4000

The Langfuse worker needs no inbound traffic beyond the health probe.

The glue you have to write

Put into a table, the real state of the tenant chain:

Segment	Automatic?	How
Keycloak → LiteLLM team	Yes	`team_ids_jwt_field` plus `team_id_upsert: true`
Keycloak → LiteLLM user	Yes	`user_id_jwt_field` plus `user_id_upsert: true`
Keycloak → Langfuse organisation	No	Static assignment at sign-up only
Keycloak → Langfuse project	No	Same
LiteLLM team → Langfuse project	No	`callback_vars` by hand, or an operator of your own
Keycloak user → trace `user_id`	No	`end_user_id_jwt_field`, or a contract with the client

The minimum viable architecture without a licence, then, is this:

One Langfuse project per tenant, created with the initialisation variables from GitOps.
That project’s keys in a Kubernetes secret.
A periodic job of your own that reads the teams through the gateway’s administration API and writes callback_vars with the corresponding keys.
People entering Langfuse through Keycloak, but assigned by hand to their organisation.
The end user identifier resolved by contract with the clients, and validated.

That is about one hundred and fifty lines of your own code. It pays to budget for them from the start, because the gap does not close by itself.

Reference configuration

general_settings:
 # only with a licence; without one, virtual keys
 enable_jwt_auth: true
 litellm_jwtauth:
 team_ids_jwt_field: "groups"
 team_id_upsert: true
 user_id_jwt_field: "sub"
 user_id_upsert: true
 end_user_id_jwt_field: "preferred_username"
 user_allowed_email_domain: "ejemplo.es"
 admin_jwt_scope: "litellm_proxy_admin"
 # the three that ship switched off
 enforce_rbac: true
 enforce_scope_based_access: true
 enforce_team_based_model_access: true
 allow_requests_on_db_unavailable: false
 proxy_batch_write_at: 60

litellm_settings:
 callbacks: ["langfuse_otel", "prometheus"]
 # without this, the engine does not know who is calling
 add_user_information_to_llm_headers: true
 turn_off_message_logging: true
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

# the two that enable the validation that by default does not happen
JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma
JWT_PUBLIC_KEY_URL=https://sso.ejemplo.es/realms/plataforma/.well-known/openid-configuration

LANGFUSE_HOST=http://langfuse-web.observabilidad.svc:3000
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...

And on the Langfuse side:

AUTH_KEYCLOAK_CLIENT_ID=langfuse
AUTH_KEYCLOAK_ISSUER=https://sso.ejemplo.es/realms/plataforma
AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING=true
AUTH_DISABLE_USERNAME_PASSWORD=true
AUTH_DISABLE_SIGNUP=true
LANGFUSE_DEFAULT_ORG_ID=plataforma
LANGFUSE_DEFAULT_ORG_ROLE=VIEWER
ENCRYPTION_KEY=... # 64 hexadecimal characters

Checklist

Define JWT_AUDIENCE and JWT_ISSUER on day one, or use the per-issuer configuration. Without that, neither audience nor issuer is validated.
Set enforce_rbac, enforce_scope_based_access and enforce_team_based_model_access to true, and test that a token from another client of the same realm is rejected.
Decide where the end user of the traces comes from and write it into the design. If it is not decided, traces come out with no person.
Separate Postgres. Separate Redis. If Redis is shared, different prefix and logical database, and review the memory policy.
Check that the proxy starts with Postgres down only if that is what you want, and that the on-call team can tell an admission 503 from a credential 401.
Alert on litellm_service_latency per service, not on Redis availability, because the circuit breaker hides the problem.
One Langfuse project per tenant with the initialisation variables, and the per-team keys in callback_vars.
Put a collector in the middle if you need an on-disk queue, tail sampling or attribute redaction.
Check that the gateway version is above 1.83.14 because of the earlier critical vulnerabilities.
Write into the risk analysis that virtual keys are credentials decoupled from the identity provider, and define the offboarding procedure.

Traps and things that are not what they seem

A properly signed JWT is not a JWT addressed to this service. Without the two environment variables, the audience is not checked.

enforce_rbac set to false does not mean there is no authorisation, it means that authorisation comes only from team membership, if there is a team claim. With no team claim and without the flag, a valid token reaches /chat/completions.

The stale copy of the public keys lasts up to seventy minutes, adding the two TTLs. Revoking a key at the provider does not take effect immediately.

With a single key in the JWKS and no key identifier, the identifier is not checked.

The trace user is set by the client. Any per-person spend report built on that field is a declaration, not a measurement.

Redis going down generates no errors, it generates per-pod limits. It is the most expensive of the failures that give no warning.

Langfuse does not start without ClickHouse or without an S3 bucket. Its failure surface is bigger than the gateway’s, and that changes how the high availability effort is shared out.

Langfuse’s per-project access control is paid for in self-hosted, as are audit logs. For an ENS file, that gets decided before building, not after.

Sharing Redis does not break through key collision, it breaks through memory policy. It is a failure that shows up under load and gets diagnosed badly.

Langfuse’s ingestion route is OTLP over HTTP only. A collector configured with the gRPC exporter points at somewhere that does not exist.

Closing

The five seams share a common pattern: they fail without warning. The token from another service gets in, the trace user comes from the client, the rate limit stops being global, the spend queue discards on reaching 64 MB and the Langfuse project does not create itself. None of those five things produces a visible error.

Day 2 work, then, consists above all of turning silences into signals. Two environment variables so that the audience is validated, three flags so that authorisation is real, an alert on service latency instead of on Redis availability, and an explicit contract on where the end user’s identity comes from.

And one decision worth taking early, before the budget is closed. The gateway’s JWT authentication and Langfuse’s per-project access control are both behind a licence. A sovereign platform can live perfectly well without either, with virtual keys and one project per tenant, but it is a different architecture and it has to be drawn as such from the start.

Sources

Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Autenticación y SSO en Langfuse, consultado el 12 de septiembre de 2026.
SCIM y API de organización, consultado el 12 de septiembre de 2026.
Integración OpenTelemetry de Langfuse, consultado el 12 de septiembre de 2026.
Postgres y caché en la documentación de self-hosting, consultados el 12 de septiembre de 2026.
Autenticación por token en LiteLLM y ajustes de configuración, consultados el 12 de septiembre de 2026.
GHSA-jjhc-v7c2-5hh6 y registro de CVE-2026-35030 en NVD.

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

Sat, 12 Sep 2026 06:00:00 +0200

Seventh article in the operational track of the control layer. The fourth dealt with the coexistence of human traffic and agentic traffic on a fixed fleet. This one deals with the prior question: how big the fleet has to be, and how big the gateway in front of it. Verified against LiteLLM 1.102.0 (commit of 11 September 2026) and vLLM 0.29.0, with load data published between May and September 2026.

TL;DR

Until this year, sizing for agents meant extrapolating a chat profile and adding margin. In 2026 the first production-scale characterisations were published and the profile turned out to be something else. These are the seven conclusions.

The median prompt of an agentic call is 68,000 tokens, of which 63,000 arrive cached. The measurement covers 13.5 million sessions, 3.2 million users, 760.5 million model calls and 44.9 trillion prompt tokens in one week of June 2026. The median input-to-output ratio exceeds 275 to 1. Any calculation done on a profile of 800 prompt tokens and 250 output tokens describes a different system.

The load is prefill-dominant when counting tokens and decode-dominant when counting time. The genuinely new tokens that have to be prefilled on each turn are only between 1.5 and 7.3 times the output, not 275. With the prefix cache working, the split of execution time runs from 91 % to 98.6 % in decode. You size by memory bandwidth and by KV cache capacity, not by prefill FLOPs.

The sizing risk is not the average, it is the collapse of the prefix cache. The median hit rate within a turn is 98 %, it falls to 55 % at the boundary between turns and to 8 % when the client changes model. With dynamically retrieved context, prefix reuse drops to 1 %. One miss turns 142,000 prefix tokens into real prefill. Session affinity per replica stops being an optimisation and becomes a capacity requirement.

The intra-turn concurrency measured in production is 1.15 at the median and 1.4 at the 90th percentile. 93 % of tool batches contain a single invocation. The image of the agent opening twenty calls in parallel does not appear in the data. The real multiplier comes from the duty cycle and from the fact that 87 % of calls are started by the agent, not by the person.

The gateway is sized per worker, and each worker is a universe. One vCPU and 4 GiB per worker, with requests equal to limits. The limit counters, the caches and the cooldowns live in the worker’s memory unless there is Redis, so a limit of 100 requests per minute with four workers and three replicas allows 1,200. The Prisma pool is also per process: the number of connections open against Postgres is workers times connection_limit.

The CPU cost that grows with agentic load is the token counter. It serialises the entire tool catalogue on every request and tokenises it. The project itself measured 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens, and the operation passes through a concurrency limiter of four per worker. With agents sending long catalogues and six-figure contexts, that limit is the ceiling before the CPU is.

Autoscaling by memory does not work on this piece and the startup probe takes five minutes. The official chart deliberately leaves memory without a target because Prisma’s query engine never returns the maximum reached. That leaves CPU at 60 %, with a startupProbe of 300 seconds that makes any new replica arrive late. For agents there are two better metrics, and they are in /metrics.

You are here: the arithmetic that comes before buying

Classic capacity planning starts from a chat SLO and crosses two budgets, the VRAM one and the time one. That method is still correct. What changes with agents are the inputs: the prompt length, the number of calls per unit of human work, the split between prefill and decode, and the very definition of an SLO. This article replaces those inputs with measured figures and drags the result through to two numbers: gateway replicas and GPUs.

Out of scope is the splitting of traffic once it is built, which was already covered by the coexistence of humans and agents and prefix routing.

The analogy: the repair shop and the front desk

An appliance repair shop sizes its staff by counting faults, not phone calls. The difference matters because the person who brings in the washing machine occupies the front desk for three minutes, while the technician who takes it apart goes back to consult the machine’s data sheet fourteen times over the afternoon. If the shop manager sizes by visits to the front desk, they end up short of technicians and overstaffed at reception.

The inference platform has the same asymmetry. The person sends one turn. The agent sends fifteen calls for that turn, and each one drags along the complete file of everything done so far. The front desk is the gateway and the technicians are the GPUs, and the usual mistake is to size both with the same counter.

There is a second detail the analogy captures well. The technician does not read the whole data sheet each time: they have it open on the bench. That is the prefix cache. The day somebody moves them to a different bench between consultations, they have to fetch it from the archive again, and the work multiplies without the number of faults having increased.

Part 1. The real load profile

For two years, any capacity calculation for agents was done on assumptions. In 2026 three pieces of work appeared that measure the matter, and it is worth reading them together because they measure different scaffolds and their figures are not averageable.

The three sources

The characterisation at scale was published by a research team working on GitHub Copilot production telemetry (arXiv 2608.00101, 30 July 2026): 13.5 million sessions, 3.2 million users, 95.1 million user turns, 760.5 million model calls and 44.9 trillion prompt tokens, in one week of June 2026. It is the largest sample published to date.

The characterisation by benchmark suites comes from an architecture team (arXiv 2605.26297, 25 May 2026), which instruments five of them (ADE-Bench, DABStep, GAIA, SWE-bench Pro and Terminal-Bench 2.0) and measures the internal split of time.

The production measurement on a different scaffold was published by the vLLM project itself together with AgentX (vLLM blog, 8 September 2026).

The three disagree on turns per session: 15 calls at Copilot’s median, 43 turns in AgentX, 62.4 in SWE-bench Pro with Qwen. It is not a contradiction, they are scaffolds with different loops. The sizing lesson is that the number of calls per unit of human work depends on the client that connects, and that is why you have to measure it at home before fixing the size.

The figures you can actually use

Quantity	Measured value	Source
Median prompt per call	68K tokens, 63K of them cached	2608.00101
Input:output ratio (median)	above 275:1	2608.00101
Median session	3 user turns, 15 calls, 4.2 min	2608.00101
Mean session	6.1 turns, 40.6 calls, 62.6 min	2608.00101
Calls per turn (median)	4.5	2608.00101
Calls started by the agent	87 %	2608.00101
Range across user profiles	50× (23K to 1.1M tokens per turn)	2608.00101
Median input context	142K tokens; output 444 tokens per turn	AgentX
Accumulated context in SWE-bench Pro	mean 68.7 to 80.1K, maximum 146 to 166K	2605.26297
Reasoning tokens over the output	29.0 % to 67.6 %	2605.26297

Two operational observations come out of that and appear in no sizing guide.

The first: agents that fail accumulate up to 1.8 times more average context than those that finish well (2605.26297). The cost does not correlate with success, it correlates with difficulty and with getting stuck. A sizing done on the task that goes well falls short.

The second: the range across user profiles is 50 to 1. A user who only converses spends 23,000 tokens per turn; one who leaves the agent in a long loop spends 1.1 million. Segmenting by usage profile pays off more than any difficulty-based routing, because the variance is in the people, not in the requests.

What to measure before calculating

Three of your own figures, taken from a week of real traffic, are worth more than any table in this article:

Model calls per hour of human work, by user profile.
Prompt length distribution (median and 95th percentile), not the mean.
Aggregate prefix cache hit rate, with the time series, not the single number.

All three come from what the gateway already stores. In LiteLLM_SpendLogs you have the tokens per request and the key and team identifiers; in proxy_server_request you have the tools block of each call; and in vLLM’s /metrics you have vllm:prefix_cache_queries and vllm:prefix_cache_hits.

Part 2. Prefill, decode and the cache trap

Here is the finding that corrects almost everyone’s intuition, including mine when I started this article.

Counting tokens, agentic load is overwhelmingly input: 275 to 1. The apparently obvious conclusion would be to size by prefill capacity, that is by FLOPs. It is false.

What has to be prefilled on each turn is not the whole prompt, but what was added since the previous turn. That addition, measured, sits between 1.5 and 7.3 times the output length (2605.26297). The rest is served by the prefix cache. And the real split of the model’s execution time comes out like this: between 91.0 % and 98.6 % in decode, between 1.4 % and 9.0 % in prefill (2605.26297).

The hit rate that sustains that split is high and has been measured in three places: theoretical from 87.9 % to 99.3 % and empirical from 84.6 % to 99.5 % (2605.26297); above 96 % in production (AgentX); and a median of 98 % within a turn in Copilot (2608.00101).

Where it falls over

The same Copilot work measures the degradation, and it is the most actionable figure in the article:

98 % median hit rate within a turn. The detail matters: the first call of a turn hits around 45 %, the second rises to 86 % and from the third onwards it stabilises between 92 % and 94 %.
55 % at the boundary between turns, with idle gaps of 4.1 minutes at the median.
8 % when the client changes model.

And there is a fourth, worse regime: with dynamically retrieved context, that is when the scaffold injects search or RAG results that change on every iteration, prefix reuse falls to 1 % or less (arXiv 2608.15127, 15 August 2026).

A cache miss with a context of 142,000 tokens turns a prefill of a few thousand tokens into a six-figure one. The work multiplies by a factor of between 20 and 60 without the number of requests having changed. That is the difference between a fleet that meets its targets and one that seizes up mid-afternoon.

Two sizing rules come from that:

Without session affinity towards the same replica, the calculated sizing is worthless. LiteLLM’s concrete mechanism is session_affinity, which is not active by default and which was already covered in the prefix routing article. Without it, the calculation has to be redone assuming a prefill regime, which is between 20 and 60 times more expensive.
Changing model halfway through a session costs the entire cache. Any difficulty-based routing policy that moves a live conversation from one model group to another pays an 8 % hit rate on the next call. If you want to route by difficulty, do it at the start of the session.

Co-batching, which is the other effect nobody measures

Mixing a short request into the same batch as long-context requests degrades the short one’s inter-token latency. The measured figure: one long request in the batch raises time per token by 38.3 %; two raise it by 79.7 % (2608.15127). It is the technical reason why separating human and agent pools improves the human experience more than the capacity split suggests, and it is also the reason not to raise --max-num-seqs above what the KV budget allows.

Part 3. The KV budget, with the numbers of the agentic case

The formula does not change with respect to classic capacity planning, but the inputs do, and the result is surprising:

$$\text{KV}_{\text{bytes/token}} = 2 \times n_{\text{layers}} \times n_{\text{kv heads}} \times d_{\text{head}} \times \text{bytes}_{\text{elem}}$$

For a 70B-class geometry (80 layers, 8 KV heads with grouped attention, head dimension 128, 2-byte elements) that gives 0.3125 MiB per token. With the median agentic context of 142,000 tokens:

$$142{,}000 \times 0.3125\ \text{MiB} \approx 43.3\ \text{GiB per sequence}$$

For a 30B-class geometry (48 layers, everything else the same) that gives 0.1875 MiB per token, that is around 26.0 GiB per sequence.

It is worth pausing on what that means. An 80 GB H100 with the weights of a 70B model in FP8 already taking up space does not have 43 GiB free for a single sequence. The conclusion is harsh and has to be written down: serving real 70B-class agentic context demands tensor parallelism of at least 2 and a quantised KV cache. And the number of simultaneous sequences that fit, which is what sets --max-num-seqs, comes from dividing the available KV by that figure, not from a chat heuristic.

$$\text{sequences} \approx \frac{\text{VRAM}_{\text{KV available}}}{\text{KV}_{\text{bytes/token}} \times \text{max context}}$$

With FP8 in the KV, the divisor halves. It is the cheapest capacity lever there is in this load profile, and its cost in quality is validated with evals, not assumed.

Part 4. From people to concurrent requests

This is where most calculations overshoot, because it is assumed that an agent equals many simultaneous requests. The data says otherwise.

The concurrency of calls within a turn, measured in production, is 1.15 at the median and 1.4 at the 90th percentile, and 93 % of tool batches contain a single invocation (2608.00101). Intra-turn parallelism is small. Subagents appear in 44 % of AgentX sessions, with a median of 4 deployments among those that use them, so there is a long tail, but it is not the usual regime.

What does multiply is the duty cycle. The mean session lasts 62.6 minutes and contains 40.6 calls. At 444 output tokens and an interactivity of around 60 tokens per second, each call occupies around 7.4 seconds of decode, which gives between 5 and 7 active minutes out of those 62.6.

$$C = D \times \rho \times A$$

where $D$ is the people working, $\rho$ the duty cycle and $A$ the amplification. With $\rho \approx 0.10$ to $0.12$ derived from the previous figures, twenty developers give between 2 and 2.4 concurrent sequences in the steady state. It is the peak, not the average, that has to be sized for.

The amplification

Three factors published separately:

Tool failures: 9 % of turns trigger them, with a compute amplification of 4 times (2608.00101).
Subagents: 44 % of sessions, median of 4 deployments (AgentX).
Internal variance: the same task from the same agent varies by up to 30 times in cost within a single trace (2608.15127).

The product of the first two gives $(1 + 0.09 \times 3) \times (1 + 0.44 \times 3) \approx 2.9$. Nobody publishes that product as a single figure, so it is flagged as my own derivation, but an amplification factor of 3 is the reasonable design provision and it fits the measured variance.

For the twenty developers in the example: between 6 and 7 concurrent design sequences, with 142K contexts. At 43.3 GiB of KV per sequence in the 70B class, that is around 300 GiB of KV cache, that is four H100s dedicated to KV alone with the weights on top. With KV in FP8, half. With a 30B-class model, a third.

That calculation, done with the chat profile of 800 prompt tokens, would have given a single GPU. That is the distance between the two regimes.

Part 5. The gateway, which also has to be sized

The proxy consumes no GPU, but it has its own arithmetic and it is the part that produces the most surprises in production, because almost all of its state is per process.

A worker is a universe

The default value of --num_workers is 1 (litellm/constants.py:57). The usual startup is uvicorn directly with uvloop (proxy_cli.py:1477), and there are alternatives with gunicorn, hypercorn and granian.

What lives in each worker’s memory, and therefore multiplies:

The in-memory cache, limited to 200 elements with a TTL of 600 seconds (litellm/caching/in_memory_cache.py:27). With many deployments, the cooldowns and the counters compete for that same space and get evicted before they expire.
The key authentication cache, with a TTL of 60 seconds (proxy_server.py:1650), adjustable with general_settings.user_api_key_cache_ttl.
The limit and budget counters when there is no Redis (parallel_request_limiter_v3.py, local_only=True paths).
The router cooldowns, with DEFAULT_COOLDOWN_TIME_SECONDS = 5 (constants.py:75).
A periodic task scheduler of its own per worker (proxy_server.py:9674), coordinated between pods with a lock in Redis.

The practical consequence: a limit of 100 requests per minute with four workers and three replicas allows up to 1,200 without Redis. With Redis, the v3 limiter registers seven Lua scripts at startup (parallel_request_limiter_v3.py:608) and the limits become global.

One detail of the v3 limiter that matters with agentic load: LITELLM_TPM_TOKEN_RESERVATION_ENABLED comes enabled by default and reserves estimated tokens before the call, which adds a round trip to Redis per request. And there is an asyncio.Lock per instance that serialises the atomic batch checks (:665), documented by the code itself as a 99th-percentile risk under contention.

If Redis goes down, requests do not fail: there is a circuit breaker with a threshold of 5 failures and a recovery of 60 seconds (constants.py:456), and the proxy degrades to local memory. What is lost silently is the globality of the limits.

Postgres

The pool is also per process. The default values are database_connection_pool_limit = 10 and database_connection_pool_timeout = 60 (proxy_cli.py:62), injected as parameters of the Prisma URL. A pod with four workers opens forty connections. Three pods, one hundred and twenty. It is the fast route to exhausting Postgres’s max_connections without having served any traffic, and the reason the official chart offers PgBouncer inside the pod itself with maxDbConnections: 20 and maxClientConn: 1000.

On the request path there is only the authentication, which is cached for 60 seconds, and the rate limiting. Spend is deferred. The SpendLogs queue is capped at 64 MB and on overflow it discards the oldest elements with an error in the log (utils.py:6704). The write interval is proxy_batch_write_at, with a default value of 10 seconds plus a random spread of up to 5 (constants.py:1719, proxy_server.py:9704); the production guide recommends raising it to 60.

With a slow Postgres, the queue grows to those 64 MB and then loses billing data without returning any error to the client. It is the silent failure mode you have to alert on.

The token counter, which is the agentic trap

This is the piece that turns a correct sizing into an insufficient one, and it appears in no guide.

Before admitting a request, the proxy counts tokens. With chat load the cost is negligible. With agentic load it is not, for two reasons that add up.

The first is the prompt length. The project itself measured the admission counting cost at 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens (benchmarks, consulted on 12 September 2026), and published that with a counter written in Rust those figures fall to 4.9, 6.8 and 10.2 ms.

The second is the tool catalogue. The function that prepares the definitions serialises the entire catalogue to a pseudo-TypeScript and tokenises it (litellm_core_utils/token_counter.py:934), plus nine fixed tokens. The cost is linear in the size of the catalogue and it is paid on every request. That is exactly the pattern of an agent connected to an MCP gateway: a long catalogue, resent on every turn. The token bill of that catalogue was already covered in the MCP gateway article; here its other face appears, which is proxy CPU.

And on top of that there is an explicit ceiling. The counting runs in a thread with a capacity limiter whose default value is 4 per worker (token_counter.py:340, constants.py:407). With 100K prompts, four concurrent counts of 100 ms mean a theoretical ceiling of around 40 admissions per second per worker before the limiter starts queueing, and that is without having called the model yet.

What is cached is the tokeniser, with an LRU of 64 entries (utils.py:2206), not the result of the count.

Three levers, in order of return: raise TOKEN_COUNTER_MAX_CONCURRENT_COUNTS if CPU is spare, reduce the tool catalogue exposed per key, and watch the gateway overhead figure in /metrics to know whether this is happening.

Streaming and admission

There is no connection limit by default. --limit_concurrency exists but is optional, and it returns 503. The built-in admission control is only activated if general_settings.max_in_flight_requests_per_worker is defined (middleware/admission_control_middleware.py:285), with max_queued_requests_per_worker and admission_queue_timeout_seconds at 1.0 by default, rejecting with 503 and a retry-after: 1 header.

For agents, that header matters: an agentic client that retries without respecting it multiplies the load exactly when the gateway is saying it cannot take any more. The combination of gateway retries was already calculated by the fourth article and it reached 45 calls per turn in the worst case.

The default request timeout is 6,000 seconds (constants.py:512). The production guide recommends 600. With long agentic tasks, leaving it at 6,000 means a hung connection occupies a slot for an hour and a half.

Resources and autoscaling

The official chart recommends 1 vCPU and 4 GiB per worker, with requests equal to limits (helm/litellm-helm/values.yaml:235), and the production documentation suggests one worker per pod scaling horizontally. The project’s own high-performance benchmark, by contrast, uses four workers per pod with PgBouncer in front.

The HPA comes with targetCPUUtilizationPercentage: 60. And there is a comment in the chart worth reading twice: memory is deliberately left without a target because the resident consumption of Prisma’s query engine is a high-water mark that is never returned, so that an HPA on memory would add replicas and never remove them (values.yaml:260).

The other problem is startup. The startupProbe has failureThreshold: 30 with a period of 10 seconds, that is up to 300 seconds before the pod is considered ready (values.yaml:109). An HPA that reacts at 60 % CPU and takes five minutes to add capacity arrives late to any agentic burst.

The alternative, optional in the chart, is to scale by Prometheus metrics: targetRequestsPerSecond and targetTokensPerSecond over rate(litellm_proxy_total_requests_metric_total[1m]) and rate(litellm_total_tokens_metric_total[1m]). The chart itself warns that the token metric lags with long streaming responses, because tokens are accounted for on completion. For agents, that is almost always.

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

Metric	What for
`litellm_in_flight_requests`	Real saturation per worker. It is the scaling metric
`litellm_overhead_latency_metric`	Proxy cost per request; it rises with token counting
`litellm_overhead_with_guardrails_latency_metric`	The same including guardrails
`litellm_request_queue_time_seconds`	Arrival through to pre-call, includes authentication and ASGI queueing
`litellm_deployment_cooled_down`	Replicas out of the pool
`litellm_service_latency`	Redis and database latency, per service

The /health/backlog endpoint returns in_flight_requests, admitted_requests, queued_requests and rejected_requests (health_endpoints/_health_endpoints.py:1871).

What it does not expose: the depth of the SpendLogs queue or its bytes, the Prisma connections in use against the limit, nor the depth of the logging worker’s queue. All three are real failure modes and have to be instrumented from outside.

At the engine

Names verified in the vLLM 0.29.0 metrics documentation:

vllm:num_requests_running, vllm:num_requests_waiting
vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:inter_token_latency_seconds
vllm:prefix_cache_queries, vllm:prefix_cache_hits, vllm:kv_cache_usage_perc

And the ones that break dashboards on upgrade:

vllm:time_in_queue_requests is duplicated by vllm:request_queue_time_seconds.
The direct prefix cache hit rate indicator disappeared. It is now computed as rate(vllm:prefix_cache_hits) / rate(vllm:prefix_cache_queries).
vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are deprecated along with CPU swapping.
vllm:gpu_cache_usage_perc became vllm:kv_cache_usage_perc, which also breaks any HPA that uses it.

The SLO worth chasing

A 95th percentile of time to first token describes nobody’s experience when the unit of work is 43 turns. Coding tasks frequently exceed ten minutes and research tasks reach hours (2608.15127).

Two indicators serve better:

90th percentile of full-task latency, measured per session, not per request. It comes from grouping LiteLLM_SpendLogs by session identifier.
90th percentile of interactivity, in tokens per second per user. It is the one AgentX uses, with values between 58.3 and 74.2 in its runs.

To autoscale the engine, the official production-stack example uses vllm:num_requests_waiting with a threshold of 5, polling every 15 seconds and a cooldown of 360. With agentic load it is worth adding vllm:kv_cache_usage_perc as a second trigger, because with 26 to 43 GiB of KV per sequence the memory pressure arrives before the queue does.

Part 7. The engine levers, ordered by return

Verified against vLLM 0.29.0, released on 9 September 2026.

--long-prefill-token-threshold 512. The AgentX measurement attributes to it +93 % in tokens per GPU per second, with a 2.3-times improvement at the 90th percentile, by attacking the head-of-line blocking that a long prefill produces in front of short requests. It is the highest-return lever documented for this profile.

KV cache in FP8. It halves the memory budget per sequence, which is the limiting resource. Validate with evals.

--max-num-queued-reqs and --max-num-queued-tokens. New in 0.29. They are admission control in the engine, and they are the right mechanism to protect the SLO instead of letting the queue grow without limit.

max_num_batched_tokens downwards. High values improve time to first token; low values improve inter-token latency. In a regime where 91 % of the time is decode, inter-token latency is what is perceived. Chunked prefill is enabled by default whenever possible in the V1 engine.

Session affinity at the gateway. It is not an engine setting, but it is what holds up everything above. Without it, that 98 % hit rate turns into random distribution across replicas.

Two version warnings: Model Runner V2 becomes the default in 0.29 and V1 is deprecated with removal planned for 0.32; and starting with python -m vllm.entrypoints.openai.api_server is deprecated.

The full calculation, from start to finish

Case: twenty developers, a coding scaffold, a 30B-class model in the fleet, maximum context of 128K, KV in FP8.

1. Steady-state concurrency. $C = 20 \times 0.11 \times 3 \approx 6.6$ sequences.

2. KV budget per sequence. 0.1875 MiB per token in BF16, half in FP8, that is around 0.094 MiB. At 128K of context: around 12.0 GiB per sequence. For 6.6 sequences with margin, rounding to 8: 96 GiB of KV.

3. GPUs. The weights of a 30B in FP8 take up around 30 GB. On an 80 GB H100 with gpu_memory_utilization of 0.90 there are around 72 GB usable per card. With tensor parallelism of 2, the weights are split: around 15 GB per GPU, so around 57 GB of KV per GPU and around 114 GB in the replica. That covers the 96 GiB calculated, and TP=2 also adds memory bandwidth, which is the resource that dominates decode. Two H100s, one replica, TP=2. It is worth seeing the contrast: on a single H100 the full weights leave around 42 GB of KV, that is three and a half sequences, less than half of what is needed. With a second replica for availability and maintenance windows: four GPUs.

4. Requests per second at the gateway. 20 people, a mean session of 40.6 calls in 62.6 minutes, duty cycle included: of the order of 0.2 to 0.3 requests per second on average, with peaks of 2 to 3.

5. Gateway workers. The ceiling is not set by throughput but by token counting: four concurrent counts per worker, at 100 ms with long prompts. With peaks of 3 requests per second and six-figure prompts, two replicas of one worker each cover the case with margin for the rolling update, at 1 vCPU and 4 GiB each. Postgres will see 20 connections.

6. What needs switching on. Session affinity, admission control at the gateway and at the engine, --long-prefill-token-threshold, and Redis so that the limits are global.

The same calculation with the chat profile would have given one GPU and one worker. The distance is not in the safety margin, it is in the inputs.

A reference `config.yaml`

model_list:
 - model_name: agentes-30b
 litellm_params:
 model: hosted_vllm/qwen3-coder-30b
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 rpm: 600
 model_info:
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072

router_settings:
 routing_strategy: simple-shuffle
 # without this, the capacity calculation in this article is worthless
 optional_pre_call_checks: ["session_affinity"]
 num_retries: 2
 allowed_fails: 3
 cooldown_time: 30

general_settings:
 # admission control: below this the queue grows without limit
 max_in_flight_requests_per_worker: 24
 max_queued_requests_per_worker: 48
 admission_queue_timeout_seconds: 1.0
 # the spend queue discards at 64 MB; shorter interval, less backlog
 proxy_batch_write_at: 60
 enable_drain_endpoint: true
 drain_endpoint_token: os.environ/DRAIN_TOKEN
 database_connection_pool_limit: 10

litellm_settings:
 # 6000 s by default: a hung connection holds a slot for an hour and a half
 request_timeout: 600
 callbacks: ["langfuse_otel", "prometheus"]
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

Environment variables that go with it:

NUM_WORKERS=1
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS=8 # default 4; raise only if CPU is spare
LITELLM_TPM_TOKEN_RESERVATION_ENABLED=true
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS=30

And on the engine side:

vllm serve Qwen/Qwen3-Coder-30B \
 --tensor-parallel-size 2 \
 --max-model-len 131072 \
 --kv-cache-dtype fp8 \
 --long-prefill-token-threshold 512 \
 --max-num-queued-reqs 64 \
 --gpu-memory-utilization 0.90

Checklist

Measure three figures of your own over a week: calls per hour of human work and per profile, prompt length distribution, and the time series of prefix cache hit rate.
Calculate the KV per sequence with the model’s real geometry, taken from its config.json, not from a table.
Set --max-num-seqs from that budget, not from a chat heuristic.
Switch on session affinity before accepting the calculated number of GPUs.
Switch on admission control in both layers and check that the client respects retry-after.
Lower request_timeout from 6,000 to 600.
Raise proxy_batch_write_at to 60 and alert on the growth of the spend queue from outside, because there is no metric.
Leave the HPA on CPU at 60 %, never on memory, and assume the 300 seconds of the startupProbe when planning the margin.
Watch litellm_in_flight_requests and litellm_overhead_latency_metric; if the overhead rises with the size of the tool catalogue, the culprit is the token counter.
Go over the dashboards for the vLLM metric renames before upgrading.
Check that no routing policy moves a live session between model groups.
Redo the calculation when a new user profile arrives: the range across profiles is 50 to 1.

Traps and things that are not what they seem

The ratio of 275 to 1 does not mean the system is prefill-bound. The new tokens per turn are between 1.5 and 7.3 times the output. The time is between 91 % and 98.6 % in decode.

The aggregate prefix cache hit rate hides the problem. An aggregate 90 % may be 98 % intra-turn and 55 % at boundaries. You have to look at the series, not the number, and segment by request type.

vLLM’s hit rate counts the reprefill after an eviction as a hit. The counter exists internally and is not exposed, so the published number is optimistic under a memory-pressure regime.

Agents that fail cost more than those that succeed, up to 1.8 times more context. Sizing on the task that goes well leaves the calculation short.

The database pool is per process, not per pod. Four workers are forty connections.

The HPA on memory does not work with this piece, and not because of a configuration mistake: Prisma’s query engine does not return the memory.

The 300-second startupProbe turns any autoscaling into late reaction. For agentic peaks, it is better to over-provision than to trust the scaling.

Raising --max-num-seqs to “make use of the GPU” degrades perceived latency: two long requests in the batch raise time per token by 79.7 %.

The token counter has a limiter of four per worker and it appears in no sizing guide. With large tool catalogues and long contexts it is the first ceiling you hit.

The figures in this article come from three different scaffolds and are not averageable. Fifteen, forty-three and sixty-two turns per session are three systems, not three measurements of the same one.

Closing

The arithmetic of sizing has not changed. What has changed are the inputs, and now there are public measurements at scale instead of intuitions. Three figures sum up the change of regime: 68,000 tokens of median prompt, 87 % of calls started by the agent rather than by the person, and a time split that is decode for more than 90 % as long as the prefix cache hits.

From that comes an order of work that is not the usual one. First, session affinity, because without it the rest of the calculation describes a different system. Second, the KV budget with the model’s real geometry, which is what sets how many sequences fit. Third, the gateway, where the ceiling is not throughput but a token counter with four threads per worker.

And a warning about your own figures. The measured range across user profiles is 50 to 1. Any platform that does not segment its measurement by profile is averaging two different populations and producing a number that describes neither of them.

Sources

Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale, arXiv 2608.00101, 30 de julio de 2026.
Agentic AI Workload Characteristics, arXiv 2605.26297, 25 de mayo de 2026.
From LLM Inference to Agentic Workloads, arXiv 2608.15127, 15 de agosto de 2026.
vLLM x AgentX: serving agentic workloads, 8 de septiembre de 2026.
vLLM metrics design y optimization guide, consultados el 12 de septiembre de 2026.
LiteLLM benchmarks y production setup, consultados el 12 de septiembre de 2026.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Autoescalado con KEDA en vLLM production-stack, consultado el 12 de septiembre de 2026.

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

Fri, 11 Sep 2026 00:30:00 +0200

Fifth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability, virtual keys covered governance and humans and agents covered the coexistence of two classes of traffic. Here a claim of mine from June gets taken apart. Everything that follows is verified against LiteLLM 1.102.0 (commit of 10 September 2026), vLLM 0.29.0 and the main branches of production-stack, llm-d and Dynamo from the second week of September 2026.

TL;DR

LiteLLM does not do prefix-aware routing, neither in the open version nor in the paid one. It is not a licensing question. A search across the whole tree for kv_aware, cache_aware, radix, prefix_cache and block_hash returns zero matches in the router. Nor is there any call from the router to /metrics or to any vLLM endpoint. In June I wrote the opposite in the inference router article and I was wrong.

What LiteLLM has is affinity, which is a different thing. Four pre-call checks pin a request to a replica by session identifier, by key hash, by previous response identifier or by exact hash of a marked prefix. None is active by default. Affinity distributes conversations; prefix routing distributes prefixes shared between different conversations, which is the case that multiplies throughput.

The only one that looks at the prompt is fragile in three ways. If the client does not send cache_control, the key comes out null and the check does nothing. If the cut point goes in the last turn, which is what the automatic injection does by default, the key changes on every turn and it never hits. And the pin’s TTL is 300 seconds hardcoded in two lines, while the provider offers one-hour caches.

The engine does not answer questions, it publishes events. There is no vLLM endpoint you can ask whether it has a prefix. What there is, is a ZMQ socket that emits stored and removed blocks in msgpack, switched off by default, with silent discarding when the consumer does not keep up. A router that wants to be exact has to replicate the engine’s hash in full: chained SHA256, deterministic seed, block granularity and the extra keys in their order.

And by default it does not publish the hits, only the additions. The index of a router that only listens drifts away from the engine’s real LRU without noticing. To see the hits you have to ask for it request by request with kv_cache_report_mode: full.

The three routers that do do it are very different from each other. The vLLM production-stack one indexes chunks of 128 raw characters in a trie that is never purged, models no eviction and dies on every restart. The llm-d one offers an approximate mode and an exact event-driven one, all its plugins are in alpha or beta, and its default tokeniser does not produce real tokens. NVIDIA Dynamo’s keeps an event-driven radix tree with a weighted cost function, and ships with the parameter that avoids starving a freshly scaled replica switched off.

The big improvements only appear in the best possible case. The two- and three-digit numbers are published by the projects themselves, with synthetic shared prefixes and a round-robin balancer as the reference. The most complete independent measurement, from Meta, obtains 2.3 times more throughput and also identifies load regimes where affinity leaves capacity at between half and two thirds of the reference. Another, on a multi-region deployment with 89 % reuse, measures single-digit improvements.

The correct architecture has two layers, not one. LiteLLM on top for identity, budget and audit, pointing at a single destination per model group. The KV-aware router underneath, inside the engine’s domain. Mixing it into a single piece is not an option that exists today.

You are here: between the gateway and the engine

This article occupies the gap between two layers that already have an article of their own. Above, the L7 switchboard and the gateway that has been operated since the first article in this track. Below, the fundamentals of the KV cache and hit rate engineering.

The gap is the piece that decides which replica each request goes to, knowing what each replica has in memory. It is the only function in the stack that cannot be resolved in the gateway or in the engine separately.

The analogy: the library with four counters

A library with four counters. Each librarian has in front of them a pile of the books they have consulted lately. When a reader comes in asking for the third volume of an encyclopaedia, if that volume is in the pile at counter two, serving them there takes seconds. At any other counter someone has to go down to the store room.

A normal balancer sends the reader to the counter with the shortest queue. It is the right decision if all the counters are interchangeable, and it is the wrong decision here, because they are not: what distinguishes them is not the queue, it is the pile.

Session affinity resolves one specific case of this. The returning reader is always sent to the same counter, because they will probably ask for the fourth volume of the same encyclopaedia. It works, and it does not resolve the general case: when a hundred different readers ask for the same third volume, session affinity spreads them across the four counters and four librarians go down to the store room for the same book.

Prefix routing is looking at what the reader asks for before assigning them a counter. It has two costs that the analogy makes visible straight away. The first is that somebody has to keep track of what is in each pile, and that bookkeeping takes work. The second is that if everybody asks for the third volume, counter two becomes the library’s only desk while the other three look on.

The correction: what I wrote in June

In the inference router article, on 2 June 2026, I closed the prefix routing section with this sentence: “vLLM Production Stack router implements it natively. NVIDIA Dynamo too. LiteLLM in its enterprise version has a beta.”

The first two hold up. The third is false and was no less false in June. While preparing the humans and agents article the first hint appeared, and full verification against 1.102.0 confirms it.

Case-insensitive search over litellm/, enterprise/ and gateway/, for the terms kv_aware, kv-aware, cache_aware, cache-aware, radix, prefix_cache, prefix-cache, kv_cache, block_hash, prefix_hash, prefix_score and longest_prefix: zero matches. Search for vllm inside litellm/router.py, litellm/router_strategy/ and litellm/router_utils/: zero. Search for num_requests_waiting, gpu_cache_usage or any scraping of the backends’ /metrics from the router: zero. LiteLLM’s router never asks the engine anything.

Search for premium_user in the files where those checks would live: a single result, and it is about something else. There is no paid version with this feature. The only routing piece that is behind a licence is per-tag budgets, in litellm/router_strategy/budget_limiter.py.

Where the error came from. LiteLLM’s documentation has an entry on dynamic TPM and RPM allocation marked as beta and enterprise, and a pre-call check called prompt_caching. Skim-read and added together, they give the impression of prefix routing in beta. Read in the code, prompt_caching is something else, and it deserves a section of its own because it is still recommended for this problem.

What LiteLLM does have: four affinities

The optional_pre_call_checks field accepts eight values. None is enabled by default: apply_default_settings passes an empty list. One of the eight, forward_client_headers_by_model_group, is declared in the type and in the interface schema and is consumed by no branch of the code. It is a dead value.

The four that pin a request to a replica are these.

Check	What it pins	Where it stores it	TTL
`session_affinity`	Session identifier	Redis with a Lua script, mirrored in memory	`deployment_affinity_ttl_seconds`, 3600 by default, refreshed
`deployment_affinity`	API key hash	Same	Same
`responses_api_deployment_check`	`previous_response_id`	Same	Same
`prompt_caching`	Exact hash of a marked prefix	`DualCache`	300 s, hardcoded

The other four are router_budget_limiting, encrypted_content_affinity, enforce_model_rate_limits and the dead value.

`prompt_caching`: why it is no use for this

The key is a SHA256 over the serialisation of the cacheable prefix, and “cacheable prefix” means everything up to and including the last block marked with cache_control: {"type": "ephemeral"}. Three consequences, checked by running the functions in isolation over the repository’s code.

If the request carries no cache_control, the extraction returns an empty list, the key comes out null and the check pins nothing. Automatic injection of cut points is only enabled with litellm.enable_anthropic_prompt_caching, which defaults to false, and only for Anthropic and Bedrock providers.

If the cut point is in the system prompt, the key does not change when turns are added and the pin works. This is the good case and it requires the client to mark it deliberately.

If the cut point is in the last message, the key changes on every turn and the pin never hits. And this is exactly the default case of the automatic injection, which places one point on the system and another at index -1. Since the second is the last, the cacheable prefix becomes the whole conversation.

On top of that there is the TTL. Two lines with ttl=300, one of them with the comment # store for 5 minutes. There is no parameter to change it. Issue 28427 has had it open for months with no response from the maintainers, and it points to the mismatch with the one-hour ephemeral caches that LiteLLM itself knows how to request via cache_control.ttl: "1h".

There is a nuance the issue does not capture and that makes diagnosis worse: in the default case the pin does not even last five minutes, because the key changes first.

`session_affinity`: the one to use

For conversations, this is the right piece and it is reasonably built. The session identifier is looked for in this exact order of headers:

x-litellm-trace-id
x-litellm-session-id
Any x-<vendor>-session-id, with the regular expression ^x-.+-session-id$ and a value of at least eight alphanumeric characters
Only if the User-Agent is Codex’s: session-id, session_id, thread-id, conversation_id
Plain x-session-id, for opencode

And if nothing matches, two fallbacks: a metadata.user_id in Anthropic format, and the W3C baggage with session.id=. The third point is the one that captures Claude Code with nothing configured.

Storage is a Lua script in Redis that does GET, SET NX EX and EXPIRE, so the first writer wins and the TTL is refreshed on every request that confirms the pin. It is an inactivity TTL, not a total session duration. Without Redis it degrades to a pod-local reservation, which is atomic but not shared.

A detail of the pipeline order that already came up in the previous article and still holds: the affinity checks run inside async_get_healthy_deployments before tag filtering. If the pin returns a single deployment and that deployment does not satisfy the request’s tag, the later filter removes it and the request fails. Affinity gets no vote on the tags.

The routers that do read the prompt, and why they are no good

LiteLLM has four pieces that read message content: AutoRouter with semantic matching, RequestComplexityRouter, AdaptiveRouter and QualityRouter. They all live in async_pre_routing_hook, which runs before looking for healthy deployments.

The reason they do not solve this problem is in what they choose. All four choose a model group, not a replica. They decide whether an easy question goes to the thirty-billion-parameter model or to the small one. None has access to the replica list or knows what is in each one’s memory.

What did land: admission control

A minor correction to the previous article, which placed this in the 1.101 branch as an upcoming feature. In 1.102.0 it is already there, in litellm/proxy/middleware/admission_control_middleware.py, with the three settings in general_settings: max_in_flight_requests_per_worker, max_queued_requests_per_worker and admission_queue_timeout_seconds, the last with a default value of 1.0 seconds.

It stays inactive as long as the first one is not declared. It exposes an authenticated /health/backlog and the three metrics litellm_admission_admitted_requests, litellm_admission_queued_requests and litellm_admission_rejected_requests_total, the last with a reason label. It is a per-process semaphore, not coordinated across pods.

What the engine exposes upwards

Before looking at the routers that do do this, you have to understand what raw material they work with. The natural question would be whether vLLM has an endpoint you can ask about a prefix. It does not.

The only cache router in the entry points is vllm/entrypoints/serve/dev/cache/api_router.py, and it only exposes destructive writes: POST /reset_prefix_cache, /reset_mm_cache and /reset_encoder_cache. All three are behind VLLM_SERVER_DEV_MODE=1 and do not exist in a normal deployment. There is no read.

What there is, is an event stream.

The ZMQ stream

It is enabled with --kv-events-config, and it is off by default. The configuration accepts a publisher (null or zmq), an endpoint (by default tcp://*:5557, which binds), an optional replay endpoint, the number of batches retained for that replay (10,000), and a high water mark limit of 100,000.

That limit is the one that matters operationally: above it, ZMQ discards events if the consumer does not keep up. Silently.

The transport is a PUB socket with msgpack serialisation and three-part frames: topic, eight-byte sequence number and payload. The sequence number lets a subscriber ask for replay from where it left off through the optional ROUTER socket. With data parallelism the port shifts by rank, so you have to subscribe to 5557+rank.

Three event types. BlockStored carries the hashes, the parent hash, the token identifiers, the block size, the LoRA adapter name, the extra keys, the medium (GPU, CPU or storage) and a session identifier. BlockRemoved carries only hashes. AllBlocksCleared carries no payload and is emitted when the prefix cache is reset.

They are published once per scheduler step, only if there are events. With the feature switched off the cost is zero. With it switched on, the full token identifiers of each block travel over the wire, so bandwidth scales with tokens per second and not with requests per second.

An example in vLLM’s own repository declares itself experimental on its eighth line.

The hash that has to be replicated

An exact router does not receive the engine’s state, it reconstructs it. For that it has to compute the same hashes, and the engine computes them like this:

# vllm/v1/core/kv_cache_utils.py
def hash_block_tokens(hash_function, parent_block_hash, curr_block_token_ids, extra_keys):
 if not parent_block_hash:
 parent_block_hash = NONE_HASH
 return hash_function((parent_block_hash, tuple(curr_block_token_ids), extra_keys))

It is chained: each hash identifies the complete prefix up to that boundary, not the isolated block. The default algorithm is SHA256, configurable with --prefix-caching-hash-algo among four values.

Five things a router has to get right for its hashes to match the engine’s:

The seed. Since 0.29, NONE_HASH is deterministic from a fixed seed, so two different processes produce the same hashes with nothing configured. It is a silent change of semantics from earlier versions, which required setting PYTHONHASHSEED. With the xxhash algorithms it is still needed.

The block size. By default 16 tokens, but the attention backend can override it if the user did not pass --block-size: 256 with sliding-window sparse attention, 64 with ROCm’s AITER, 64 or more with FlashAttention on XPU. A router must not assume 16, it has to read it from the event.

The hashing granularity, which is not necessarily the physical block size. There is --prefix-match-unit, which with several cache groups resolves to the greatest common divisor of the cacheable groups’ sizes.

The extra keys, in their order: LoRA adapter name (by name, not by identifier), multimodal identifiers with their relative offset inside the block, cache_salt only in block zero, and the hash of the prompt embeddings. Sampling parameters do not go in, which is correct: the prefix KV does not depend on the temperature.

And truncation. By default the hashes travel over the wire as 64-bit integers truncated from a SHA256, not as the full bytes. It can be disabled with VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=0. With the default value, a router has a theoretical collision risk that the engine does not have.

The hits are invisible by default

This is the detail that takes the most effort to discover and has the most consequences. The events are incremental: only what gets newly cached is announced. A reused block generates no event.

A router that only listens sees the additions and the removals, but does not see the hits. Its notion of what is recent and what is cold drifts away from the engine’s real LRU with nothing to warn it.

A full mode exists, and it is per request. sampling_params.extra_args["kv_cache_report_mode"] = "full" makes BlockStored events be emitted for blocks that were hits too. The function’s docstring says so plainly: it generates events so that external consumers, “for example a gateway”, know which blocks were reused. That is to say: for this to work, the router has to inject a parameter into every request it forwards.

The metrics, and what broke

For aggregate measurement two counters remain: vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, both in tokens and not in requests, with model and engine labels. There are two equivalents for the external connector, which allow a hit in HBM to be distinguished from one brought in from CPU or NVMe.

And there are two removals that break existing configurations. vllm:gpu_prefix_cache_hit_rate no longer exists: zero appearances in the repository. It was deprecated in 0.8, hidden in 0.9 and removed in 0.10. And vllm:gpu_cache_usage_perc was renamed to vllm:kv_cache_usage_perc, which breaks Grafana panels and autoscalers that depended on the old name.

The hit rate has to be derived in PromQL with rate() over the two counters. The one that appears in the text log is something else: it uses a window of the last thousand requests.

And it has a bias you need to know about before taking decisions with it. When a request is evicted, its blocks return to the free queue keeping the hash, so the later reprefill finds them again and counts as a hit. A high hit rate can mean useful reuse between users or it can mean the engine is evicting and recomputing in a loop. The internal statistics separate the two things with a preempted field, which is not exposed as a Prometheus metric.

The last token

A small detail with an effect on long conversations: the maximum hit length is the request’s token count minus one. There is always something to recompute in order to get logits. Since the reservation requires alignment to the block size, in practice that can force recomputing the whole final block. With blocks of 16 it is noise. With a backend that prefers 256, it is up to 256 tokens recomputed per turn.

The ones that do do it

vLLM production-stack

Eight routing logics in a single --routing-logic selector: roundrobin, session, kvaware, loadaware, prefixaware, disaggregated_prefill, disaggregated_prefill_orchestrated and priority. Two of them, loadaware and priority, are not in the Helm chart’s enum and are only enabled through extraArgs.

The naming trap is that prefixaware and kvaware are radically different things.

prefixaware is a hash trie in the router’s process. It chunks the prompt string every 128 characters, not tokens, and hashes each chunk with 64-bit xxhash. Its own docstring says so: the chunk size is “in number of characters”. That 128 is hardcoded, there is no flag or Helm value to change it.

Four consequences come out of that. --prefix-min-match-length is measured in characters and is quantised to multiples of 128. A shared prefix that differs by one character inside the first chunk breaks the whole match, because there is no partial match inside a chunk. For chat, it concatenates the content of all the messages with newlines and without applying the chat template, so the indexed string is not what the engine tokenises. And the tie-break between tied replicas is random.choice, without looking at load.

The trie has no notion of saturation. A popular prefix concentrates all the traffic on the same replica indefinitely. The docstring assumes the simplified model and declares it: “we assume there is no prefix cache eviction”.

It has no pruning either. HashTrie only has insertion and longest prefix lookup; there is no deletion method. When a replica disappears, its URLs stay forever in every node’s sets, and filtering happens at query time. It works, and the trie grows monotonically: each distinct prompt adds its length divided by 128 permanent nodes, with no bound, no TTL and no LRU. The router’s memory limit in the chart is 1,000 MiB. By contrast, the round-robin router in the same file does bound its caches to 1,024 entries.

And it dies on every restart: it is in-process memory state, with no snapshot and no initial load.

kvaware and loadaware are better and have a different price. They do not index themselves: they ask the LMCache controller, which knows about real tokens. That requires lmcache==0.3.11 and vllm==0.13.0 pinned exactly, and the official router image drags in the whole of vLLM. The limitation to read before planning anything: they only look at request_json.get("prompt", ""), with an open TODO for chat completions. With /v1/chat/completions, both tokenise an empty string.

loadaware is the only one of the three that breaks affinity on saturation, with the formula score = relative_match - beta * relative_load and beta = 1.0 by default. The authors’ own reading: a replica at twice the average load loses the equivalent of a full cache hit. The guide suggests 0.25 to favour cache and 2.0 to favour balancing.

On published results, the documentation’s benchmarking file literally says that the measurement platform is coming soon. The project blog’s two posts claim between three and ten times lower response latency, and only one declares a partial configuration: Llama 3.1 70B on four nodes with tensor parallelism 2 over 80 GB A100s, inputs of 9K tokens and outputs of 10, against AIBrix 0.2.0 and a flat deployment. Charts with no labelled points, no absolute figures and no declaration of which routing logic was used in each.

llm-d

There has been a reorganisation here that invalidates a good part of the third-party documentation. The Endpoint Picker code left the Gateway API Inference Extension repository and was merged into llm-d’s, which has been renamed to llm-d Router. The Kubernetes project keeps the InferencePool API and a lightweight reference EPP that does nothing about prefixes. If you look for KV-aware routing in the extension’s repository, today it is not there.

The InferencePool is at v1 and is GA. The other two APIs moved to the llm-d.ai group at v1alpha2, and when they coexist with the old ones the EPP prefers the new ones and ignores the old.

Scoring is a weighted sum. The default weights, when no configuration is passed: queue 2, KV utilisation 2, prefix 3. Any scorer referenced without a weight counts as 1.0. There is no configuration through environment variables: the old ENABLE_*_SCORER no longer exist, everything goes in an EndpointPickerConfig in YAML.

The prefix scorer consumes an attribute published by a producer, and there are two producers.

The approximate one computes the hashes in the EPP itself without talking to the engine, and keeps a local LRU per pod and block pair that is populated after the decision. It is an approximation by construction. It has a hard floor: a configured block size below 64 tokens is raised to 64 at request time, because the index stores one entry per pod and block and going down to 16 would multiply memory by four.

The exact one builds the real index from the engine’s ZMQ events, with one subscriber per pod that is installed and torn down with the endpoint’s registration and deregistration. It has adapters for vLLM and SGLang, deduplicates evictions by reference counting, and the replay buffer that allows the index to be rebuilt after an EPP restart requires vLLM 0.26.0 or above.

The trap is in the tokeniser. The EPP tokenises through a plugin, and the default backend is estimate: byte packing with no tokeniser, roughly 4 bytes per token, with identifiers that do not correspond to real engine tokens. The exact scorer needs real tokens, and for that you have to explicitly configure the vllm backend, which does HTTP against the engine’s /v1/completions/render endpoints. If it is omitted, the auto-created estimate satisfies the dependency and the system degrades silently.

On maturity, the file itself declares it: all in-tree plugins are in alpha or beta, and they will be promoted to stable when the project approaches 1.0. An alpha plugin configured without the corresponding flag makes the EPP fail to start.

And on replicas, the warning is explicit: active-active mode should be avoided with approximate prefix routing, because the EPP replicas do not share state and each one only sees the traffic it has served itself, which significantly degrades the hit rate. The exact index is safe in high availability, and even so the reference guides fix a single replica because the accounting of in-flight requests is local to the process.

The cost is the best documented in the whole field, and it deserves a table. Simulator with Qwen3 8B, 100 pods, 100K input tokens:

Requests/s	Cap on tokens to match	Peak CPU	Peak memory
5.0	4,096	1.19 cores	0.26 GiB
5.0	100,000	3.82 cores	0.65 GiB
98.7	4,096	35.17 cores	2.46 GiB
98.8	100,000	46.50 cores	3.41 GiB

With outputs of 10K tokens at 50 requests per second, 32.53 cores and 12.54 GiB. The maintainers’ sizing rule is half a core to one core per request per second on large agentic loads. And idle consumption scales with the number of pods because of continuous scraping: with 100 pods, about 7.5 cores with no traffic.

NVIDIA Dynamo

A precise event-driven index, with a global radix tree in each frontend’s process. Since all the replicas consume the same event plane, there is no need to synchronise routers with each other for the prefix state.

The cost function is explicit and weighted. It discounts from the prefill the blocks that overlap in GPU, host and disk with different weights (1.0 for the overlap credit, 0.75 for host, 0.25 for disk), and adds the potential decode blocks and the active requests.

There is a parameter to look at before deploying: --router-kv-overlap-score-credit-decay is at 0 by default, that is, disabled. Its description says what it is for: preventing loaded, cache-rich replicas from winning again and again while the freshly autoscaled ones receive too little traffic. With the default value, a new pod has no cache, therefore never wins the scoring, therefore never generates cache.

The approximate mode exists (--no-router-kv-events, with a TTL of 120 seconds) and the documentation says it is not the recommended route in production.

Dynamo is the most honest project in its measurement methodology: its comparison guide recommends contrasting --router-mode random against kv over the public Mooncake trace from FAST'25, with a declared cache ratio of 59 %, and it generates the table locally instead of publishing a number.

The rest of the field

SGLang has renamed its router to sgl-model-gateway, version 0.3.2 within SGLang 0.5.19. Its cache_aware policy is an approximate radix tree over raw characters, with switching to shortest queue when the system is unbalanced according to two combined thresholds. One detail that can cost an afternoon: the Rust defaults and the Python CLI defaults do not match, and on the maximum tree size they differ by four orders of magnitude (10,000 nodes against 2^26).

AIBrix 0.7.0 offers a standard mode with a local hash table and a KV event synchronisation mode behind a flag and a build tag, with vLLM 0.7.0 or above.

Ray Serve LLM has PrefixCacheAffinityRouter with an explicit alpha API warning, a character-based prefix tree in a decoupled actor, and its imbalance threshold at infinity by default, that is, out of the box it never breaks affinity on load.

Envoy AI Gateway 1.1.0 implements none of this and delegates to InferencePool and the EPP. KServe the same, with its LLMInferenceService at v1alpha1. Mooncake Conductor is still an unimplemented proposal.

Comparison table

Project	Version and date	Index	Where it lives	Kubernetes	Declared maturity
production-stack `prefixaware`	main branch, 09-Sep-2026	Approximate, character trie, no eviction	Router process	Optional	WIP in the README
production-stack `kvaware`/`loadaware`	same	Exact via LMCache controller	LMCache controller	Optional	No chat completions
llm-d Router approximate	v0.10.0, 17-Aug-2026	Approximate, LRU per pod	EPP process	Yes	Beta, active-active discouraged
llm-d Router exact	same	Exact through ZMQ events	EPP process	Yes	Beta
NVIDIA Dynamo	v1.4.2, 27-Aug-2026	Exact, event-driven radix tree	Frontend process	No	GA with experimental subsystems
SGLang `sgl-model-gateway`	0.3.2, 03-Sep-2026	Approximate, character radix	Gateway process	No	Default policy
AIBrix	v0.7.0, 16-Jun-2026	Both	Gateway plugin	Yes	Synchronisation behind a flag
Ray Serve LLM	Ray 2.58	Approximate, characters	Ray actor	No	Declared alpha
Envoy AI Gateway	v1.1.0, 21-Aug-2026	None of its own	Delegates to EPP	Yes	Stable 1.x API

How much it really improves

Here two types of source have to be separated.

The projects’ own numbers. Red Hat published the llm-d case in May: Qwen3 32B, eight vLLM pods over 16 H100s with tensor parallelism 2, synthetic shared-prefix load with 150 groups of five prompts, 6,000 tokens of system prompt, 1,200 of question and 1,000 of output. Result: up to 109 % more throughput and up to 99 % less TTFT, around 200 concurrent users within SLO against around 20. The repository’s result files give more detail: peak throughput from 6,986 to 14,892 output tokens per second, p90 TTFT from 135.5 seconds to 0.26.

That same file publishes the regression, and that is why it is worth reading: p50 inter-token latency rises by 22.4 % with vLLM and by 45.4 % with SGLang. The trade-off is declared: routing by affinity concentrates more concurrent work on the pods with a warm cache.

It is the best possible case. A synthetic shared prefix and a reference that is a flat Kubernetes Service.

The independent measurements. There are three and they say different things.

CacheRoute, from Meta, August 2026, is the most complete. Llama 3.3 70B in fp8, 30 targets with tensor parallelism 2 over 60 H100s, a semi-synthetic trace from production telemetry with 128,824 business keys, and all the alternatives reimplemented in the same harness instead of comparing against a dumb balancer. It obtains 176 requests per second with a p99 SLO of 3.5 seconds against 76 for the best competitor, and a hit rate of 93.2 % against 72.0 %.

And it publishes two negative findings that nobody else publishes. There are load regimes where affinity reduces capacity to between half and two thirds of the reference, to the point that the authors make a mirrored-traffic trial mandatory before deploying. And growing the fleet can cool a prefix even though total cache capacity grows, because the time between revisits stretches beyond the eviction window. Scaling horizontally can make the hit rate worse.

GORGO, June 2026, measures on a multi-region deployment with a dataset where intra-user prefix reuse is 89.4 % and the average prompt is almost 18,000 tokens. An improvement of 6.9 to 15.5 % in p95 TTFT. With extremely high reuse, the improvement is single-digit, because network latency and queuing dominate.

LAAR, from IBM Research Tokyo, April 2026, is the uncomfortable counterpoint: in its comparison, session affinity routing was the worst of the baselines, and at 64K contexts a load-based routing gives lower absolute latency at the cost of fewer hits.

The joint reading fits in three conditions. The big improvements require high, concentrated prefix reuse, long prompts, and a weak reference. Take away any of the three and the numbers deflate.

The five failure modes

Hot spot from affinity. The busiest bucket inherits the load’s skew, and tail latency starts following the hottest target instead of the fleet average. Mitigations exist in the four main products and in three of the four they ship switched off: Dynamo’s credit decay at 0, Ray’s imbalance threshold at infinity, and production-stack’s prefixaware with none at all.

Starvation of the freshly scaled pod. A new pod has no cache, so it never wins the scoring, so it never warms up. Dynamo documents it as the reason for the decay parameter. I found no primary documentation from any of the projects about the specific interaction with KEDA.

Cost of the index. The llm-d numbers in the previous table are the best public reference that exists. Raising the cap on tokens to match from 16,384 to 400,000 can more than double the EPP’s CPU with little traffic.

Traffic with unique prefixes. Each project degrades to something different. SGLang routes to the smallest tree, which is not the same as the least loaded. Ray falls back to power of two choices. Dynamo has an explicit preset, --load-aware, to say “I want the load model without reuse”. And production-stack’s prefixaware keeps inserting into the trie, that is, keeps paying the cost while receiving nothing.

Sibling bursts. Parallel sampling, best of N, or an agent that opens five branches at once. With event-driven routing, no engine has yet emitted the block stored event, all the siblings score zero overlap, and the prefix is precomputed on every replica. Dynamo has a prediction TTL for this; the others do not.

The architecture that comes out of all of the above

Two layers, with the responsibilities separated.

On top, LiteLLM, with what it does do well: non-forgeable identity through the virtual key, budgets, spend logging, concurrency limits, fallback between providers and API unification. For the gateway, each model group has a single destination: the URL of the router below.

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://kv-router.inferencia.svc:8000/v1
 api_key: os.environ/VLLM_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 1
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

Below, the KV-aware router, inside the engine’s domain, talking to the vLLM replicas and consuming their events.

Four consequences of this separation that have to be accepted before building it.

Per-replica visibility in LiteLLM is lost. For the gateway there is a single deployment, so its cooldowns, its per-deployment counters and its strategies stop meaning anything. Observability of which replica served what goes downstairs.

Retries stack up. The client SDK retries, LiteLLM retries, and the router below may have its own failover. In the previous article 45 calls per turn came out with three layers; with four it is worse. Lowering num_retries in the gateway to 1, or to 0, is the sensible decision when there is a router below that already retries.

LiteLLM’s session affinity stops making sense for the KV, because it no longer chooses a replica. It is kept if it is used for something else, such as pinning a conversation to a specific provider.

And the choice of the router below conditions the rest of the cluster far more than the gateway’s does. llm-d requires Kubernetes 1.32 or above, Gateway API, the inference extension and a conformant Gateway. Dynamo does not require Kubernetes. production-stack has a Helm chart and works outside it. That decision is the expensive one, not the gateway’s.

When not to build this

Three cases where the correct answer is not to do it, at least not yet.

A single replica per model. With one replica, the engine’s prefix cache already does all the work and there is nothing to route. That is the case for a good part of the four H100 deployments with one large model and tensor parallelism.

Low or scattered prefix reuse. Before building anything, the measurement is rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) per replica, with a mental correction for the bias from reprefills after eviction. If it is already high with blind distribution, reuse is concentrated inside each conversation, and there LiteLLM’s session affinity costs one line of configuration and resolves the case.

Traffic where the network dominates. The GORGO result is the warning: with enormous prompts and a distributed deployment, the improvement is eaten up by queuing and network latency.

The case where it does pay off is specific: several replicas of the same model, a large system prompt shared between different users, or a RAG with a corpus of documents that repeat across requests. There the prefill saved is real and it is most of the work.

Checklist

Measure the current hit rate per replica with the two vLLM counters before touching anything, and check whether the engine is evicting with vllm:num_preemptions_total.
Check that prefix caching is switched on. It is no longer --enable-prefix-caching that switches it on: the default value is derived from the model and the log of the decision is at DEBUG level. What is actionable today is --no-enable-prefix-caching to switch it off.
Audit the prompts before routing: a timestamp or a session identifier at the start of the system prompt breaks the match from block zero and no routing layer fixes it.
If the KV event stream is enabled, size the bandwidth by tokens per second and watch discarding at the high water mark. With data parallelism, subscribe to one port per rank.
If an exact router is chosen, verify the vLLM version against the one the replay buffer requires.
If llm-d is chosen, explicitly configure the vllm tokenisation backend. The default estimate does not fail, it degrades.
If production-stack is chosen with kvaware or loadaware, check first whether the traffic goes through /v1/chat/completions, because today it does not work there.
Review the load-based affinity-breaking parameter, because in three of the four main projects it ships disabled.
Lower num_retries in LiteLLM when there is a router below that already does failover.
Trial with mirrored traffic before changing production. It is the explicit recommendation of the only independent measurement that went looking for bad regimes, and found them.

Traps and things that are not what they seem

LiteLLM has no prefix routing in any version. Neither the open one nor the paid one. I got it wrong in June.
prompt_caching does nothing if the client does not send cache_control, and with the default automatic injection it fails on every turn because it marks the last message.
That pin’s TTL is 300 hardcoded seconds, incompatible with the one-hour caches that LiteLLM itself knows how to request.
production-stack’s prefixaware indexes characters, not tokens, in non-configurable chunks of 128, and its trie is never purged and models no eviction.
production-stack’s kvaware and loadaware do not support chat completions. They tokenise an empty string.
The default tokeniser in llm-d’s EPP is a byte estimate, and with it the exact scorer degrades silently.
llm-d’s active-active mode is discouraged with approximate prefixes, and the reference guides fix one replica even with the exact one.
Dynamo’s overlap credit decay ships at zero, which can leave a freshly autoscaled replica with no traffic.
The SGLang router’s default values differ between the Rust and the Python CLI, by as much as four orders of magnitude.
vllm:gpu_prefix_cache_hit_rate no longer exists and vllm:gpu_cache_usage_perc is now called vllm:kv_cache_usage_perc. Old panels and autoscalers are measuring nothing.
The hit rate counts the reprefill after eviction as a hit. A high number can be reuse or it can be thrashing.
KV events do not publish the hits by default, only the additions, unless full mode is requested request by request.
The hashes travel truncated to 64 bits unless it is disabled with an environment variable.
The default block size is 16, and the attention backend can change it without anyone asking.
The Endpoint Picker left the Gateway API Inference Extension repository. Looking for it there today leads nowhere.

Closing

June’s error has a reading that goes beyond the erratum. It came from reading a product’s documentation and not its code, and from the fact that LiteLLM’s documentation uses the word “caching” in a pre-call check that does something else. The distance between what a name suggests and what a function does is, at this point in the track, the most repeated pattern of the whole series.

The architectural conclusion is that KV-aware routing is not a gateway function. It is not one today in LiteLLM and probably should not be: it requires keeping an index synchronised with each engine’s memory, replicating its hash function, consuming an event stream and sizing CPU per request per second. That belongs to the engine’s domain, not to that of identity and budget.

What remains is a two-part decision. The first is whether the traffic deserves it, and that is answered with two Prometheus counters and without installing anything. The second is which of the three routers fits the cluster you already have, and there the deciding variable is not the hit rate but whether you are willing to bring in Gateway API and the inference extension, or to pin vLLM 0.13.0, or to operate a Dynamo frontend.

And the honesty the material demands: the only independent measurement that specifically went looking for regimes where this makes things worse found them, and they are not marginal. Half to two thirds of reference capacity is a serious regression. That no project publishes that side of the curve does not mean it does not exist in theirs.

Sources

LiteLLM, Router - Load Balancing (estrategias, comprobaciones previas, afinidad): https://docs.litellm.ai/docs/routing.
LiteLLM, código: litellm/router.py, litellm/router_utils/prompt_caching_cache.py, litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/litellm_pre_call_utils.py, litellm/proxy/middleware/admission_control_middleware.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
vLLM, código: vllm/v1/core/kv_cache_utils.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_manager.py, vllm/distributed/kv_events.py, vllm/config/kv_events.py, vllm/v1/metrics/loggers.py, vllm/v1/metrics/stats.py: https://github.com/vllm-project/vllm.
vLLM, KV offloading usage (niveles CPU, filesystem y objeto; compartición entre procesos): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html.
vLLM, notas de la versión 0.29.0 (semilla determinista, seguimiento de propiedad en eventos, flags de admisión): https://github.com/vllm-project/vllm/releases/tag/v0.29.0.
vLLM, RFC 16669, publicación de bloques KV y métricas: https://github.com/vllm-project/vllm/issues/16669.
vLLM production-stack, código: src/vllm_router/routers/routing_logic.py, src/vllm_router/prefix/hashtrie.py, src/vllm_router/parsers/parser.py: https://github.com/vllm-project/production-stack.
LMCache Lab, resultados de production-stack frente a AIBrix (parte interesada): https://blog.lmcache.ai/en/2025/03/06/open-source-llm-inference-cluster-performing-10x-faster-than-sota-oss-solution/.
llm-d Router, código y documentación de operaciones (pesos por defecto, dimensionamiento del EPP, aviso de activo-activo): https://github.com/llm-d/llm-d-router.
llm-d, guía de enrutado preciso por prefijo y resultados publicados (parte interesada): https://github.com/llm-d/llm-d.
llm-d Router, incidencia 1290, estado de prefijo no compartido entre réplicas del EPP: https://github.com/llm-d/llm-d-router/issues/1290.
Red Hat, Same 16 GPUs, twice the users (parte interesada): https://www.redhat.com/en/blog/same-16-gpus-twice-users-inference-aware-routing-llm-clusters.
Kubernetes SIG Network, Gateway API Inference Extension (traslado del EPP y del BBR): https://github.com/kubernetes-sigs/gateway-api-inference-extension.
NVIDIA Dynamo, diseño y ajuste del router KV: https://github.com/ai-dynamo/dynamo.
SGLang, sgl-model-gateway: https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway.
AIBrix, enrutado por caché de prefijo: https://github.com/vllm-project/aibrix/blob/main/pkg/plugins/gateway/algorithms/prefix_cache_readme.md.
Ray Serve LLM, enrutado consciente de prefijo (API en alfa): https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html.
Huang Cheng (Meta), CacheRoute, agosto de 2026: https://arxiv.org/html/2608.19677.
Yuan et al., DualMap, febrero de 2026: https://arxiv.org/abs/2602.06502.
Ricci Toniolo et al., GORGO, junio de 2026: https://arxiv.org/html/2602.11688.
Yoshimura, Chiba y van de Beek, Accuracy Is Speed (EuroMLSys ‘26), abril de 2026: https://arxiv.org/html/2604.15732v1.

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

Thu, 10 Sep 2026 08:00:00 +0200

Fourth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability and virtual keys covered governance. Here comes the new tenant: agents, which speak the same protocol as humans and behave in a different way. Everything that follows is verified against LiteLLM 1.100.1 and vLLM 0.29.0, both from the first ten days of September 2026.

TL;DR

The day a team connects an agent to the corporate gateway, the platform changes regime without anyone touching a YAML. These are the seven conclusions.

LiteLLM’s priority parameter prioritises nothing under normal conditions. It is still marked as beta and its queue is only consulted when all the deployments in the group are in cooldown. With free capacity, the only thing it produces is a response header. On top of that, the success branch does not pop the item off the queue, so the list grows indefinitely; and on /chat/completions the value 0, which is the one used by all the documentation examples, is falsy in Python and disables queuing.

Separating traffic is done with two pools, not with priorities. The non-forgeable way is model_group_alias at key or team level: the client always asks for qwen-30b and the credential decides whether that resolves to qwen-30b-interactivo or to qwen-30b-agentes. The way that does not touch clients is tag_regex against the User-Agent, with a pattern like ^User-Agent: claude-code\/, which serves to classify load and not to control access, because that header is set by the client.

The useful admission control arrives in the 1.101 branch, with max_in_flight_requests_per_worker and a per-process queue that returns 503 with retry-after. It rejects before authenticating, so what it discards does not appear attributed to any key.

max_parallel_requests is only applied at virtual key level. The team, user, end customer and organisation endpoints accept the field and store it in the database, and nobody reads it. global_max_parallel_requests is a no-op unless the old limiter is reactivated, and general_settings.max_parallel_requests is read by no code path.

vLLM with --scheduling-policy priority does not push ahead anyone who is already generating. The only thing that changes is who it evicts when KV cache runs short. If the agents have filled max_num_seqs, a chat with the highest priority waits just the same. And the two knobs that in 2025 allowed short prompts to get ahead of long ones, max_num_partial_prefills and max_long_partial_prefills, disappeared from the code in 0.27.

The retry product reaches 45 calls per agent turn. Three from the client SDK times three from the router times up to five fallback groups. The proxy sets max_retries=0 on the SDK it uses itself, but it cannot touch the client’s. The timeout is per attempt, not per request, and LiteLLM’s Timeout comes out as a 408, which is retryable.

A single 429 takes a deployment out of the pool for five seconds, without going through the allowed_fails threshold. With an agentic burst against a fleet of four replicas, the whole pool enters cooldown and the gateway returns a 429 of its own to everybody, humans included.

You are here: the gateway layer with two classes of traffic

In the seven-layer stack this still lives in the gateway layer, with one difference compared to the previous three articles. There the variable was load: more requests, more teams, more budget to share out. Here the variable is the shape of the load.

A gateway sized for three hundred engineers using a chat works. The same gateway, with thirty of those engineers running coding agents, serves a fraction of the human traffic with the previous latency and the rest degrades. The GPUs stay just as busy, the utilisation metrics stay good, and the support team gets complaints from people waiting nine seconds for a paragraph.

The analogy: the registry desk and the courier with two hundred files

A registry office serves people in order of arrival. Citizens come in one at a time, present a piece of paper, and leave. The system is fair and the queue moves.

One Tuesday a courier turns up with a trolley of two hundred files. He joins the queue like anyone else, because rules are rules, and when his turn comes he occupies the desk for forty minutes. Behind him are twelve people with one piece of paper each. Nobody has broken any rule. The courier has the same right as everyone else and uses the same procedure. What fails is that the office was designed assuming a certain size of transaction and another one three orders of magnitude bigger has turned up.

The real solutions for an office are three, and all three have an exact equivalent in the platform. Open a desk for bulk deliveries, which is separating pools. Put up a capacity sign and send people back later, which is admission control with its 503. And say that files are handed in during the afternoon, which is the part that on an inference platform can almost never be applied, because the agent does not negotiate schedules.

What does not work in the office is giving the citizen a priority number when the courier is already at the desk. That number does not interrupt the transaction in progress. It is, literally, what priority does in the two layers of the platform, and it is worth keeping in mind while reading the next section.

The two kinds of traffic, measured

Before configuring anything, you have to be able to state the difference in numbers, because every later adjustment comes out of it.

Trait	Human chat	Agentic loop
Input per request	hundreds of tokens	tens of thousands, and it grows with each turn
Output per request	hundreds of tokens	from two tokens (a tool call) to several thousand
Concurrency per user	1	3 to 20 requests in flight
The metric that hurts	time to first token	total turn throughput
Tolerance for waiting	seconds	minutes
Cancelled requests	rare	routine, by the agent’s own decision
Retries	the browser’s	the SDK’s, plus the loop’s
Witness when it fails	the user	nobody

Two rows deserve comment because they contradict intuition.

The first is the metric row. An agent does not perceive time to first token, so on a shared engine it is the natural candidate to give up that metric. The operational conclusion is that the agent pool can be configured for aggressive throughput, with large batches and deep queues, while the interactive pool is configured for the opposite.

The second is the cancellations row. An agent that launches three speculative continuations and discards two generates two aborted streams per turn. At the gateway, that touches spend accounting, the release of concurrency counters and the open connection to the engine. LiteLLM 1.100 handles it reasonably well, and it did not six months ago.

What does not work: `priority` in LiteLLM

The first reaction of any operator is to look for a priority field. LiteLLM has one, documents it on its Request Prioritization page, and that page carries the beta label and the sentence saying it is for testing. The label is correct and has to be taken literally, because behind it there are four things.

The queue is only consulted when the whole group is in cooldown. The scheduler’s poll() method returns true if there is any healthy deployment, without touching the queue. With available capacity, which is the normal situation, the request goes straight through and priority orders nothing. The only thing that changes is that the response carries the x-litellm-request-prioritization-used header.

The queue does not empty. In that success branch there is no heappop and no call to the cleanup, which only happens in the expiry branch. The list under the key scheduler:queue:{model} grows without bound, is serialised whole into Redis on every insertion, and there is no maximum size and no discard policy. When the group does enter cooldown, the head of the heap is occupied by fossil entries from requests that finished hours ago, and new ones compare their identifier against that head, never match, and wait until the timeout runs out.

On /chat/completions, the highest priority disables the feature. The code reads the value with a kwargs.get("priority") or self.default_priority, and in Python 0 or None is None. Since in this scheduler the lowest number is the highest priority, priority: 0, which is the value in the documentation’s own examples, goes down the path without queuing. On /completions the check is done with is not None and it works; the two routes do not behave the same.

And default_priority in router_settings breaks the request. If the client does not send a priority but the global setting is in place, the value is truthy, the scheduling branch is entered and a function is invoked that requires the positional argument nobody has injected. The setting has no associated test and does not appear in the router_settings reference.

There is also a /queue/chat/completions, marked as experimental and hidden from the OpenAPI schema, which reads data["priority"] with no default value: a request without that field raises an exception that comes out as a 400 authentication error, with a message that has no relation to the cause.

There is a priority feature that is solid, and it is a different thing. The v3 dynamic limiter shares out token and request per minute quota between classes with weights, through litellm.priority_reservation with values like {"premium": 0.75, "standard": 0.25}. It shares out capacity, it does not order a queue: below the saturation threshold it lends free capacity and above it applies the weights. It never makes anyone wait, either it passes or it returns a 429. It requires Postgres and an enterprise licence, and it is still marked as beta.

What does work: two pools

Effective separation is done earlier, by deciding which set of replicas each class of traffic goes to. LiteLLM offers two mechanisms with very different security properties.

By credential: `model_group_alias` at key or team level

Router settings are resolved in the order key, team, global. Among those that accept that override is model_group_alias, and there lies the piece:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 model_info:
 id: chat-01
 max_input_tokens: 32768
 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 model_info:
 id: agentes-01
 max_input_tokens: 131072

And on the agent team’s key:

curl -X POST "$PROXY/key/generate" \
 -H "Authorization: Bearer $MASTER_KEY" \
 -d '{
 "team_id": "plataforma-agentes",
 "models": ["qwen-30b"],
 "router_settings": {
 "model_group_alias": {"qwen-30b": "qwen-30b-agentes"}
 }
 }'

The client asks for qwen-30b and knows nothing about any of it. The resolution is tied to the credential, so the caller cannot get around it by changing a header. The same list allows routing_strategy, fallbacks, context_window_fallbacks, retry_policy, cooldown_time and allowed_fails to be set per key or team, which gives different reliability policies for each class of traffic on the same proxy instance.

It should not be confused with access groups (model_info.access_groups), which are permissions and not routing: they decide whether a key can invoke a model name, not which replica the request goes to.

By `User-Agent`: `tag_regex`

When separate credentials cannot be issued, because the agentic client uses the same personal key as that person’s chat, LiteLLM classifies by regular expression against the request headers:

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 tag_regex: ["^User-Agent: claude-code\\/", "^User-Agent: .*codex"]
 model_info: {id: agentes-01}
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 tags: ["default"]
 model_info: {id: chat-01}

router_settings:
 enable_tag_filtering: true
 tag_filtering_match_any: true

The proxy composes strings of the form User-Agent: <value> and tests them with re.search. Exact tag matching takes precedence over the regex. The documentation itself carries a warning that has to be respected: the User-Agent is written by the client, so this is traffic classification and not a security boundary. As load sharing it is enough, because an agent that lies about its identity to sneak into the interactive pool is a governance problem, not a routing one.

The behaviour when no deployment carries the requested tag has a nuance that comes as a surprise in production. If the tag is unknown to the group, the request falls through to the pool marked with tags: ["default"]. If the tag exists in the group but no healthy deployment carries it at that moment, the request fails with the tag configuration error, even though a default pool is available. To get deterministic fallthrough to the default pool you have to set model_info.allow_fail_open: true.

A different strategy per pool with `routing_groups`

router_settings:
 routing_strategy: simple-shuffle
 routing_groups:
 - group_name: interactivo
 models: [qwen-30b-interactivo]
 routing_strategy: simple-shuffle
 - group_name: agentes
 models: [qwen-30b-agentes]
 routing_strategy: simple-shuffle

The recommendation to leave simple-shuffle on both is not laziness, and sustaining it requires looking at what the alternatives do on a homogeneous vLLM fleet.

least-busy keeps a complete dictionary under a single key and updates it with non-atomic reads and writes, without using INCR, so with several proxy replicas increments are lost. It has no TTL, so a request that never completes leaves the counter raised forever and that replica is permanently excluded. Counters can go negative. And when the minimum corresponds to an identifier that is no longer in the healthy list, it falls back to random.choice without warning. The pattern of requests that do not complete is exactly that of an agent that cancels.

latency-based-routing measures seconds per output token, not wall-clock latency. A chat turn of twenty tokens looks extremely slow by that metric and an agent generation of two thousand tokens looks extremely fast, so mixing both kinds of traffic in the same group poisons the signal. It also seeds new deployments with zero latency, and with the buffer defaulting to zero, a freshly started replica takes all the traffic until its first success callback.

usage-based-routing-v2 uses minute windows that reset all at once, and it accounts for tokens when the request completes, so long generations in flight are invisible to the router. With long outputs, it systematically underestimates the real load.

cost-based-routing sorts by the sum of the input and output unit prices and takes the first, with no random tie-break. With replicas of the same model at the same price, all the traffic always goes to the same place. And a model that is not in the price map gets a default cost of 5.0 per token, a figure designed to deprioritise it.

Admission control: the new piece in the 1.101 branch

When the agent pool saturates, what you want is to reject fast and with a code the client understands, not to accumulate requests inside the proxy until memory runs out. That arrives with per-worker admission control, added in the 1.101 branch, in release candidate as this is written.

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

It is a semaphore with a per-uvicorn-process queue, without Redis. When the queue fills or the wait expires it returns 503 with retry-after: 1 and a body with "type": "overloaded_error". It is observed through /health/backlog, which exposes requests in flight, admitted, queued and rejected, and through the Prometheus litellm_admission_* metrics, whose rejection counter distinguishes queue_full from queue_timeout.

Two properties have to go into the runbook. The middleware rejects before authenticating, so what is discarded is not attributed to any key in the spend records, and the real number of requests rejected per team cannot be reconstructed from the spend table. And the limits are read on the first request: changing them requires a restart.

Concurrency per key, and the three settings that do nothing

max_parallel_requests is the right brake for a specific agent, and its configuration surface is full of dead ends.

The only level where it is applied is the virtual key. The v3 limiter, active by default since 1.94, has descriptors for user, team, team member, end customer, organisation, model per key, tag and agent, and all of them carry only request and token counters per window. The only reference to max_parallel_requests is the one in the key descriptor. The old limiter says so in a comment: supporting it for model, user and team is pending.

At the same time, /team/new, /user/new, /customer/new and the organisation endpoints accept the field, validate it and write it to Postgres, and the documentation shows it in their examples. Nobody reads it afterwards. A team with max_parallel_requests: 20 in its row has no concurrency limit at all.

global_max_parallel_requests is injected into the request metadata and is only consulted by the old limiter, which is disabled unless LEGACY_MULTI_INSTANCE_RATE_LIMITING=true is set. The server settings documentation still describes it as a global limit coordinated through Redis. And general_settings.max_parallel_requests does not appear in any execution path: its only two appearances are the copy from the database and the type schema of the interface.

What does work, in detail:

curl -X POST "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" \
 -d '{"team_id":"plataforma-agentes","max_parallel_requests":8,"rpm_limit":600}'

Counting is done with a sorted set in Redis scored by Redis’s own clock, with Lua scripts to acquire and release. Each slot expires after 3,600 seconds, which is at once the maximum request duration the meter can track and the time a leaked slot takes to heal by itself. Release is hooked to success, to failure and to client disconnection during streaming, which was the classic leak with clients that cancel.

One live leak remains and it affects exactly the case in this article: every tool call through the MCP gateway acquires a slot and does not release it. The MCP route invokes the pre-call hook, which acquires, and none of the release functions is called from that module. With an agent that makes twenty tool calls per turn and a key with eight slots, the 429 arrives on the first turn and lasts an hour.

The order of evaluation matters for interpreting 429s. First the window counters, requests and tokens per minute, are checked, and only if they pass is the concurrency slot acquired. The code comment explains why: the other way round, every rate limit rejection would leave an orphan slot. In practice, with a low rpm you always see the requests type 429, and the max_parallel_requests one only appears when the relationship between concurrency and duration makes it binding first. The message distinguishes the two cases with the Limit type: field and the rate_limit_type header.

A minor detail with consequences on the client: the response to a concurrency rejection carries retry-after and reset_at computed from the window size, sixty seconds by default, when a concurrency meter has no window. A well-behaved client that respects that header will wait a minute when it might have had a free slot in two seconds.

Session affinity: the KV cache that gets thrown away

An agent turn resends the whole conversation context, system prompt and tools included. If that request lands on a different replica from the previous turn’s, vLLM’s prefix cache does not have the prefix and the prefill is recomputed in full. With twenty turns and four replicas balanced blindly, the long prefill is paid for fifteen times over.

LiteLLM has two mechanisms for this and they are not equally good.

The prompt_caching check computes a SHA-256 of the cacheable prefix, defined as everything up to the last block with cache_control, and pins the deployment with that hash. It has two problems. The TTL is hardcoded to 300 seconds in two places in the code, with no way to configure it, something which is reported and open; with one-hour caches, affinity is lost after five minutes of inactivity. And the key is an exact hash, so a client that moves the cut point forward on every turn, which is exactly what agents do to extend the cached prefix, generates a new key every time.

The session_affinity check is the one to use:

router_settings:
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

It pins the deployment with an atomic Lua script in Redis, with a configurable TTL that defaults to one hour and that is refreshed on every request, so it bounds the idle time between turns and not the duration of the conversation. If the pinned deployment enters cooldown, the request goes out through the normal strategy and keeps the pin so it can come back when it heals.

The session identifier comes from a precedence chain that starts with x-litellm-trace-id, continues with x-litellm-session-id and then accepts any header of the form x-<something>-session-id whose value looks like an identifier. That includes the session header Claude Code already sends, with nothing configured on the client. Whatever matches populates both the session identifier and the trace identifier, which links this to the correlation from the first article in the track.

One dangerous interaction when combining it with tags: affinity is evaluated before tag-based routing. If the pin reduces the list to a deployment that does not satisfy the request’s tag, the request fails instead of falling through to the default pool, unless allow_fail_open has been enabled. Retagging a pool with live sessions breaks them.

And a clarification that saves a pointless search: LiteLLM does not do prefix routing. There is no prefix tree and nothing that directs the request to the vLLM replica that already has that prefix warm. What there is, is affinity by session or by key. For the other thing you need a specific router underneath the gateway, and that architecture has its own article pending.

The engine: what vLLM can and cannot do

Everything above shares out requests between engines. Inside one engine, the question is what happens when a twenty-token request and a sixty-thousand-token one live side by side.

Priority exists and does less than it looks like. --scheduling-policy priority turns the waiting queue into a heap ordered by priority, arrival time and identifier, with the lowest number first. The request can carry the priority field in the body, and there is also the X-Vllm-Priority header, which takes precedence over the body and is not documented anywhere.

What priority does not do is push ahead of whoever is already generating. In the running queue loop, when KV cache runs short, the policy only changes who is chosen as the victim of eviction: with priority the one with the worst priority and arrival time is evicted, and with fcfs the last in the queue. In the waiting queue loop, if there are no blocks available the iteration is cut short, without evicting anyone. And the max_num_seqs cap is a hard cut. If the agents have filled that cap, a chat with maximum priority waits just the same. There has been an open issue since April and an unmerged pull request about this.

That nuance about the victim is not negligible, because with fcfs the victims are the most recently admitted, that is, the chats that have just come in. Switching the priority policy on already improves the situation even if it pushes nobody ahead.

Eviction is always by recomputation. The swap-to-host-memory mode and the enumeration that controlled it disappeared from the code; all that is left is freeing blocks and returning the request to the queue with the computed token counter at zero. An evicted request loses all its prefill, and the prefix cache recovers part of it, but the perceived time to first token restarts.

Two admission settings help and do not appear in the narrative documentation. watermark reserves a fraction of free KV blocks on admission, with the declared purpose of avoiding repeated eviction, and it ships disabled. And scheduler_reserve_full_isl, active by default, checks that the full input length fits in the cache before admitting, instead of looking only at the first chunk.

Chunked prefill is active by default and prioritises decode. The scheduler walks the whole running request loop before touching the waiting queue, so a long agent prefill does not freeze a chat’s generation in progress. That protects the inter-token latency of what has already been admitted, and it does not protect the time to first token of what is waiting.

Here there is a breaking change that forces a review of the manifests. max_num_partial_prefills and max_long_partial_prefills were removed in 0.27. They were precisely the mechanism that allowed short prompts to get ahead of long ones. A deployment with those flags in its args does not start. long_prefill_token_threshold still exists, and its meaning changed: today it is a per-request chunk size cap, not a classification of long requests.

The partial replacement arrives in 0.29 with max_num_queued_tokens and max_num_queued_reqs, admission valves that return 503 when the pending prefill work exceeds the target. The code itself frames it as a quality of service mechanism for time to first token, with the formula of setting it to the latency target multiplied by prefill throughput. They are blind to the class of traffic: they reject a chat just the same as an agent.

And the default value of --max-num-batched-tokens depends on the hardware and on the start-up mode. With vllm serve on H100 or H200 it is 8,192, on cards of 160 GiB or more it is 16,384, and on the rest 2,048. The 2,048 that appears in the scheduler configuration only applies to tests.

The metrics for diagnosing coexistence, with the exact names from 0.29:

Metric	What for
`vllm:num_requests_waiting`	queue depth, the autoscaling signal
`vllm:num_requests_waiting_by_reason`	separates waiting for capacity from waiting in a blocked state
`vllm:num_preemptions_total`	evictions, the signal that the two classes are treading on each other
`vllm:kv_cache_usage_perc`	cache pressure, the cause of the evictions
`vllm:request_queue_time_seconds`	what a chat waits with the agents inside
`vllm:prefix_cache_hits_total` and `_queries_total`	measured in tokens, not in requests

A warning for anyone with old alerts. The log warning about evicted sequences that appears in the optimisation documentation belongs to the previous version of the engine and no longer exists in the code. Preemptions are only visible as a counter. An alert based on searching for that string in the logs has not fired for months, and not because there are no evictions.

On isolation between classes inside one engine, the short answer is that it does not exist. There are two queues and neither is per class of traffic. The conclusion, which is the one that holds up this article’s architecture: effective isolation means separate fleets, and on top of them the gateway deciding who goes to which.

Retries: the product nobody calculates

A human who gets an error sees it and decides. An agentic loop retries, and beneath it there are two more layers that also retry.

Layer	Default
Client SDK	2 retries, 3 attempts
LiteLLM router	2 retries, 3 attempts
Fallbacks	up to 5 groups

The product is about 45 calls to the engine for one agent turn, before counting the loop’s own retry. LiteLLM does what it can: when the call comes from the router, it forces max_retries=0 on the SDK it itself uses against the engine. What it cannot touch is the client’s SDK. The only layer that breaks the product is that one, and it is broken by setting max_retries=0 on the client and delegating the retry to the gateway.

The retryable errors are 408, 409, 429 and anything 500 or above. And there lies the problem with long reasoning: litellm.Timeout is a subclass of OpenAI’s timeout exception and comes out with code 408, which is retryable, so the most expensive request of all, the one that expired after ten minutes of generating, is launched again in full. Twice by the router, and as many times again by the client.

On timeouts you have to be precise because the code comment is: the configured timeout is per attempt, not per request. A request_timeout: 600 with two retries and two fallback groups gives a worst case of almost two hours before the client receives anything. The default value is still 6,000 seconds, as was already covered in the day 2 article.

The retry storm has two specific causes and both are default settings.

The first: a single 429 takes the deployment out of the pool, without going through the allowed_fails threshold. The cooldown logic has an explicit branch which, on a 429 in a group with more than one deployment, returns immediate cooldown. With an agentic burst against four replicas, all four leave the pool at once and the gateway answers everybody with its own 429 of “no deployments available”. The default cooldown is five seconds, which in a sustained burst gets renewed.

The second: backoff is zero while the pool looks healthy. The function that computes the wait returns 0 if the healthy deployment list is not empty. Exponential backoff with jitter, capped at eight seconds, only kicks in when there is nothing healthy left. Under partial degradation, the proxy hammers away without waiting and only slows down once everything has already fallen over.

On top of that, the engine’s Retry-After header is only respected if its value is between 1 and 60 seconds. A longer limit window is ignored and replaced by the proxy’s own backoff.

The settings that cut the problem off:

router_settings:
 num_retries: 1
 retry_after: 2 # real floor; the default 0 is hammering
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3 # stops a 429 taking the deployment out
 retry_policy:
 TimeoutErrorRetries: 0 # do not relaunch the expensive request
 AuthenticationErrorRetries: 0
litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false

That last setting deserves an explanation, because it is a silent way of breaking an agent. With drop_params: true, LiteLLM removes from the body any parameter the destination provider does not support, and in that branch it writes no warning at all. The candidates for disappearing are tools, tool_choice, response_format and parallel_tool_calls. An agent that has tools taken away receives free text where it expected a function call, and interprets it as the model having decided not to use tools.

Worse still is prompt-based emulation. For a non-OpenAI-compatible provider that does not support tools, LiteLLM does not fail: it turns the tools into text inside the prompt, forces JSON format, discards tool_choice, and on the way switches on a process-global variable that makes any other tool-less provider in the same process go to emulation instead of erroring.

With that in mind, the warning about fallbacks is clear: LiteLLM does not check capabilities before falling back. The function that knows whether a model supports function calling exists and is only used in an informational endpoint. The pre-call checks filter by context window, request limit and region, and nothing else. A fallback list that goes from a model with tools to one without them degrades the agent silently. There are guards for provider-scoped resources, files and batches, so the absence of the capability guard is an omission and not a design oversight.

A month’s budget in twenty minutes

A runaway agent consumes in minutes what a team consumes in weeks, and the operational question is how long the gateway takes to find out.

The answer has changed, and the old mental model, the one where spend consolidates every minute and until then there is no brake, is no longer correct. Since 1.84 there are two new pieces. An optimistic reservation before the call, which estimates the maximum cost of the request and reserves it atomically in the counter before heading out to the engine, reconciling it with the real cost on completion. And a counter increment after the call that is explicitly awaited, with the comment that this way the counter is up to date before the next request goes through authentication. Budget checks read that counter, with Redis first and seeding from the database when cold.

Batch writing to Postgres still exists, every 10 to 15 seconds, and it affects persistence and what is shown in the interface, not the cut-off. The production documentation’s recommendation to raise that interval to 60 does not relax the control, and it does delay by a minute what /key/info shows, something which frequently generates operational panic. And the budget rescheduler’s ten minutes do not consolidate spend: they fire the job that resets expired budgets.

The overspend window then comes down to the cost of the requests in flight, provided there is Redis. Without Redis, the counters are per pod, and with N replicas you can spend up to N times the budget. That is the configuration failure that matters and it is the same Redis that the day 2 article declared non-optional for other reasons.

Two settings that do not appear in the public documentation, only in the data model descriptions, and that decide behaviour at the worst moment:

general_settings:
 fail_closed_budget_enforcement: true # false by default: if the counter fails, it lets things through
 # disable_budget_reservation: do not set it to true

With the default value, if the counter backend does not respond, the proxy fails open and carries on serving. For an agent in a loop that is exactly the opposite of what you want.

Time for the honest warning: this route is still unstable. There are open issues in both directions, one where the key and user budget limit stops being applied while spend is recorded, and another where a 429 is returned for stale spend in the reservation counters while the key query shows spend below the limit. Project budgets are not on the atomic reservation route and concurrency overshoots them. The practical conclusion is not to trust the hard cut-off to the budget alone.

On the manual cut-off, there is an asymmetry to know about before you need it. /key/block invalidates the local cache, the Redis one, and notifies all workers by publish, so with Redis the effect is almost immediate. /customer/block does the same. /team/block only writes to the database, without invalidating or notifying, so it takes however long the management object cache TTL takes, sixty seconds by default. To cut a team off dead you have to block its keys one by one, or lower user_api_key_cache_ttl. And /user/block no longer exists, even though the docstring of the function that replaced it still shows that route in its example.

On alerts, two numbers. The default thresholds warn at 85 % and 95 % of the budget. And deduplication of those alerts is 24 hours per event and identifier, so crossing 85 % generates one warning and then silence for a day. For agentic traffic, where 85 % and 100 % can be twenty minutes apart, the second warning arrives when it is no longer any use.

Attribution: who the user is when the caller is an agent

With agents, the question of who the spend is charged to has two answers and both are needed: the agent that executes and the human on whose behalf it acts.

The chain that extracts the end customer identifier tries the standard headers first, then the configurable header mapping, and only then the user field of the body, the litellm_metadata.user of the Anthropic format, the metadata.user_id and the safety_identifier. The last four are body fields and are therefore written by the client. The code itself warns about it: for untrusted callers the identifier has to be set from headers or from a middleware on the server, to avoid impersonation.

Out of that comes the correct pattern, which is the inverse of the one people put together out of inertia: the virtual key identifies the agent and is the non-forgeable part, and the header or the user field identify the human and serve for attribution, not for control. An agent that lies about the human behind it is still bounded by the budget and the limits of its key.

The spend table stores user, end_user, team_id, organization_id, request_tags, session_id and agent_id, with daily aggregation per agent, so the detail is available once it is sent. The x-litellm-agent-id header is what populates that last field.

A note about the MCP gateway, which on a platform with agents ends up being the other entry door. Tool calls do appear in the spend records, with call types of their own and a metadata block that stores tool name, arguments, result and server. What is missing is token counting: the cost is 0.0 by default and is only populated if the administrator configures a price per query, per server or per tool. It is the same pattern of silent zero cost that already came up with unpriced models in the first article in the track.

And the permission setting to review on day one: access control for MCP servers is open by default. If no level of the hierarchy defines a list, the request reaches every configured server. general_settings.require_key_mcp_access_defined: true inverts that inheritance, and it is not the default value.

A reference `config.yaml`

All of the above, together, on a fleet split into two pools:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018 # without this, silent zero cost
 output_cost_per_token: 0.00000072
 model_info:
 id: chat-01
 max_input_tokens: 32768 # without this there is no context filter

 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072
 tag_regex: ["^User-Agent: claude-code\\/"]
 model_info:
 id: agentes-01
 max_input_tokens: 131072
 allow_fail_open: true

router_settings:
 routing_strategy: simple-shuffle
 enable_pre_call_checks: true
 enable_tag_filtering: true
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600
 num_retries: 1
 retry_after: 2
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3
 retry_policy:
 TimeoutErrorRetries: 0
 AuthenticationErrorRetries: 0

general_settings:
 fail_closed_budget_enforcement: true
 require_key_mcp_access_defined: true
 user_api_key_cache_ttl: 10
 proxy_batch_write_at: 10
 # 1.101 branch onwards:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false
 cache: true # Redis, not optional

And the start-up of the two engines, with the deliberate asymmetry:

# interactive pool: protect the first token
vllm serve Qwen/Qwen3-30B \
 --scheduling-policy priority \
 --max-num-seqs 64 \
 --max-model-len 32768 \
 --max-num-batched-tokens 4096

# agent pool: throughput and deep queues
vllm serve Qwen/Qwen3-30B \
 --max-num-seqs 256 \
 --max-model-len 131072 \
 --max-num-batched-tokens 8192

Checklist

Issue separate keys for agents, with model_group_alias at key or team level. It is the only separation the client cannot evade.
Add tag_regex over the User-Agent as a safety net for agents that use personal credentials, with a default pool and allow_fail_open: true.
Leave simple-shuffle on both pools. The other four strategies have verifiable defects on homogeneous replicas.
Enable session_affinity with a one-hour TTL. Do not use prompt_caching for this while its TTL stays hardcoded.
Declare max_input_tokens on every deployment. Without that value, the context window filter is not applied and it fails silently for models that are not in the price map.
Declare per-token prices on every self-hosted model, or the agents’ spend will be zero and their budget will never run out.
max_parallel_requests on the agent’s key, never on its team. On the team it is stored and not applied.
Lower num_retries to 1, put a floor on retry_after, raise cooldown_time and disable retry on timeout. On the client, max_retries=0.
fail_closed_budget_enforcement: true and mandatory Redis. Without Redis, N replicas are N times the budget.
Start the interactive engine with --scheduling-policy priority even though it pushes nobody ahead: it changes who it evicts.
Alert on vllm:num_preemptions_total and vllm:num_requests_waiting_by_reason, not on log strings that are no longer emitted.
Prepare the cut-off: block by key, never by team, and verify the real propagation time in your own deployment.

Traps and things that are not what they seem

priority: 0 on /chat/completions disables prioritisation. It is the value in the documentation examples and it is falsy in Python.
max_parallel_requests on a team is stored and not applied. The field exists in the API, in the database and in the documentation; the limiter only reads it from the key descriptor.
global_max_parallel_requests is a no-op with the default limiter, and the server settings documentation still describes it as active.
Every MCP call leaks a concurrency slot, which only heals when its hour expires. With agents calling tools in a loop, the 429 appears early and lasts a long time.
The TTL of prompt cache affinity is 300 hardcoded seconds. For agent conversations, session affinity is the right option.
Affinity is evaluated before tag-based routing. A pin to a deployment that stops satisfying the tag makes the request fail instead of falling through to the default pool.
LiteLLM’s timeout is per attempt. The worst case is the timeout multiplied by attempts and by fallback groups.
A 429 takes the deployment out of the pool without going through allowed_fails, and backoff is zero while anything healthy remains.
The engine’s Retry-After is ignored if it asks for more than 60 seconds.
drop_params: true removes tools and response_format without leaving a warning, and the fallback does not check whether the destination model supports function calling.
/team/block does not invalidate the cache. It takes the full TTL; /key/block does propagate by publish in Redis.
Access to MCP servers is open by default if no level defines a list.
max_num_partial_prefills and max_long_partial_prefills no longer exist in vLLM. A manifest with those flags does not start from 0.27 onwards.
vLLM’s priority does not push ahead of running requests, and with max_num_seqs full it pushes nobody ahead.
The eviction log warning that appears in the vLLM documentation is no longer emitted. Only the Prometheus counter is left.

Closing

The coexistence of humans and agents on the same platform is not solved with a priority field, and that is the conclusion that takes the most effort to accept because the field exists and looks made for this. Neither the gateway nor the engine has a usable notion of class of service today: one orders a queue that is almost never consulted, the other picks who to evict. What they do have is the ability to direct each request to a different place, and that is where the win is.

The architecture that comes out of all of the above fits in one sentence. Two vLLM fleets with opposite parameters, a credential that decides which one each request goes to, session affinity so as not to throw away the prefill work, concurrency limits on the agent’s key and not on its team, retries trimmed back on the three layers where they multiply, and budgets that fail closed.

The rest is measurement. If vllm:num_preemptions_total rises in the interactive pool, the two classes are treading on each other despite the separation. If chat queue time grows while total throughput holds, the partition is badly sized. And if the agents’ spend comes out as zero, the problem is not coexistence but undeclared prices, which is where this track started.

Sources

LiteLLM, [BETA] Request Prioritization: https://docs.litellm.ai/docs/scheduler.
LiteLLM, Router - Load Balancing (estrategias, afinidad de sesión, comprobaciones previas): https://docs.litellm.ai/docs/routing.
LiteLLM, Tag Based Routing (incluye tag_regex y el ejemplo con clientes agénticos): https://docs.litellm.ai/docs/proxy/tag_routing.
LiteLLM, Router Settings for Keys and Teams (resolución clave, equipo, global): https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.
LiteLLM, Server Tuning (control de admisión por worker): https://docs.litellm.ai/docs/proxy/server_tuning.
LiteLLM, Dynamic TPM/RPM Allocation (reserva de prioridad, beta y enterprise): https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.
LiteLLM, Budgets, Rate Limits: https://docs.litellm.ai/docs/proxy/users.
LiteLLM, Production Best Practices: https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Reliability (fallbacks, ventana de contexto, enfriamientos): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, código: litellm/scheduler.py, litellm/router_strategy/tag_based_routing.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/hooks/parallel_request_limiter_v3.py, litellm/proxy/middleware/admission_control_middleware.py, litellm/router_utils/cooldown_handlers.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de la afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en cada llamada MCP: https://github.com/BerriAI/litellm/issues/34534.
LiteLLM, incidencia 26672, presupuesto de clave y usuario no aplicado: https://github.com/BerriAI/litellm/issues/26672.
LiteLLM, incidencia 872, comprobación previa de soporte de llamada a funciones, cerrada sin implementar: https://github.com/BerriAI/litellm/issues/872.
vLLM, Optimization and Tuning (prefill troceado, prioridad de decodificación): https://docs.vllm.ai/en/v0.29.0/configuration/optimization.html.
vLLM, Metrics design (nombres, convención _total, autoescalado como problema abierto): https://docs.vllm.ai/en/v0.29.0/design/metrics.html.
vLLM, Data Parallel Deployment (caché KV independiente por réplica): https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment.html.
vLLM, código: vllm/v1/core/sched/scheduler.py, vllm/v1/core/sched/request_queue.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/metrics/loggers.py: https://github.com/vllm-project/vllm.
vLLM, incidencia 40004, la prioridad no desaloja peticiones en ejecución: https://github.com/vllm-project/vllm/issues/40004.

LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

Tue, 08 Sep 2026 07:00:00 +0200

Second article in the operational track of the control layer. The operational pair with Langfuse covered the seam between the gateway and observability; here everything stays inside the gateway. The prior decisions, which gateway and why, are in choosing the OSS gateway and in the L7 inference router.

TL;DR

LiteLLM Proxy starts from a twenty-line config.yaml and serves traffic that same afternoon. The operational work consists of correcting seven defaults designed for a scenario other than an in-house inference factory.

One uvicorn worker per pod. The project’s production guide asks for --num_workers 1 on Kubernetes and horizontal scaling, with 1 vCPU and 4 GiB per worker set at the same time as requests and as limits. The 4 GiB are a floor, not a target: the Prisma query engine marks a resident memory high-water mark that grows up to the largest statement it has executed and that glibc does not return to the system.

Autoscale on CPU, never on memory. A direct consequence of the above. Memory goes up and never down, so an HPA on memory scales and never scales back. The recommended target is targetCPUUtilizationPercentage: 60.

The Postgres connection arithmetic. database_connection_pool_limit is 10, and the number of connections is instances times workers times the pool. The Helm charts ship maxReplicas: 100, on the order of a thousand connections, far above what a stock Postgres accepts.

Redis stops being optional from around a thousand requests per second or ten instances. Without it, each instance applies its limits on its own, caches are not shared, and spend updates against the same rows produce locks and exhaust the connections with FATAL: sorry, too many clients already.

Three health endpoints and only two are usable as probes. /health/liveliness and /health/readiness require no authentication and answer fast. Bare /health demands a key and fires a real request against every model in the catalogue, with its cost in tokens.

The resilience numbers are 2, 3 and 5: two retries, three allowed failures and five seconds of cooldown, the last one despite the function’s own documentation saying the default is 1.

request_timeout is 6,000 seconds. A hundred minutes holding a connection against an engine that does not answer.

And a warning about the performance figures: the project publishes 8 ms of p95 overhead on one benchmark and 257.7 ms of p99 on another. Both are theirs, and the difference is in the shape of the load.

You are here: the gateway layer, the day after

In the seven-layer stack, this article lives entirely in the gateway layer. The difference with the selection article is the one that separates buying from maintaining: there the question was which piece to put in front of the fleet, and here it is what happens when that piece has been serving for six months, the team has tripled the traffic and someone has added an agent that fires twenty calls per interaction.

The gateway has an uncomfortable property worth keeping in mind: it sits on the critical path of every request and contributes not a single token. All it does is coordination, and any coordination that fails turns into an outage of the whole inference service, even with the GPUs perfectly healthy.

The analogy: the switchboard and its notebook

A hospital switchboard does three jobs at once. It puts calls through, which is the urgent part. It writes down in a notebook who called, to which extension and for how long, which is what sustains the billing. And it consults a list of extensions to know where to put each call through.

The three jobs compete. If the operator stops to write each line in the notebook before putting the next call through, the waiting queue grows. If there are three operators and each keeps her own notebook, the totals do not add up. And if the list of extensions is consulted in a filing cabinet on another floor, every call costs the trip.

The three tensions have an exact equivalent in LiteLLM. The notebook is the SpendLogs in Postgres, which is why the transaction buffer in Redis exists. The three operators with three notebooks are the pod replicas applying limits independently. And the filing cabinet on another floor is usage-based routing, which adds a Redis lookup inside the request path.

The process model: one worker per pod

The production documentation is explicit: on Kubernetes, --num_workers 1 and scale with replicas. On a virtual machine with no orchestrator, NUM_WORKERS equal to the number of vCPUs.

The reason is not ideological. An additional worker inside the same pod shares the container’s memory limit with the others, multiplies the database connections by the same factor, and registers a copy of every periodic job. With separate replicas, each one has its own resource budget, its own connection quota and a place where the scheduler can put it.

The recommended sizing is 1 vCPU and 4 GiB per worker, and the part that gets overlooked is that these go as requests and as limits at the same time. Setting less than 4 GiB produces an OOM restart loop that shows up late, once some large request has grown the query engine’s footprint.

resources:
 requests: { cpu: "1", memory: "4Gi" }
 limits: { cpu: "1", memory: "4Gi" }

That footprint needs to be described precisely, because it determines the autoscaling policy. The Prisma query engine keeps a resident memory that behaves as a high-water mark: it grows up to the largest statement the process has executed and does not come back down, because glibc does not return that freed memory to the system. A pod that served a spike yesterday still shows the footprint of that spike today.

Two rules come out of this. The first: autoscale on CPU and never on memory. An HPA on memory sees a metric that only goes up, scales at the peak and never scales back. The recommended target is targetCPUUtilizationPercentage: 60, lower than the 80 the charts ship by default, because the startup probe allows up to 300 seconds before passing the first readiness check and there needs to be headroom for the new pod to be ready before the saturated one goes down.

The second: bound the process lifetime. --max_requests_before_restart 10000 recycles the worker before the high-water mark matters.

Postgres: the arithmetic that blows up when you scale

database_connection_pool_limit is 10 by default. The total number of connections the deployment opens is replicas times workers times pool, and it has to be compared against the Postgres max_connections.

The formula the documentation gives is the inverse, and it is the one to apply when sizing:

database_connection_pool_limit = MAX_DB_CONNECTIONS / (instances × workers)

The detail that hurts in production is in the Helm charts, which ship autoscaling.maxReplicas and keda.maxReplicas at 100. With the default pool, that is on the order of a thousand connections at full scale-out, far above what an untouched Postgres accepts. And the moment the HPA reaches those replicas is exactly the moment of highest load, so connection exhaustion arrives when it can least be afforded.

There are two more valves on the same path. Provider errors are written to the database by default, and under a sustained engine failure that inflates the spend table without contributing anything that is not already in the metrics. They are switched off like this:

general_settings:
 disable_error_logs: true # stops writing provider errors
 proxy_batch_write_at: 60 # batches the spend writes

disable_spend_logs: true is the radical version, which removes the per-request detail from the interface and leaves the cost in Prometheus and in the tracing backend. Before going that far you have to decide whether that table is the system’s audit record, which is the discussion in the previous article: if it is, it cannot be switched off, and the way out is the Redis buffer.

Redis: when it stops being optional

Without Redis, LiteLLM works. Each instance keeps its own in-memory cache and applies its own counters. The consequences are three, and all of them show up as you grow.

Requests-per-minute limits are applied per instance, so a limit of 100 with five replicas is in practice a limit of 500. Cache hits are local, so the same repeated question hits once in five. And there is no leader election, so periodic jobs run in every process at once.

The threshold the documentation gives for enabling the transaction buffer is a thousand requests per second or ten instances. Below that, each instance updates the key, user and team rows directly; above it, they all write to the same rows, locks appear and Postgres starts rejecting with FATAL: sorry, too many clients already.

general_settings:
 use_redis_transaction_buffer: true

The metrics to watch when that is enabled are litellm_in_memory_spend_update_queue_size, litellm_redis_spend_update_queue_size and litellm_pod_lock_manager_size. If the first grows without the second coming down, the flush to Postgres is not keeping up.

A note about the routing strategy, which interacts with this. simple-shuffle consults nothing external. Usage-based routing does, and it adds a round trip to Redis inside the request path. On a homogeneous vLLM fleet behind the same model, the improvement it brings rarely pays for that latency; on a heterogeneous fleet, the conversation is different and the router article covered it.

Background jobs register per worker

This detail appears in no getting-started guide and produces considerable bewilderment when it is discovered. The proxy’s periodic jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every job, all of them running with no coordination if there is no Redis for leader election.

The variable that separates them is LITELLM_JOB_ROLE: value serving on the pods that serve traffic and a separate replica with value worker for the jobs. With the one-worker-per-pod recommendation the problem shrinks, but it does not go away while there are several replicas.

Probes: three endpoints and only two are usable

The three endpoints exist and do different things:

Endpoint	Authentication	What it does	What it is for
`/health/liveliness`	No	Returns `I'm alive!`, or 503 during shutdown	`livenessProbe`
`/health/readiness`	No	State of the process and the database. 503 if the configured Postgres does not answer	`readinessProbe`
`/health`	Yes	Fires a real request against every model in the catalogue	Manual diagnosis, never a probe

The last row deserves insisting on. /health spends tokens on every call, one per configured model. Set as a probe, with the Kubernetes default interval and a fleet of eight models, that is thousands of inference requests a day that serve nobody.

There are more endpoints useful for diagnosis: /health/readiness/details, /health/services?service=langfuse to check one specific callback, /health/history and /health/latest for the history, and /health/backlog, which returns the in-flight requests and shows up in the latency section. /health/drain exists but returns 404 unless it is enabled with enable_drain_endpoint: true, and it is protected with X-Drain-Token.

The periodic model checks are configured separately and not in the probe:

general_settings:
 background_health_checks: true
 health_check_interval: 300 # seconds
 background_health_check_model_groups: ["llama-70b", "qwen-30b"]

And one setting with consequences in deployments with no Internet egress: allow_requests_on_db_unavailable: true allows serving traffic with the database down, but leaves /health/readiness always returning 200. With that set, the readiness probe stops detecting the very failure it was designed to detect, and budget and model errors keep blocking just the same.

Retries, cooldowns and fallbacks: the real numbers

The defaults, read from the project’s code and not from the documentation:

Constant	Value	What it governs
`DEFAULT_MAX_RETRIES`	2	Retries before moving on to fallbacks
`DEFAULT_ALLOWED_FAILS`	3	Failures before cooling down a deployment
`DEFAULT_COOLDOWN_TIME_SECONDS`	5	Seconds a deployment stays out
`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD`	1000	Minimum requests before applying the cooldown logic with a single deployment
`DEFAULT_REQUEST_TIMEOUT_SECONDS`	6000.0	Request timeout

Besides the failure counter, there is a proportional criterion: a deployment enters cooldown if 50% of its requests fail in any given minute.

On the cooldown there is a discrepancy worth knowing about before debugging blind. The docstring of the router function says cooldown_time defaults to 1, and the code uses the constant of 5. The code wins.

The fallbacks are three separate lists and are applied after the retries are exhausted:

router_settings:
 num_retries: 2
 timeout: 120
 fallbacks: [{"llama-70b": ["qwen-30b"]}]
 context_window_fallbacks: [{"llama-70b": ["llama-70b-128k"]}]
 content_policy_fallbacks: [{"llama-70b": ["llama-70b-sin-guardrail"]}]

A note on the behaviour when everything is in cooldown: if no deployment is left available in the group, LiteLLM falls back onto a specific model_info.id, skipping the cooldown check. It is a deliberate degradation, serve something rather than return 503, and it is worth knowing it exists because it masks the real state of the fleet.

And a capability lost along the way: since version 1.85.0, the mock_testing_fallbacks parameter and its two siblings are stripped from requests arriving through the proxy and have no effect. Fallbacks can no longer be tested against the proxy with a doctored request; it has to be done against litellm.Router directly, in a test. Anyone who had a smoke test built on that has it broken without warning.

The six thousand second timeout

It deserves its own section because it is the setting with the worst damage-to-effort ratio in the whole system. request_timeout is 6,000 seconds. A hundred minutes. An inference engine that stops answering without closing the connection holds a worker for that long, and with one worker per pod that is an entire pod out of service per trapped request.

general_settings:
 request_timeout: 600

Ten minutes is still generous for a long generation, and it bounds the damage.

Overhead: the project’s two figures

LiteLLM publishes two measurements of its own overhead that differ by a factor of thirty, and both are legitimate because they measure different loads.

The benchmarks page, with Locust, a thousand users with think time and a fake endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms at 1,170 requests per second. With two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures correspond to around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, from July 2026, measures the Python proxy at 257.7 ms of added p99 and 329.5 MB of peak memory, with no callbacks, no spend tracking and no persistence. The Rust variant, in beta, gives 0.7 ms.

The reading for an in-house platform: the first figure is the best case with human clients, and the second is the order of magnitude to plan for agentic load, which arrives in a closed loop against fast responses. There is one reported case that resembles this scenario closely: an in-house OpenAI-compatible engine served around 16 requests per second directly and around 9 through LiteLLM, with the degradation growing with concurrency, on a 4 vCPU, 8 GB pod.

The latency gap the metrics cannot see

LiteLLM’s timers start when its handler starts. The time the request spends queued in the uvicorn event loop, before getting there, appears in none of its metrics. The example in its own troubleshooting guide is a case in which LiteLLM records 10 seconds and the user experiences 20.

It is detected by comparing two sources: GET /health/backlog or the litellm_in_flight_requests gauge against the response time measured by the load balancer in front. If they diverge, the gap is in the input queue and the answer is more replicas, not more tuning.

Two headers are always on and serve to watch it without instrumenting anything: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms. For the second, the documentation sets the suspicion threshold at 100 ms, above which the diagnosis is that the payloads are too large.

And the first cause of latency that guide mentions is none of the above: LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and with payloads over 2 MB that alone can cost between 2 and 5 seconds per request. A log level set to debug a problem and forgotten there produces exactly the problem it was meant to debug.

The failure mode to put in the runbook

There is an open issue describing a cascade with a characteristic shape. Under sustained 429s from the engine upstream, the pods stopped answering readiness probes during startup, Kubernetes killed them on failed probes, and the restart put more pressure on an upstream that was already saturated. Peaks of 500 requests per second, average of 60, between two and five replicas with 1.3 CPU and 4 GB.

The shape of the failure is what matters more than the specific case: the engine’s saturation turns into a restart loop of the gateway, and from the outside it looks like the problem is the gateway. The defences are the ones already in this article, applied together: startupProbe with a wide margin so readiness does not compete with startup, request_timeout bounded so trapped requests get released, cooldowns that take the deployment returning 429 out of rotation, and CPU autoscaling at 60% so the capacity exists before it is needed. It fits with what was covered in incident response runbooks.

Upgrading without cutting traffic

The deployment is a normal Deployment with RollingUpdate, maxUnavailable: 0 and maxSurge: 1, and there is no further mystery to it except on one point: schema migrations. LiteLLM uses Prisma and applies migrations at startup. With several replicas coming up at once after a version change, the migration must be run by a single process, which is another reason for the dedicated replica with LITELLM_JOB_ROLE: worker.

Before a version bump, two checks that pay for themselves: read the release notes looking for changes in the metadata emitted to the callbacks, because a change like that breaks saved dashboards without touching the service, and verify that the new configuration starts with --detailed_debug on a separate pod before applying it to the fleet.

Go-live checklist

--num_workers 1 and scaling by replicas.
requests equal to limits, 1 vCPU and 4 GiB per worker.
HPA on CPU at 60%. Never on memory.
database_connection_pool_limit computed from max_connections, and the chart’s maxReplicas lowered to something the database can take.
Redis as soon as there is more than one replica, and use_redis_transaction_buffer from ten instances or a thousand requests per second.
livenessProbe on /health/liveliness, readinessProbe on /health/readiness, and /health out of the probes.
startupProbe with a margin of up to 300 seconds.
request_timeout: 600.
LITELLM_LOG out of DEBUG, and disable_error_logs: true.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, litellm_in_flight_requests and the response time of the load balancer in front.

Traps and things that are not what they look like

The HPA scales up and never back down. It is measuring memory, which in this process only goes up.

The pods die in a loop right after deploying. Less than 4 GiB of limit. The footprint appears with the first large request, not at startup.

Postgres starts rejecting connections at the peak. The chart has maxReplicas: 100 and the pool defaults to 10.

The per-minute limits are enforced badly. There is no Redis, and each replica keeps its own count.

The health check consumes tokens. Someone set /health as the readiness probe.

The readiness probe returns 200 with the database down. allow_requests_on_db_unavailable is set.

A deployment keeps receiving traffic after failing. The cooldown is 5 seconds, not minutes, and with a single deployment in the group you have to pass a thousand requests for the logic to apply.

The fallback test no longer tests anything. The simulation parameters are stripped from requests arriving through the proxy since 1.85.0.

The p99 is ten times worse than the published benchmark. The benchmark has think time and the real load is agentic.

Closing

None of these settings is complicated and all of them are one line. What unites them is that their defaults describe a small deployment, with one instance, a human client on the other side and a commercial provider taking care of capacity. An in-house inference factory is the opposite on all four axes: several replicas, automated clients, and an engine whose capacity is finite and known.

Of the ten points in the checklist, three decide almost everything in an on-premise deployment: the process model with its sizing, the database connection arithmetic, and the request timeout. The other seven are the ones that avoid the Saturday phone call.

Sources

LiteLLM, Production best practices (topología, workers, memoria, pool de conexiones, LITELLM_JOB_ROLE, buffer de transacciones en Redis, request_timeout): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Health checks (los tres endpoints, comprobaciones de fondo, enable_drain_endpoint): https://docs.litellm.ai/docs/proxy/health.
LiteLLM, Reliability (fallbacks, reintentos, deprecación de los parámetros de simulación en la 1.85.0): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, Benchmarks y Load test advanced (las cifras de sobrecarga por número de instancias y la advertencia de la ley de Little): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/load_test_advanced.
LiteLLM, Rust AI Gateway benchmarks (los 257,7 ms de p99 del proxy en Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, Latency overhead (el hueco de la cola de uvicorn, las cabeceras de diagnóstico, el coste de LITELLM_LOG=DEBUG): https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, DB info (disable_spend_logs, disable_error_logs y qué se pierde con cada uno): https://docs.litellm.ai/docs/proxy/db_info.
LiteLLM, litellm/constants.py (los valores por defecto de reintentos, fallos permitidos, enfriamiento y timeout): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
LiteLLM, litellm/router.py (la discrepancia entre el docstring de cooldown_time y la constante que usa el código): https://github.com/BerriAI/litellm/blob/main/litellm/router.py.
LiteLLM, issue #15526 — cascada de 429 sostenidos, sondas de readiness fallidas y bucle de reinicios: https://github.com/BerriAI/litellm/issues/15526.
LiteLLM, issue #21046 — degradación de throughput contra un motor propio compatible con OpenAI: https://github.com/BerriAI/litellm/issues/21046.

Crossplane: the control plane that turns your cluster into a private cloud API

Mon, 31 Aug 2026 09:30:00 +0200

Second article in the batch about rebuilding the cloud experience on-premise. The general map placed the gaps for you; this one goes into the provisioning control plane, that API which on-premise does not come for free. The third moves up a layer, to application-centric self-service with KubeVela and Score.

TL;DR

Crossplane solves the “open a ticket and wait” gap for you. In the cloud you ask for a database through an API and it appears with permissions, backups and monitoring; on-premise you do that provisioning by hand or with scripts. Crossplane brings the Kubernetes model of declarative APIs to infrastructure: your platform team defines an API of its own (a database, an inference endpoint) and the developer consumes it with a kubectl apply of a few fields, while controllers inside the cluster reconcile the desired state against reality and correct drift continuously. It graduated from the CNCF in November 2025, with more than three thousand contributors and adopters such as Nike, SAP, IBM or NASA’s science cloud.

The difference from Terraform is one of nature, not of features: Terraform applies a plan on demand and leaves; Crossplane is a control plane that lives inside the cluster and never stops reconciling. That gives you drift self-correction and native GitOps, and in exchange it takes away the safety net of plan and forces you to operate the control plane itself, with providers that create hundreds of resource definitions and state that lives in etcd. And a warning about something awkward up front: Crossplane is not an imperative aws ... style CLI, it is declarative first. On the fit with GPU inference, which until recently was only a pattern with no public case, you now have a real reference, Modelplane, from the Crossplane team itself, although at version 0.1 and composing vLLM directly without going through KServe. I go into the detail at the end.

The analogy: the thermostat against the light switch

Think of it this way: Terraform is a light switch. You flip it, the light comes on, and it stays there until you touch it again. If somebody else changes it, the light obeys whoever touched it last, and you do not find out until you go and look. It is provisioning by one-off action: you run it, it applies, it finishes.

Crossplane is a thermostat. You do not tell it “turn the heating on”, you tell it “I want twenty-one degrees”, and from then on a mechanism watches the temperature and acts on its own, continuously, to hold it. If somebody opens a window and the thermometer drops, the thermostat reacts without you intervening. That is continuous reconciliation: you declare the desired state once and a controller takes care of reality not straying from it.

The analogy also warns you about the cost. A thermostat is more convenient than a switch, but it is one more device you have to install, power and fix when it fails, and if you program it badly it adjusts the temperature to something you did not want, without asking. A switch almost never breaks and you always know what state it is in. Crossplane gives you the convenience of the thermostat in exchange for you operating the thermostat.

What Crossplane is and what state it reaches 2026 in

Crossplane was born at Upbound in 2018 and followed the CNCF maturity path without shortcuts: sandbox in 2020, incubation in 2021 and graduation on 6 November 2025. Graduation matters because it is the seal the CNCF reserves for projects it considers production-ready, the same level as Kubernetes or Prometheus. The figures that came with the announcement are solid: more than three thousand contributors from more than 450 organisations, more than a thousand pull request authors (which puts it in the top 10 % of the foundation’s projects by that metric), more than a hundred releases and two completed security audits. Among the more than seventy public adopters you have Nike, Nokia, Grafana, NASA’s science cloud, SAP and IBM.

The project’s official description is “the cloud native control plane”, and that phrase sums up exactly what it does. The main vendor behind it is still Upbound, which at graduation framed the project’s future towards “AI-native” infrastructure, a point I come back to when talking about inference, with the caveat that it is a vendor’s positioning, not a proven case.

The release cadence is quarterly, with a nine-month support window and three actively maintained versions at any time. As of writing, the v2 branch is the current one, with v1.20 as the last close of the v1 branch.

The v2 turn (which changes how you think about it)

Crossplane 2.0 reached general availability in August 2025 and brings four structural changes you need to know about, because a good deal of the old material describes a model that is no longer the recommended one.

The first is that resources are now namespaced by default. Both composite and managed resources, which used to be cluster-scoped, now live in a namespace, which enables fine-grained access control and natural multi-tenancy, exactly what you need in a sovereign platform with several teams.

The second is that a composition can include any Kubernetes resource, not only Crossplane resources. Previously a composition was limited to Crossplane’s own managed resources; now it can include Deployments, Services or third-party CRDs, so you can compose abstractions mixing application and infrastructure in a single object. This change is what makes the inference pattern you will see viable.

The third is that Claims disappear. In the v1 model there was a duality between the composite resource, cluster-scoped, and the Claim, its namespaced face which the developer consumed. Now that the composite resource is itself namespaced, that duality is redundant: the developer consumes the composite resource with no intermediary. If you come across documentation talking about Claims, it belongs to the earlier model.

The fourth is a new type, Operations, for one-off or scheduled operational tasks that run a pipeline of functions to completion, like a Kubernetes Job but inside the Crossplane model.

There is one removal that deserves its own paragraph, because it is the most surprising: in v2 native patch-and-transform was withdrawn, the classic patching mechanism of compositions. It was not relegated, it was removed. Everything now goes through composition functions, and I get to that in a moment. Migration from v1 to v2 is mostly non-breaking and the v1 APIs still work, so adopting v2 is optional, but the new model is the one you have to learn.

The model, piece by piece

Crossplane has dense jargon and I will take it apart calmly, because once the pieces are placed the whole thing is coherent.

A provider is a package that installs controllers and resource definitions to talk to an external API, be it AWS, GCP, a remote Kubernetes cluster, Helm or Terraform. It is installed declaratively, by applying a Provider object. The providers for the big cloud providers are generated with Upjet, a framework that produces the code from the Terraform providers, which explains why there are so many of them and also why some are enormous.

A managed resource is the one-to-one representation, inside Kubernetes, of a resource from the external provider: a bucket, an instance, a database. It is the unit the controller reconciles. When you create a managed resource of type bucket, the provider’s controller creates the real bucket and then watches that it keeps existing with the configuration you declared.

On those bricks you build the abstraction, which is where the real value is, with three concepts:

The CompositeResourceDefinition, or XRD, defines the schema of your own API: which fields the user can ask for. It is what turns “a database” into a resource type with its parameters.

The Composition is the template and the logic that, given one of those resources, materialises the N managed resources needed underneath. The documentation says it bluntly: a composition is a pipeline of composition functions.

The Composite Resource, or XR, is the concrete instance: the object the developer creates with kubectl apply, with the few fields the XRD exposes.

The complete pattern is this: your platform operator publishes the abstraction (the XRD plus the Composition, which say “this is what a database means here”), and the developer only applies a composite resource with four fields. How it is built underneath stops mattering to them, just as in the cloud you do not care how AWS assembles your managed database.

Composition functions: the real logic

The composition mechanism in v2, and the only one, is composition functions. A composition is a pipeline of functions that Crossplane invokes over gRPC; each function receives the observed state, the desired state, its input and a shared context, and chains its output to the next one. The YAML in the examples is illustrative, but underneath it is not processed as YAML: it is executed as functions.

The available functions cover several styles. You have function-patch-and-transform, which reimplements the old model now as just another function. There is function-go-templating for Helm-style templates, function-kcl for the KCL language, function-python for logic in Python and function-cue for CUE.

The reason for the change is what you want to understand. Classic patch-and-transform had no loops or conditionals, so an operation as simple as transforming an array of strings turned into dozens of lines of patching, and the community went as far as calling it outright bad. Functions give you full programming logic (loops, conditionals, external calls) in a real language. The honest flip side is that the complexity does not disappear: it shifts from “writing verbose patches” to “writing and operating functions”, which is a different kind of work, not less work.

Crossplane against Terraform, OpenTofu and Pulumi

This is the comparison everybody makes, and I will make it properly because the difference is one of category. According to Pulumi’s own documentation, which compares honestly, Crossplane is a declarative control plane with continuous reconciliation inside Kubernetes, while Terraform and Pulumi are infrastructure as code executed one-off, client-side and on demand.

The practical consequences line up along a few axes. Faced with configuration drift, Crossplane corrects it continuously and automatically, while Terraform and Pulumi only detect it when you run a refresh. Crossplane needs a cluster to run on, because it is controllers; Terraform and Pulumi do not. Crossplane is GitOps-native, applied with kubectl and fitting with Flux without an adapter; Pulumi needs its Kubernetes operator for the same thing. And as for language, Crossplane uses YAML and CRDs with logic in the functions, while Pulumi uses general-purpose programming languages.

The point most often forgotten is that you do not have to choose one or the other. There is a Terraform provider (and an OpenTofu one, recommended for new versions because the Terraform one was frozen at the last freely licensed version) that runs HCL inside Crossplane, wrapping your existing modules as a managed resource. You can reuse your whole investment in Terraform and still gain continuous reconciliation for whatever needs it. If you already have Terraform in production, this hybrid pattern is usually your sensible route, not the rewrite.

On comparative adoption you have to be sober: there is no official, reliable “Crossplane against Terraform” figure, and the most repeated documented criticism is that Crossplane lacks the mature module ecosystem Terraform has been accumulating for years. That is a real qualitative difference, not a detail.

The awkward question: is it an AWS-style CLI?

Since on this blog I have been pulling on the thread of CLI provisioning, I will answer directly: no. Crossplane does not offer you an imperative aws s3 create-bucket style experience. It is declarative first. You provision with a kubectl apply of a resource, and the crossplane binary is there to build, package, test and debug, not to “create a bucket” in one go.

The subcommands make that clear. crossplane render renders a composition and its functions locally, so you can test without a cluster. The crossplane xpkg build, push, install family manages the OCI packages of providers, configurations and functions. And crossplane beta validate, trace and top validate schemas, draw the resource tree and show consumption inside the cluster. It is the toolbox of whoever builds the platform, not of whoever consumes it day to day. Upbound’s up CLI adds more, but oriented towards its commercial product.

The conclusion matters for setting expectations: the imperative AWS-console-style experience, if you want it, gets built on top of Crossplane, with Backstage, a portal or your own wrapper. Crossplane gives you the declarative API; the friendly face you put on separately. If what you are after is an imperative provisioning CLI, you fit better with the model of KubeVela and Score, which is precisely the subject of the next article.

On-premise providers: provisioning without touching AWS

For a sovereign platform your question is what you can provision without setting foot in a public cloud, and the answer is more than it seems, with uneven maturity.

The most versatile is provider-kubernetes, which manages any Kubernetes object in remote clusters through an Object resource. It is the basis of the multi-cluster on-premise pattern: from a central cluster you materialise resources in your secondary RKE2s. Alongside it, provider-helm installs and manages Helm charts declaratively through a Release resource, letting you deploy things like vLLM or KServe as part of a composition.

For the identity and storage part you have concrete pieces: provider-keycloak configures Keycloak realms, clients and roles declaratively, useful for the sovereign IAM we already set up in the article on authentication with Keycloak; and provider-minio manages buckets and policies of on-premise S3-compatible storage. For vSphere you do not have a first-class native provider, but you cover it by generating one from the Terraform provider, the usual pattern for VMware infrastructure.

An honest warning: for KubeVirt, Harvester or CloudNativePG there are no mature, first-class native providers; in practice you manage them by applying their CRDs through provider-kubernetes or by wrapping their Terraform. It works, but it is not the polished experience of a dedicated provider. With the combination of provider-kubernetes, provider-helm, provider-minio and provider-keycloak, plus provider-terraform for whatever is missing, you can assemble a one hundred per cent on-premise self-service catalogue, with the caveat that maturity varies piece by piece.

The fit with GPU inference: from pattern to Modelplane

This is the section that in the first version of this text carried the most caution, because there was no public case of Crossplane orchestrating GPU and inference and it all remained a design pattern. Halfway through 2026 that changed, and the case was published by the very team that makes Crossplane.

Modelplane is an open source control plane for AI inference, Apache-licensed and with a first version 0.1 from June 2026, and the interesting thing is how it is built: entirely on Crossplane compositions and composition functions in Python, with no controllers of its own. It composes vLLM directly, with the vllm/vllm-openai image, without going through KServe, and its self-service API for the ML teams is a ModelDeployment resource where you declare the model, the engine, the topology and the hardware. Underneath, a ModelReplica resource represents each replica, and the platform team declares its GPU pools with InferenceCluster and InferenceClass resources, using an attribute model and CEL expressions taken from Kubernetes dynamic resource allocation. Its most advanced piece is a two-level fleet scheduler written as a pure function, which places replicas across the whole infrastructure and delegates to each cluster’s scheduler. It is proof that the pattern works, with the warning that it is a version 0.1 under open development, not something to put into production blindly, and that adopting it ties your platform to its resource types.

The other recent enabler is in KServe, which introduced a type dedicated to large models, LLMInferenceService, still in alpha and built on the llm-d project, with prefill and decode separation and multi-node execution, and which runs vLLM as its runtime. It is different from the classic InferenceService, reserved for predictive ML. That new type is your ideal brick if you prefer composing KServe instead of bare vLLM.

With that, your own golden path stops being a hypothesis. You define an XRD, for example InferenceEndpoint, with fields such as the model, the number and type of GPU and the replicas. Its composition, taking advantage of the fact that in v2 it can include any Kubernetes resource, materialises in a single pipeline the team’s namespace, the quota for nvidia.com/gpu and a KServe LLMInferenceService pointing at the model with vLLM runtime, and the developer only applies the InferenceEndpoint with four fields, knowing nothing about KServe or the quotas underneath. KServe supports vLLM and GPU natively, as we saw in the article on KServe.

The honest reading is that you are at the frontier, not on well-trodden ground. Outside Modelplane, made by Upbound, there is still no body of independent cases assembling Crossplane with vLLM, and the KServe type is in alpha. It is a coherent architecture, with a real reference backing it and plenty of room for you to be among the first to document it on your own iron.

What it costs you to operate your own control plane

No tool that gives you back control is free, and Crossplane has well-documented costs you need to budget for beforehand, not discover afterwards.

The learning curve is steep, acknowledged even in the coverage of its graduation. Adopters report debugging difficulties when a composition or a provider misbehaves, and they miss Terraform’s module ecosystem.

Compositions tend towards complexity. There are practitioners describing definitions spanning thousands of lines with poor navigation, and the absence of an integrated testing framework. Functions improve this, but as I said, they shift the complexity rather than remove it.

There is no plan. Unlike Terraform’s plan and apply cycle, Crossplane applies changes immediately. You lose the safety net of reviewing what is going to happen before it happens, with the risk of modifications you did not want.

Operating the control plane itself costs. Big providers create hundreds of resource definitions (the GCP one reached 347), with a risk of API server unavailability and of leaving your cluster hanging. State lives in etcd, which complicates disaster recovery and migrating control between clusters. And there is one specific danger: accidental deletion of cloud resources when upgrading XRDs, providers or compositions, because of the ownership Crossplane keeps over what it created.

Some have decided not to adopt it. The Masterpoint team ruled it out for the lack of data sources and ignore_changes, for incomplete coverage of AWS services, and for the cost of abandoning their accumulated Terraform experience. It is a legitimate decision, and quoting it to you is part of an honest article.

How it lives alongside what you already have

If you already run a platform, your question is not Crossplane yes or no, but Crossplane with what. Three relationships matter to you.

With Backstage they are complementary, not rivals. Backstage is the portal and the templates that generate manifests or pull requests, day zero; Crossplane provisions and reconciles the real state, day two. The usual pattern, which some call the golden triangle, is Backstage as the interface, Crossplane as the control plane and Argo CD or Flux as GitOps. Each one does a layer.

With Flux, which you probably already use, the relationship is additive. Flux reconciles manifests towards the clusters, but it does not give you a platform abstraction API: it does not know what “a database” is, it only applies the YAML you hand it. Crossplane adds exactly that abstraction layer, and the two coexist, with Flux deploying the composite resources and Crossplane materialising them. If you already have Flux, Crossplane’s value is precisely that self-service API layer, in exchange for the cost of operating the control plane I described above.

With KubeVela and Score, the subject of the next article, the boundary is one of layer: Crossplane answers “give me a Postgres”, from the infrastructure side; KubeVela and Score answer “deploy my application”, from the workload side. They do not compete, they stack.

For your sovereign inference factory

The practical decision boils down to this. Crossplane makes sense when you want to expose a self-service API of your own over your on-premise infrastructure and you are willing to operate a control plane to get it. If your platform is used by two teams who know each other and you provision with a tidy Flux repository, Crossplane is cost without return for now. The trigger is scale: when manual provisioning turns into a ticket bottleneck, the declarative API starts paying back its cost.

Your sensible entry route is not rewriting everything. It is starting with a provider that already serves you (provider-kubernetes and provider-helm cover a lot), wrapping your existing Terraform with the corresponding provider instead of throwing it away, and modelling a single abstraction of clear value, probably the InferenceEndpoint we discussed, as a golden path. From there, you grow on demand. And always with a clear account of having traded the safety net of plan and a handful of scripts for a thermostat you have to keep switched on.

The next article moves up a layer, from the infrastructure resource to the application workload, with KubeVela and Score, which is where the friendly CLI that Crossplane deliberately does not give you finally appears.

Sources

CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
Crossplane Blog, Announcing Crossplane’s CNCF Graduation — https://blog.crossplane.io/crossplane-cncf-graduation/
Crossplane Blog, Announcing Crossplane 2.0 — https://blog.crossplane.io/announcing-crossplane-2-0/
Crossplane Docs, What’s New in v2 — https://docs.crossplane.io/latest/whats-new/
Crossplane Docs, Compositions — https://docs.crossplane.io/latest/composition/compositions/
Crossplane Docs, Release Cycle — https://docs.crossplane.io/latest/learn/release-cycle/
Crossplane Docs, Crossplane CLI — https://docs.crossplane.io/latest/cli/
Upbound, Crossplane Graduates From CNCF, Upbound Redefines AI-Native Infrastructure — https://www.upbound.io/blog/crossplane-graduates-from-cncf-upbound-redefines-ai-native-infrastructure
InfoQ, Crossplane Reaches Production Maturity by Graduating CNCF — https://www.infoq.com/news/2025/11/crossplane-grad/
Pulumi Docs, Pulumi vs. Crossplane — https://www.pulumi.com/docs/iac/comparisons/crossplane/
Upbound Marketplace, provider-terraform — https://marketplace.upbound.io/providers/upbound/provider-terraform/latest
GitHub, provider-kubernetes — https://github.com/crossplane-contrib/provider-kubernetes
GitHub, provider-helm — https://github.com/crossplane-contrib/provider-helm
GitHub, provider-keycloak — https://github.com/crossplane-contrib/provider-keycloak
VSHN, provider-minio — https://github.com/vshn/provider-minio
Masterpoint, Crossplane: Why it Didn’t Work for Us — https://masterpoint.io/blog/passing-on-crossplane/
CECG, Crossplane: the good, the bad and the ugly — https://www.cecg.io/blog/crossplane-the-good-the-bad-the-ugly
Taloflow, Backstage vs Crossplane for Platform Engineering — https://www.taloflow.ai/guides/comparisons/backstage-vs-crossplane-platform-engineering
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/
Modelplane — https://modelplane.ai · https://github.com/modelplaneai/modelplane
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

Mon, 31 Aug 2026 09:30:00 +0200

Second post in the platform and self-service run, after Backstage and before Knative. The question here is not what Kubeflow is, but which parts of Kubeflow still make sense once you have built half a platform on your own.

TL;DR

Kubeflow graduated in the CNCF on 17 August 2026, and that headline hides the important one: it is no longer a monolithic platform, it is an umbrella of independent subprojects, each with its own repository, its own version and its own life. That changes how you have to evaluate it entirely. The 2020 question, should I install Kubeflow?, has become the 2026 one, which piece of Kubeflow do I install, and at what cost?

For a platform that already has RKE2, vLLM, KServe, Flux and Volcano or Kueue, the honest answer is that Kubeflow as a whole duplicates almost everything you already have. KServe no longer lives under Kubeflow: it left, and since November 2025 it has been a CNCF project in its own right, so anyone serving models with KServe does not need Kubeflow to serve. The Trainer leans on Volcano and Kueue, it does not replace them. And the full manifest drags in Istio, Dex and cert-manager as hard dependencies that clash with what you already operate. The only thing that may be worth installing on its own is Pipelines, as an orchestrator for data and training flows, and perhaps the Trainer for its fine-tuning templates. The rest you either already have, or do not need.

The analogy: the hypermarket and the specialist shop

Kubeflow was born as a hypermarket: a single building with everything an ML team might need, from notebooks to the inference server, taking in pipelines and hyperparameter tuning along the way. It made sense in 2019, when assembling each piece separately was a project in itself and there were no standards.

By 2026 the hypermarket has been converted into a street of specialist shops. The butcher’s counter, KServe, went independent, opened its own premises and is doing better alone. The fishmonger’s, the Trainer, now depends on the central market next door, Volcano and Kueue, instead of having its own cold store. And the shared building still demands that you put in its wiring, its plumbing and its security system (Istio, cert-manager, Dex) even though you already have your own.

Anyone arriving today with a half-full fridge, which is the case for anyone with a platform already running, does not need the whole hypermarket. They need to know which shop to walk into for what they are missing, and to avoid buying twice what they already have at home.

What Kubeflow is in 2026

Kubeflow was created by Google in 2017 and graduated in the CNCF on 17 August 2026, reaching the top level alongside Kubernetes or Prometheus. The graduation figures are serious: more than 6,600 contributors from over a thousand organisations, and adopters such as Bloomberg, NVIDIA, Red Hat or Spotify.

The structural change is the one that matters for the decision. The full distribution, the Kubeflow Community Distribution, is numbered by year and month (the current one is 26.03, from March 2026, on a six-monthly cadence), but each component is by now a subproject with its own repository and its own cycle, usable independently. The distribution only packages them together. That means “installing Kubeflow” is no longer an atomic decision: it is a shopping list.

Component by component

Kubeflow Pipelines: the piece you may actually want

Pipelines orchestrates containerised ML workflows. They are written in Python with its SDK, which compiles to an intermediate YAML representation so they are portable, and underneath they run on Argo Workflows. It is the piece most likely to add value to an existing platform: a flow of “ingest, embeddings, fine-tuning, evaluation, deployment” is exactly what you need to operate RAG and adapters repeatably.

The small print has two parts. The first is that if you already use plain Argo Workflows, Pipelines adds a layer of SDK, interface and metadata on top of the same engine: it does not bring a new scheduler, it brings the typed component model and the runs interface. The second is that its metadata and lineage system, MLMD, is being removed from Pipelines itself, so it is not wise to build the platform’s traceability on that particular piece.

Kubeflow Trainer: it leans on your scheduler, it does not replace it

The old Training Operator, the one with PyTorchJob and TFJob, has been rewritten as Kubeflow Trainer, with a single TrainJob API that unifies all frameworks. Version 2.2 positions itself explicitly for distributed training and LLM fine-tuning, with support for PyTorch, DeepSpeed, HuggingFace and company.

A common expectation needs correcting here. The Trainer does not replace Volcano or Kueue: it leans on them. The TrainJob carries a podGroupPolicy field that automatically creates the Volcano PodGroup objects for gang scheduling, and it integrates with Kueue’s queues. If you already built that scheduler, as we saw in its post, the Trainer adds the TrainJob abstraction and some fine-tuning templates on top of what you already have, not a new scheduling layer. It may be worth it for the templates; not for the scheduler.

Katib: only half relevant

Katib does Kubernetes-native AutoML: hyperparameter optimisation with algorithms such as Bayesian optimisation, TPE or Hyperband, plus architecture search. In the LLM era it has repositioned itself, with official documentation for tuning fine-tuning hyperparameters and even for tuning RAG pipelines.

The practical reading is lukewarm. Architecture search is irrelevant for anyone doing LoRA and RAG. Hyperparameter optimisation can be useful for learning-rate or LoRA-rank sweeps, but it competes with doing it from the SDK of the QLoRA runbook itself or with Ray Tune, and it rarely justifies dragging in Katib for that alone.

KServe: the piece that is no longer in Kubeflow

This is the fact that settles half the evaluation. KServe left Kubeflow. It was born inside the project in 2019, was donated to the Linux Foundation in 2022, was renamed from KFServing to KServe and graduated from Kubeflow that same year, and since November 2025 it has been a CNCF incubating project in its own right. In the 2026 taxonomy it appears as an ecosystem project, external, not as a core component.

The consequence is direct: anyone already serving with KServe does not need Kubeflow to serve. And KServe is exactly where the LLM action is: its 0.15 release strengthened the vLLM backend and introduced an LLMInferenceService resource with disaggregated serving, prefix caching, per-variant autoscaling and OpenAI-compatible APIs. If your serving layer is vLLM on KServe, you already have that layer complete and Kubeflow adds nothing to it.

Model Registry, Notebooks and the rest

The Model Registry, now reorganised under the name Kubeflow Hub, is an index of models, versions and metadata. It stores pointers and states, not the bytes. Against MLflow, it lacks the experiment and metric tracking that is MLflow’s strength, to the point that the guides recommend combining them rather than substituting one for the other; and against an OCI registry, it does not package the binary, which still lives wherever you put it. It is still a young component, below version 1.0.

The Notebooks live in two versions, the stable 1 and a 2 redesigned on CRDs that is still in alpha. The Spark Operator remains in the core; Feast, by contrast, has left the core and is now an ecosystem project, a nuance worth being clear about before counting on it.

Multi-tenancy: powerful and tied to Istio

Isolation between teams is done with Profiles, a CRD that wraps a namespace and gives it RoleBindings, ServiceAccounts and Istio authorisation policies that validate an identity header derived from OIDC. It is a complete model, but it has an Achilles heel: it depends entirely on the Istio sidecar and on that header. If traffic bypasses the mesh or someone forges the header, isolation breaks, and all multi-tenant security ends up coupled to Istio, which clashes head-on if you already operate another mesh or no service mesh at all.

The cost of installing the whole thing

Here is the main reason not to install the full distribution on top of an existing platform. The 26.03 manifest drags in as hard dependencies: Istio, cert-manager, Dex for OIDC, OAuth2-Proxy, and Knative Serving and Eventing for KServe. It recommends 16 GB of RAM and 8 vCPU as a minimum, and the aggregate of all components comes to around 4.4 CPU cores and 12 GB of memory for the control plane alone.

The project itself acknowledges the problem. There is an open thread in its repository, bluntly titled “community feedback on Kubeflow’s complexity”, where it is admitted that Kubeflow practically forces a dedicated cluster because of the assumptions it makes about what is installed, that Istio and Dex should be swappable and not fixed dependencies, and that a lighter Helm that installs only what is needed is missing. For anyone already operating cert-manager, an ingress and GitOps with Flux, the full manifest does not coexist: it duplicates and tramples the base infrastructure.

Kubeflow against the alternatives

The field is wide and almost all the comparative literature is written by vendors with a product of their own, so it has to be read with tweezers. With that caution:

Plain Argo Workflows is the engine Pipelines uses underneath; if all you need is container graphs, Pipelines is overhead. MLflow is strong in experiment tracking and lightweight registry, installs in minutes, and does not orchestrate distributed training: it is complementary, and in fact we use it in the post on prompt versioning. Flyte is a typed orchestrator that many perceive as lighter to operate than full Kubeflow. And Ray on Kubernetes is the most serious competitor for LLM workloads, because with a single runtime it covers training, serving and tuning, overlapping at once with the Trainer, Katib and KServe.

Where Kubeflow wins is in breadth and in governance: a CNCF-graduated umbrella with adoption by large companies. Where it loses is in operational weight, in hard dependencies, and in the fact that its best pieces are already independent projects that do not require the rest.

The real fit with LLM workloads

The project has moved towards GenAI, and that has to be acknowledged: there is an SDK with LLM fine-tuning templates, the Trainer adds support for reinforcement post-training methods, and new agent-oriented pieces have appeared. It is not empty marketing, there is real work there.

But the question for a specific platform is not whether Kubeflow does LLM: it is which part I have not already duplicated. On a stack of RKE2, vLLM, KServe, Flux and Volcano or Kueue, the balance looks like this:

Component	Does it add anything new to your stack?
KServe	No: you already have it, and it is no longer Kubeflow’s
Trainer	Only the fine-tuning templates; you already have the scheduler
Katib	Marginal: LoRA HP sweeps, little more
Pipelines	Possibly yes, as a flow orchestrator, but it runs on Argo
Model Registry	Optional: competes with MLflow and with your OCI registry
Notebooks	Depends on what you use for notebooks today
Multi-tenancy	No, if you already isolate by namespace without tying everything to Istio

Operational traps and honest scepticism

Do not install the full manifest on a platform that already works. You are going to duplicate Istio, cert-manager and the rest, and fight a dependency war with no prize at the end. If something in Kubeflow interests you, install it as a standalone subproject.

KServe is not Kubeflow. It is the most common category error of 2026. If someone justifies building all of Kubeflow “to serve models”, you already know they have not checked that KServe left long ago and lives on its own.

The Trainer does not save you the scheduler. It needs Volcano or Kueue underneath. If you expected it to solve GPU gang scheduling by itself, it does not.

Multi-tenancy ties you to Istio. Before adopting Profiles, ask yourself whether you want security between teams to depend on one specific mesh. If you already isolate well by namespace and RBAC, you may not need that layer.

MLMD is on its way out. Do not build your pipelines’ traceability on the metadata system the project itself is retiring.

For an inference factory

The conclusion is comfortable to apply. Kubeflow in 2026 is not a yes-or-no decision, it is a menu you order from à la carte. On a platform that already serves with KServe, schedules with Volcano and deploys with Flux, the menu shrinks a great deal.

If you need to orchestrate data and training flows repeatably, and you do not want to build plain Argo Workflows with its ergonomics, Pipelines is the piece worth evaluating, installed standalone. If you do distributed fine-tuning frequently and want a uniform abstraction on top of your scheduler, the Trainer may save you some templates, knowing that it leans on the Volcano you already have. Everything else you either have, or a specialist piece covers better.

The expensive mistake would be installing the whole hypermarket to buy bread. Kubeflow stopped being that; treat it as what it is, a street of shops, and walk only into the one you are missing. The next piece in this run, Knative, is precisely one of those specialist shops, the one that decides whether switching off the GPU when nobody is using it pays off or costs you dear.

Sources

CNCF, CNCF announces Kubeflow’s graduation — https://www.cncf.io/announcements/2026/08/17/cncf-announces-kubeflows-graduation-solidifying-the-standard-for-cloud-native-ai-operations/
CNCF, Kubeflow unveils new cloud native innovations to supercharge AI — https://www.cncf.io/blog/2026/07/28/kubeflow-unveils-new-cloud-native-innovations-to-supercharge-ai/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
Kubeflow Docs, Introduction / components — https://www.kubeflow.org/docs/started/introduction/
Kubeflow Docs, Pipelines overview — https://www.kubeflow.org/docs/components/pipelines/overview/
Kubeflow Trainer Docs, Volcano gang scheduling and Kueue — https://trainer.kubeflow.org/en/latest/operator-guides/job-scheduling/volcano.html
Kubeflow Docs, Migrating to Kubeflow Trainer v2 — https://www.kubeflow.org/docs/components/trainer/operator-guides/migration/
Kubeflow Docs, Katib overview — https://www.kubeflow.org/docs/components/katib/overview/
Kubeflow Docs, Hyperparameter optimization for LLM fine-tuning — https://www.kubeflow.org/docs/components/katib/user-guides/llm-hp-optimization/
Kubeflow Docs, Model Registry overview — https://www.kubeflow.org/docs/components/model-registry/overview/
Kubeflow Docs, Multi-tenancy design — https://www.kubeflow.org/docs/concepts/multi-tenancy/design/
GitHub, kubeflow/manifests (dependencias y recursos) — https://github.com/kubeflow/manifests
GitHub, kubeflow/manifests #2451: community feedback on complexity — https://github.com/kubeflow/manifests/issues/2451
InfoQ, Kubeflow expands AI capabilities as CNCF graduation nears — https://www.infoq.com/news/2026/08/kubeflow/

Backstage as a self-service portal: giving your teams the LLM platform without giving them the cluster

Mon, 31 Aug 2026 09:00:00 +0200

This article opens the platform and self-service layer: the one that decides how you hand everything underneath to the teams that are going to use it. The three articles in this batch are Backstage (the portal), Kubeflow (the ML toolbox) and Knative (switching the GPU off when nobody is using it).

TL;DR

Backstage solves a real problem of a platform that has grown: nobody knows what is there, who maintains it, or how to ask for a new one without opening a ticket. Its answer is three pieces, a software catalogue that models every component and its relations, templates that create new resources through a form instead of through copy and paste, and TechDocs to keep documentation next to the code. On top of those three sits the concept that holds up everything else, the golden path: the supported route made also the easiest one, so that people take it because it is paved, not because they are forced onto it.

The price is high and it should be said plainly. Backstage is not a product you install, it is a Node and React application that you compile and maintain as your own code, at a declared cost of two to three full-time engineers, and with upgrades as its biggest pain point. As of August 2026 it is still a CNCF incubating project, not graduated, and it has no native entity type for models, datasets or inference endpoints: that gets modelled by hand, or you wait for a proposal that is still open. The decision is not about features, it is about scale: below a certain organisation size, a portal costs more than it saves.

The analogy: the industrial estate directory

A single unit needs no sign. You know what is inside because you built it. An estate with two hundred units, on the other hand, needs a directory at the entrance: which company sits on each plot, what it does, who is responsible, and an office where you can request a new plot with the utility connections already in place instead of negotiating each one separately.

An LLM platform starts out as a single unit. One cluster, one model, the team that built it. Six months later it is an industrial estate: fifteen models served, three RAG pipelines, LoRA adapters everywhere, and a question that comes back in every meeting, who maintains this? Backstage is the directory for that estate, with two additions an ordinary directory does not have: the office that hands over turnkey plots, and the archive of blueprints for each unit. The catalogue is the directory, the templates are the office, TechDocs is the archive.

The analogy also warns about the cost. The directory does not maintain itself. Somebody has to update it when a company moves, and a directory with half the plots mislabelled is worse than no directory at all, because people stop trusting it. In Backstage that failure has a name, catalog rot, and it is the leading cause of a portal being abandoned.

What Backstage is and what state it reaches 2026 in

Backstage was born inside Spotify and donated to the CNCF, which accepted it in September 2020 and moved it to incubating level in March 2022. That is where it still is: as of August 2026 it is incubating, not graduated, a detail that matters when someone sells it as a de facto standard. It is by far the most widely adopted developer portal (the official 2025 review declares more than 3,400 organisations using it, 31,000 GitHub stars and more than 250 plugins), but adoption is not governance maturity, and a large share of those plugins is maintained by the community, not by the core.

The architecture has kept changing pieces under the feet of whoever operates it. The New Backend System stabilised in 2024 and is now the standard. The New Frontend System became the default mode for new applications in version 1.49, of March 2026. The cadence is one minor release a month, and that rhythm is precisely the origin of the biggest operational problem, which we will see further down.

The three pieces

The software catalogue

The heart of Backstage is an entity catalogue. Every piece of the platform is described in a catalog-info.yaml file that lives next to its code, and Backstage discovers it by crawling the repository. The entity types are fixed: Component (something that runs or compiles), API (a contract that a component exposes), Resource (infrastructure that a component needs), System (a set of components that form a unit), Domain (a set of systems), plus Group and User for ownership.

The power is in the relations rather than in the entities, which Backstage computes and navigates: ownedBy connects each thing to its team, providesApi and consumesApi draw who depends on whom, partOf groups components into systems, dependsOn traces the dependency graph. With that, the meeting question (who maintains this, and what breaks if we touch it?) stops being answered from memory and starts being answered by query.

There is a design decision that conditions everything that follows: Backstage advises against creating new entity types. Plugins are compiled expecting the usual types, and an invented type leaves them blind. The supported route is to reuse an existing type and distinguish it with the spec.type field (for example a Resource with type: llm-model), adding whatever custom fields you need to the spec.

The templates

The second piece is the Scaffolder, the template engine. A template is an entity of type Template with three parts: the parameters requested through a form, the steps that get executed, and the actions that do the real work (create a repository, write the catalog-info.yaml, register the component, trigger the GitOps pipeline). The developer goes to /create, picks a template, fills in the form, and in a single operation has the resource created, catalogued and deploying.

This is what turns a directory into an office. Without templates, Backstage only lists what already exists. With templates, it is the door through which new things are created, and that door is where the golden path materialises.

TechDocs

The third piece is TechDocs, documentation as code. Documents are written in Markdown next to the code, generated with MkDocs in the CI pipeline, and served inside the portal itself, linked from each component’s page. The advantage is in the incentives: documentation that lives far from the code rots; documentation that lives next to it and gets reviewed in the same pull request has some chance of staying true.

The golden path, which is what all of this is about

Spotify coined the term golden path for the recommended and supported way to build something. It has two halves: the blessed tooling, visible in the portal, and a step-by-step tutorial in the documentation. The idea, in the formulation that platform reports keep repeating, is that nobody forces you to use the paved road; people take it because it is paved.

The data supports the golden path being the deciding variable. The 2026 State of Platform Engineering report, covering 518 professionals, puts voluntary adoption above 80 % where there is a well-designed golden path and below 20 % where there is not, and estimates that around 70 % of platform initiatives fail to reach significant adoption without one. It is a figure that reorders priorities: a portal without golden paths is a pretty catalogue nobody uses.

In Backstage, a golden path is implemented as a Scaffolder template. For an LLM platform, a typical golden path would be “serve a new model”: the form asks for the model, the GPU size, the owning team and the SLO; the template creates the repository with the InferenceService, writes the catalog-info.yaml, registers it under the right system, and opens the pull request against the GitOps repository you already have with Flux. What used to be half a day of copying a colleague’s YAML and guessing the values becomes a two-minute form with the correct defaults.

Authentication and permissions

For a sovereign platform the identity piece matters as much as the catalogue. Backstage ships connectors out of the box for GitHub, GitLab, Google, Microsoft, Okta, generic OIDC and SAML, among others. For on-premise the relevant one is Keycloak, which fits in like any other OIDC provider, with one nuance worth keeping in mind: the Keycloak connector is maintained by the community, not by the core, so its maintenance and its tightness fall on a third-party plugin. Once you have set up Keycloak as we saw in the article on authenticating MCP, the portal hooks into the same identity issuer.

The permissions framework appears today as stable in the documentation and supports RBAC, ABAC and custom logic. The small print: by default the endpoints are not protected, and the permission policy is written by the integrator. The most widely used RBAC solution with a graphical interface comes with the Red Hat distribution, not with the core. In other words, serious access control means work on top of what comes in the box.

The real cost, no frills

This is where an honest article parts ways with the brochure. Backstage is not an installable product, it is a framework that you compile and maintain as your own code. That has measurable consequences.

The most quoted analysis, from the managed Backstage vendor Roadie, puts the effort at around three full-time engineers in the first year and two from the second onwards, with a time to production of six to twelve months in self-managed mode. The figure comes from a vendor with a commercial interest in you buying its managed version, so it should be taken as an order of magnitude and not as a neutral data point, but the direction is confirmed by independent sources: ongoing maintenance takes up around a fifth of engineering time, and upgrades are the biggest pain point, cited by more than half of those who adopt it. With one minor release a month and architectural pieces shifting under your feet, keeping up to date is a permanent job.

The other cost is about adoption, not engineering. A widely repeated critical analysis points out that many organisations settle at around 10 % internal usage, against the 99 % voluntary usage Spotify achieved at home, and attributes it to three recurring failures: the catalog rot we already mentioned, the lack of a product owner treating the portal as a product, and the illusion that Backstage can be adopted over what already exists when in practice it only standardises new things well. The sentence that sums up the disenchantment is that Backstage is a free portal in the same way you would have a free car if somebody tipped all the parts of a Chevrolet onto your desk.

Applied to an LLM platform: what is there and what is not

Here an expectation needs correcting. In 2026 Backstage has no native entities for models, datasets or inference endpoints. There are three things and none of them is what you would hope for:

An open proposal (RFC 33060, from February 2026) to model model servers as entities of type API with type: ai-model-server. It is under discussion, not implemented.

An official module that introduces an AiResource type, which sounds like what we are after but is for something else: governing the skills and rules of coding assistants, not registering inference models.

And one real pattern, the MLOps templates that AWS documents on top of SageMaker, where Backstage acts as the Scaffolder that creates environments and pipelines, not as the model registry. It is a cloud case, not on-premise, and it uses Backstage for its template half, not for its model catalogue.

The practical conclusion for your own stack is twofold. First, what Backstage brings to an LLM platform today is above all the template half: the “create a new inference service” golden path wired into your GitOps. Second, if you want to catalogue models, today that is done by hand with custom spec.type values, treating the model as a Resource and its endpoint as an API, until the proposal matures. Registering the model’s bytes remains the job of the OCI registry, not of the catalogue.

The infrastructure plugins that are worth it in this stack do exist and are reasonably alive: the Kubernetes one (core), the Flux one for seeing GitOps state from the portal, the Grafana one for embedding dashboards in a component’s page, and the ArgoCD one for anyone using Argo instead of Flux.

Alternatives: when not to set up Backstage

The honest decision is not which portal, but whether a portal. The SaaS-type alternatives (Port, Cortex, OpsLevel, Humanitec) win on time to first value, measured in days instead of months, and on not having to maintain the framework, in exchange for a per-user fee that scales badly and for giving up pure on-premise. Roadie offers managed Backstage, which removes the framework maintenance while keeping its extensibility, in exchange for a per-developer fee and for taking your catalogue outside.

For a sovereign platform, where on-premise and data ownership are requirements, pure SaaS is usually ruled out by design, which leaves the real choice between self-managed Backstage and no portal at all. And there the criterion is size. Below a handful of teams, a well-maintained README and a tidy GitOps repository do the same job as the catalogue, without the cost of two engineers. The portal starts paying off when the cognitive load of knowing what is there and how to ask for it exceeds the cost of maintaining the tool that answers it.

Operational traps and honest scepticism

The catalogue rots if it is not automated. A catalog-info.yaml written by hand and never updated is lying within a few weeks. The only defence is generating and validating those files in the pipeline, not trusting people to maintain them.

Without a product owner, the portal dies. Backstage is not infrastructure you install and forget, it is an internal product with users you have to understand. Without somebody treating adoption as an objective, it ends up as a catalogue nobody looks at.

Upgrades are a recurring tax. One release a month and architectural pieces mid-migration mean the portal is never finished. That maintenance has to be budgeted from the start, not discovered in the second quarter.

Backstage deploys nothing. It is a portal, not an engine. Behind each template there has to be your GitOps, your operators and your cluster doing the real work. If that base is not solid, the portal just puts a pretty façade over chaos.

The data model for AI does not exist yet. Anyone setting up Backstage expecting a ready-to-use model catalogue is going to find themselves modelling by hand. It is doable, but it is work, and you should know that beforehand.

For an inference factory

Backstage fits at the high maturity level of a platform, not at the start. In the five maturity levels it is a tool of the level where the platform already serves several teams and the bottleneck has stopped being technical and become one of coordination. Three concrete decisions follow from all of the above.

The first is not setting it up too early. If the platform is used by two teams who know each other, the portal is cost without return. The trigger is cognitive load, not the calendar.

The second is starting with the templates, not with the catalogue. The value you notice from day one is the golden path that turns half a day of copying YAML into a two-minute form. The complete catalogue, with all its relations, is a bigger job that can come later.

The third is budgeting maintenance as a function, not as a project. Two part-time engineers indefinitely, an owner watching adoption, and automatic generation of the catalogue entries. Without that, the portal degrades until somebody asks why nobody uses it, and the answer is that it stopped being true.

The portal is the face your platform shows to whoever uses it. It is worth it when there is enough platform behind it that showing it tidily saves more than maintaining the shop window costs. Before that, it is the shop window of a shop that has no stock yet. The other two articles in this batch go into that stock: the ML toolbox with Kubeflow and the savings of switching the GPU off with Knative.

Sources

CNCF, Backstage (project page) — https://www.cncf.io/projects/backstage/
Backstage, Backstage Wrapped 2025 — https://backstage.io/blog/2025/12/30/backstage-wrapped-2025/
Backstage Docs, Software Catalog: descriptor format — https://backstage.io/docs/features/software-catalog/descriptor-format/
Backstage Docs, Well-known relations — https://backstage.io/docs/features/software-catalog/well-known-relations/
Backstage Docs, Extending the model — https://backstage.io/docs/features/software-catalog/extending-the-model/
Backstage Docs, Software Templates — https://backstage.io/docs/features/software-templates/
Backstage Docs, TechDocs — https://backstage.io/docs/features/techdocs/
Backstage Docs, Authentication — https://backstage.io/docs/auth/
Backstage Docs, Permissions overview — https://backstage.io/docs/permissions/overview/
Backstage GitHub, RFC #33060: modeling AI model servers in the catalog — https://github.com/backstage/backstage/issues/33060
Roadie, Backstage: how much does it really cost? — https://roadie.io/blog/backstage-how-much-does-it-really-cost/
Earthly, Backstage is at the peak of its hype — https://earthly.dev/blog/backstage-is-at-peak-hype/
AWS Prescriptive Guidance, Accelerate MLOps with Backstage and SageMaker templates — https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-mlops-with-backstage-and-sagemaker-templates.html
Spotify Engineering, How we use golden paths to solve fragmentation — https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem
The New Stack, DORA 2024: AI and platform engineering fall short — https://thenewstack.io/dora-2024-ai-and-platform-engineering-fall-short/
bex.co, State of Platform Engineering Report Vol. 4: golden path — https://bex.co/blog/2026/08/06/platform-engineering-report-vol-4-golden-path

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

Sun, 26 Jul 2026 20:00:00 +0200

This blog has devoted a good part of its 145 articles to what happens inside the inference process and another good part to what happens around it. What is missing is a thread that cuts across all of that and answers a question that takes thirty seconds to ask in an audit and does not get answered in three weeks in production: when the endpoint returns a token, where did that model come from, who signed it, and who proves that the binary running on the GPU is the one that was approved?

This series walks that chain backwards from the only visible point, the endpoint: the control plane and the API contract (this article), the registry and the distribution of the bytes (2/4), the signing and provenance that let you trust them (3/4), and the identity and isolation of whoever runs them (4/4). All of it with projects under the governance of CNCF, LF AI & Data or the Linux Foundation: a chain of trust that depends on a proprietary product is no better than a support contract.

TL;DR

A vLLM Deployment serves tokens perfectly well. What it does not give you is a contract: no model versioning, no rollout by percentage, no scale-to-zero, and no common surface for the thirty non-LLM artifacts of an inference factory.
KServe provides the vocabulary: InferenceService, ServingRuntime/ClusterServingRuntime, storage initializer and storageUri. It has been CNCF incubating since 29 September 2025, after seven years as KFServing and a spell in LF AI & Data. The cadence has accelerated: v0.18.0 on 29 April 2026 and v0.19.0 on 14 June 2026.
The LLM branch is recent and different: the LLMInferenceService CRD (serving.kserve.io/v1alpha1) arrived in v0.16 in November 2025, on top of llm-d, with spec.prefill, spec.worker, spec.parallelism and spec.router.
The Open Inference Protocol (V2) is the contract: /v2/health/*, /v2/models/<m>, /v2/models/<m>/infer, in REST and gRPC, implemented by Triton, OpenVINO Model Server, Seldon MLServer and AMD Inference Server. It is tensor-centric, and the LLM world standardised de facto on the OpenAI API: KServe serves both, with the OpenAI ones under the openai/ prefix.
The honest position: the portability of an LLM today comes from the OpenAI API, not from V2. Accepting the dual track is correct; pretending there is a single standard is not.
KServe is overkill if you serve one model on one node. It starts to pay off from the fifth heterogeneous artifact onwards, and it competes on a different axis from the LLM operators: contract and lifecycle versus engine and topology.

The analogy: the station master and the track gauge

An inference factory is like a railway station with several lines.

The station master does not drive any train. His job is the service timetable: which formation leaves from which platform, how many units are coupled each hour, when an engine is pulled out for maintenance and, when new rolling stock arrives, how to divert ten per cent of the passengers to the test train before committing the whole service. That is the control plane. KServe does not generate a single token, but it decides what runs, with what configuration, with how many replicas and with what traffic split. The trains are the engines (vLLM, SGLang, TensorRT-LLM, Triton, TEI), and a competent station master does not marry a manufacturer.

And then there is the track gauge: as long as everybody builds to the same one, any locomotive rolls on any line and changing supplier is a commercial decision, not a civil engineering project. That gauge is the Open Inference Protocol, the contract that makes a client written against Triton work against OpenVINO Model Server without touching a line.

The analogy has a twist that needs no explanation in Spain: the agreed gauge is not the gauge the new rolling stock runs on. The conventional network was built to Iberian gauge and the high-speed one to standard gauge, and the practical solution was not to convert the network but to install gauge changers at the boundary points. The same thing has happened in inference: V2 is the agreed gauge and the OpenAI API is the gauge everything generative runs on. What sustains interoperability is the changer, not a single standard. We will come back to it.

A `Deployment` serves tokens, but offers no contract

In vLLM on Kubernetes we built a service with Deployment, Service, HPA and an initContainer that pulls the weights down. It works, and for one model on one node it is the right answer. What it does not give you, and cannot give you without reinventing it every time:

A stable API contract. The endpoint is “whatever the image you put there exposes”: moving from vLLM to TensorRT-LLM changes routes, error format and metric names, and the clients find out in production.

Versioning with semantics. The model is a path inside a volume; knowing which version a pod is serving requires kubectl exec. There is no declarative field that says “this service serves artifact X” and can be audited from outside.

Progressive model rollout. A RollingUpdate replaces pods; it does not split traffic between two versions by percentage. For that you need an L7 gateway in front, like the one in canary, blue-green and shadow, or a control plane that emits it.

Scale-to-zero. The HPA does not go below one replica: for a model queried twelve times a day, that is an 80 GB GPU tied up twenty-four hours.

And, above all, a common surface for what is not an LLM. A real factory does not serve three LLMs: it serves two or three LLMs, four embedding models, a couple of rerankers, ticket classifiers, a PII detector, an OCR, industrial vision and three scikit-learn artifacts that nobody remembers training. Fifteen or thirty artifacts, each with its own runtime and its own way of saying “I am ready”. The cost is not in serving the first one: it is in making sure the thirtieth does not cost thirty times what the first one did.

KServe: from KFServing to a CNCF incubating project

KServe was born in 2019 as KFServing, inside Kubeflow, in a collaboration between Google, IBM, Bloomberg, NVIDIA and Seldon. In February 2022 it was donated to the LF AI & Data Foundation; in September 2022 it split from Kubeflow and was renamed. The relevant jump: on 29 September 2025 the CNCF accepted it directly at incubating level, without going through sandbox, with a public announcement on 11 November. The figures it declared then, more than 300 contributors, 19 maintainers, more than 30 adopting organisations (Bloomberg, Red Hat, Cloudera, Nutanix, SAP, NVIDIA), come from an interested party, but the list is verifiable and the 625 contributing organisations on the CNCF dashboard support the thesis of governance not captured by a vendor.

On cadence you have to be precise, because the cliché of “KServe is slow” no longer describes reality:

Version	Date	What it brought
v0.15.0	31 March 2025	`LocalModelCache`, multi-node inference, OpenAI-compatible embeddings API, Gateway API in raw mode, KEDA integration
v0.16.0	November 2025	`LLMInferenceService`; rename of `RawDeployment` to `Standard` and of `Serverless` to `KNative`; progressive rollout in raw mode
v0.18.0	29 April 2026	WVA + KEDA/HPA autoscaling for LLMISvc, `/v1/responses` route (OpenAI Responses API), `restricted` Pod Security Standards by default on LLMISvc, namespace-scoped `ModelCache`
v0.19.0	14 June 2026	Routing by model name, static LoRA adapters, balancing across heterogeneous GPUs, dual REST/gRPC routing in Standard mode, automatic migration to `InferencePool` v1

Two major releases in under two months, with the bulk of the changelog in the LLM branch. The project is not slow: it has pivoted from classic predictive serving to generative serving. The consequences run in both directions.

The negative one is called ModelMesh. It was the high-density mode (hundreds of small models sharing processes, with dynamic load and unload) and it was, on paper, the natural answer to the problem of thirty heterogeneous artifacts. Its last release was v0.12.0 on 9 July 2023, and the kserve/modelmesh-serving repository was archived read-only on 14 April 2026. Red Hat published specific guidance for migrating to Standard mode. If somebody proposes ModelMesh in a design today, it is a sign that the documentation they read is three years old.

Anatomy of the control plane

`InferenceService`: predictor, transformer, explainer

The core CRD is InferenceService (serving.kserve.io/v1beta1), and its contribution, the one that became the common vocabulary of the field, is to break an inference service down into three components each with its own contract:

predictor: mandatory. It declares modelFormat, optionally runtime, the storageUri the weights come from and the protocolVersion it exposes.
transformer: optional. Pre- and post-processing. That it is a separate component, rather than code inside the server, is what allows it to be scaled separately (it is usually CPU-bound while the predictor is GPU-bound) and versioned separately.
explainer: optional, and the one that has aged worst: the V2 protocol does not support the explain endpoint, which only exists in V1. Anyone who needs explainability with a standard contract has a real gap here.

All three share ComponentExtensionSpec: minReplicas, maxReplicas, scaleTarget, scaleMetric, containerConcurrency, timeout and canaryTrafficPercent.

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

A ServingRuntime is a pod template that knows how to serve one or several formats. It declares supportedModelFormats (with name, version, autoSelect and priority), protocolVersions (v1, v2) and containers, with args parameterised by templates in the {{.Name}} style that are substituted with metadata from the InferenceService.

Selection is what makes this scale. If the InferenceService names a runtime, the controller looks for it first in the namespace and then at cluster level; if it does not name one, it picks among those with autoSelect: true for that modelFormat, breaking ties by priority. That is what lets a platform publish a curated catalogue (“here, huggingface is served with this vLLM and these flags”) without the product teams having to know what --gpu-memory-utilization is. In multi-tenant setups, the ClusterServingRuntimes belong to the platform and the InferenceServices to the teams: the cleanest separation-of-duties line KServe offers.

The storage initializer and `storageUri`

The storageUri field triggers the storage initializer: an init container that downloads the artifact to /mnt/models before the server starts. Schemes supported as of July 2026: s3://, gs://, https://, pvc://, hf://, oci:// and git repositories; credentials are attached via a ServiceAccount with annotated secrets. That oci:// connects with the second article of the series: the weights can live in the same registry as the images, with the same access control and the same signing capability.

The deployment modes

This is where nearly everything operational is decided. It is selected with the serving.kserve.io/deploymentMode annotation.

	KNative (formerly Serverless)	Standard (formerly RawDeployment)	ModelMesh
Resources emitted	Knative `Service` + `Revision`, `VirtualService` with Istio	`Deployment`, `Service`, `HPA`, Gateway API `HTTPRoute`	Its own high-density runtime
Dependencies	Knative Serving + Istio (or an alternative gateway)	Kubernetes 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1 and a controller	Its own components
Autoscaling	KPA (concurrency / RPS)	HPA or KEDA	—
Scale-to-zero	Yes, with an activator that holds the request	No for HTTP requests	—
`canaryTrafficPercent`	Yes, by revision weights in Istio	Silently ignored (issue #5335, open since 2 April 2026)	—
Sidecars in the pod	`queue-proxy` (+ `istio-proxy`)	None	—
Status as of July 2026	Current	Current and the default mode of the LLM branch	Archived on 14 April 2026

That table has an uncomfortable reading: the two most quoted KServe capabilities, scale-to-zero and canary by percentage, live in the mode that drags Knative and Istio along, while the mode the project has pivoted towards is the other one.

The LLM branch: `LLMInferenceService`

Since v0.16 KServe has a separate CRD for LLMs: LLMInferenceService, in serving.kserve.io/v1alpha1. It is not an extension of the classic InferenceService and does not share its API version, and confusing the two is the first mistake made by anyone arriving from old documentation.

It is built on llm-d, the project that Red Hat, Google, IBM, CoreWeave and NVIDIA donated to the CNCF. The division of roles the project itself declares is clean: KServe is the control plane (lifecycle, scaling and operational governance) and llm-d contributes distributed scheduling, with GPU utilisation, queue depth, cache residency and SLA. In the analogy: llm-d decides which line each train takes; KServe decides which trains exist and when they are withdrawn.

The fields that matter: spec.model (URI, invocation name, scheduling criticality and LoRA adapters, with static adapter reconciliation from v0.19, which connects with multi-LoRA serving); spec.template (single node, or the decode pool); spec.worker (multi-node, which triggers LeaderWorkerSet); spec.prefill (the disaggregation we analysed in disaggregated serving turned into a declarative field); spec.parallelism (tensor, data and expert parallelism); and spec.router (gateway, route, scheduler), where KServe leans on the Gateway API Inference Extension, v1.5.0 of 19 April 2026, already GA, with automatic migration to InferencePool v1 from v0.19.

The endpoint it exposes is OpenAI-compatible: /v1/chat/completions with streaming, and since v0.18 also /v1/responses. That detail underpins the next section: the LLM branch of KServe does not speak V2, it speaks OpenAI.

For multi-node there is the kserve-huggingfaceserver-multinode runtime, based on vLLM over Ray, with workerSpec.tensorParallelSize and workerSpec.pipelineParallelSize. Its restrictions are hard: it requires a ReadWriteMany PVC, it only works in Standard mode, it does not support autoscaling and it requires exactly one head pod. A reference 4×H100 SXM node with NVLink needs none of this, because a 70B fits with tensor parallel 4 on a single node; you need it from there upwards.

The Open Inference Protocol: the contract itself

The routes and the payload

The Open Inference Protocol (historically the “V2 Inference Protocol”) lives in kserve/open-inference-protocol, with REST and gRPC specifications and versioning under SemVer 2.0. Its HTTP surface is deliberately small:

Resource	Verb	Route
Server metadata	GET	`/v2`
Liveness	GET	`/v2/health/live`
Server readiness	GET	`/v2/health/ready`
Model metadata	GET	`/v2/models/<model>[/versions/<v>]`
Model readiness	GET	`/v2/models/<model>[/versions/<v>]/ready`
Inference	POST	`/v2/models/<model>[/versions/<v>]/infer`

The payload is tensorial: id, a list of inputs (each with name, shape, datatype and data) and optionally the desired outputs. The types are BOOL, signed and unsigned integers from 8 to 64 bits, 16-, 32- and 64-bit floating point, and BYTES. In gRPC the same contract is GRPCInferenceService, with six RPCs: ServerLive, ServerReady, ModelReady, ServerMetadata, ModelMetadata and ModelInfer. For high-volume embeddings, the serialisation difference is not cosmetic.

Who actually implements it

The specification repository declares KServe, NVIDIA Triton, Seldon MLServer and Core v2, OpenVINO Model Server, AMD Inference Server and TorchServe as adopters. Not all entries are worth the same as of July 2026:

Triton implements it in a documented way (“Triton exposes HTTP/REST and gRPC endpoints based on the standard inference protocols proposed by the KServe project”), plus its own extensions. It is the de facto reference implementation.
OpenVINO Model Server exposes the KServe API over gRPC and REST, the TensorFlow Serving one and OpenAI-compatible endpoints: the clearest example of a server that speaks both track gauges.
TorchServe forces you to be honest. Its official documentation carries the warning: “This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches.” An adopter on zero maintenance is not an adopter: it is a row nobody has updated.

On the health of the standard, the most eloquent data point is not in any press release: the specification repository does not have a single published release, has accumulated something like thirty commits and seventy-five stars, and its governance consists of monthly meetings and a Slack channel. There is also a “vendor neutral” fork at open-inference/open-inference-protocol with more commits than the original and no traction whatsoever. V2 is a stable contract, widely implemented and barely evolving. For a serialisation format that can be a virtue; for a standard aspiring to cover the generative world, it is a death sentence.

The `generate` extension and why it did not win

The standard did try to cover generation: the specification contains a generate_rest.yaml with two routes, /v2/models/<m>/versions/<v>/generate and .../generate_stream, and a minimalist schema: a request with text_input and parameters (temperature, top_p, max_tokens defaulting to 20, stop, details); a response with text_output, model_name, model_version and, if requested, finish_reason and logprobs; streaming over server-sent events. It is clean, it is enough for a simple completion and nobody uses it: the KServe generative inference documentation, as of July 2026, lists four endpoints, and none of them is generate:

Task	Endpoint in KServe
Text generation	`openai/v1/completions`
Chat	`openai/v1/chat/completions`
Embeddings	`openai/v1/embeddings`
Reranking	`openai/v1/rerank`

The openai/ prefix exists, according to the documentation itself, “to avoid confusion with the V1 API protocol”, and it is changed with KSERVE_OPENAI_ROUTE_PREFIX. It is plumbing that says a lot: the in-house standard was relegated to a defensive prefix while the outside one takes all the traffic.

The tension, unvarnished

V2 solved the problem of 2021, and it solved it well. A tensorial, typed contract with metadata and health checks, in REST and gRPC, is what you need to serve a classifier, a vision model, a ranker or an embedding, and it is still the right answer for that half of the catalogue: swapping an embedding server from MLServer to OpenVINO Model Server without touching clients is the track gauge doing its job.

The LLM world standardised somewhere else, not by committee, but because the entire client ecosystem (SDKs, agent frameworks, gateways, evaluation) was written against the OpenAI API, and any engine that wanted to be used had to speak it. What that means for an architecture:

There is not one contract: there are two, and both are legitimate. Designing as if there were one does violence to half the catalogue.
LLM portability is superficial. It covers chat and completion well; as soon as you use detailed logprobs, guided decoding, priority headers, prefix caching parameters or tool calling with specific schemas, you leave the common subset and get tied back to the engine. It is the warning from structured output: what is portable is the envelope, not always the contents.
The gauge changer is the gateway. What makes the set switchable is the translation and routing layer of choosing an OSS LLM inference gateway and the router as an L7 gateway, not a single standard.
Crossing over loses the versioning. V2 has /versions/<v> in the route; the OpenAI API only has the model field. The version has to be encoded in the name (asistente-70b-2026-07) or managed in the control plane. That is the difference between being able to audit which version answered and not being able to.

Four manifests you can actually apply

Platform runtime for the blog’s reference node, 4×H100 SXM 80 GB with NVLink, with tensor parallel 4:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
 name: vllm-h100-tp4
spec:
 annotations:
 prometheus.kserve.io/port: "8080"
 prometheus.kserve.io/path: "/metrics"
 supportedModelFormats:
 - name: huggingface
 version: "1"
 autoSelect: true
 priority: 10
 protocolVersions:
 - v2
 containers:
 - name: kserve-container
 # ALWAYS pin by digest in production; the tag is only for readability
 image: vllm/vllm-openai:v0.11.0
 command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
 args:
 - --port=8080
 - --model=/mnt/models
 - --served-model-name={{.Name}}
 - --tensor-parallel-size=4
 - --max-model-len=32768
 - --gpu-memory-utilization=0.92
 - --enable-prefix-caching
 - --kv-cache-dtype=fp8
 resources:
 requests: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 limits: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 # the probe that stops the pod dying while the model loads
 startupProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 15
 failureThreshold: 40 # up to 10 minutes of loading
 readinessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 10
 livenessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 30

The InferenceService that consumes it, with KEDA on vLLM metrics:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 serving.kserve.io/deploymentMode: "Standard"
 serving.kserve.io/autoscalerClass: "keda"
spec:
 predictor:
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 # the OCI registry from article 2/4 of this series
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-07"
 autoScaling:
 metrics:
 - type: External
 external:
 metric:
 backend: "prometheus"
 serverAddress: "http://prometheus.monitoring.svc.cluster.local:9090"
 query: 'sum(vllm:num_requests_waiting{model_name="asistente-70b"})'
 target:
 type: Value
 value: "8"

The canary, with its small print:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 # WATCH OUT: canaryTrafficPercent only takes effect in KNative mode.
 # In Standard mode it is silently ignored (issue kserve/kserve#5335, open).
 serving.kserve.io/deploymentMode: "KNative"
spec:
 predictor:
 canaryTrafficPercent: 10
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-08"

KServe keeps three references in the status (LatestReadyRevision, LatestRolledoutRevision and PreviousRolledoutRevision): promoting means setting the percentage to 100, reverting means pinning traffic on the previous one, and a revision that does not become ready automatically receives zero traffic.

And the two calls. Against the V2 contract, for a classifier served with MLServer or Triton:

# metadata: which model it is, what inputs it expects, what types
curl -s http://clasificador.inferencia.example/v2/models/clasificador-tickets | jq .

# readiness of the specific model, not of the server
curl -s -o /dev/null -w '%{http_code}\n' \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/ready

curl -s -X POST \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/infer \
 -H 'Content-Type: application/json' \
 -d '{ "id": "req-4711",
 "inputs": [ { "name": "input-0", "shape": [1, 4], "datatype": "FP32",
 "data": [5.1, 3.5, 1.4, 0.2] } ],
 "outputs": [ { "name": "output-0" } ] }'

Against the same control plane, but on the OpenAI-compatible surface of the LLM:

curl -s -N -X POST \
 http://asistente-70b.inferencia.example/openai/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{ "model": "asistente-70b",
 "messages": [ {"role": "user",
 "content": "Summarise last night'\''s incident report."} ],
 "max_tokens": 256,
 "stream": true }'

Two contracts and two payload shapes, with the same InferenceService as the unit of governance. That is the value proposition of a control plane: the contract is chosen by the model and the management is single.

Autoscaling and the problem of zero

Three autoscalers coexist, and choosing the wrong one is the most common cause of disappointment. KPA, in KNative mode, scales by concurrency or RPS and is the only one that goes down to zero in a usable way, because Knative provides an activator that holds the request while the pod starts; its metric does not distinguish a 30-token request from an 8,000-token one. HPA, in Standard, scales by CPU or memory: for LLMs it is useless, because the CPU of a saturated vLLM pod and that of an idle one look too much alike. KEDA, in Standard, is the right option, the one we develop in autoscaling with KEDA: it is enabled with serving.kserve.io/autoscalerClass: "keda" and autoScaling.metrics accepts external metrics against Prometheus (vllm:num_requests_running or vllm:num_requests_waiting) or, with the OpenTelemetry add-on, pushed metrics. Since v0.18 there is also Workload Variant Autoscaling, which uses HPA or KEDA as a backend.

That leaves the serious problem, which no autoscaler solves: zero. A 70B model in fp8 is on the order of 70 GB to move into HBM, and that path, from disk to HBM, is measured in minutes: the first request after idling takes that long, or it fails.

Worse: the installation documentation is explicit as of July 2026: “Scale from Zero is currently not supported in Standard mode for HTTP requests”. The mode the project has pivoted towards, and the only one that supports KEDA, does not know how to come up from zero on an HTTP request: KEDA can go down to zero (default cooldown of 300 seconds), but the way back requires an activator that Standard does not have. Anyone who wants real zero with HTTP activation still needs Knative.

The levers, in order of effectiveness: do not scale the hot model to zero (minReplicas: 1 for anything with an SLA; zero is for the long tail); attack the load time, the only solution that is not a patch, with the techniques from speeding up cold start; use LocalModelCache, available since v0.15 and extended to LLMInferenceService in v0.19, to preload artifacts into a node PVC; and set correct probes, which is what comes next.

Operational traps

The probe that kills the pod while the model loads. The most frequent and the most expensive to diagnose, because the symptom is a CrashLoopBackOff with no error in the server logs. Until early 2026 the KServe vLLM templates used initialDelaySeconds of 120 to 300 seconds on liveness, with the result described in issue #5062 (12 February 2026, now resolved): the fast models waited too long and the large ones died before finishing loading. The fix applied: a startupProbe of up to ten minutes and probes with no initial delay. A legacy runtime has to be reviewed by hand. In KNative mode there is a worse variant (issue #3795, closed as not planned): what dies is not the model container but the queue-proxy, which does not respond OK within the deadline and takes the whole pod down with it.

canaryTrafficPercent silently ignored. It deserves its own trap because there is no error and no condition in the status to give it away: in Standard mode the field does nothing. The team believes it is serving 10 % with the new version and it is serving 100 %. While issue #5335 remains open, in Standard the canary is done with HTTPRoute weights by hand, or in the gateway.

Canary and scarce GPUs do not get along. A 10 % canary of a 70B with tensor parallel 4 costs a whole 4-GPU node, not 10 % of the resources, because the minimum granularity is a complete replica. During the window there are two versions occupying twice the hardware, or capacity has to be taken from the stable one. That is the difference between a microservice canary and a model canary, and the reason a shadow over duplicated traffic often pays off better.

storageUri and credentials. Storage initializer failures are always the same ones: a ServiceAccount without the annotated secret, a KMS-encrypted bucket with a role that has no decryption permission, an S3-compatible but not exactly S3 endpoint, or intermittent 403s, with a specific fix in v0.18. It is diagnosed in the init container logs, not in the server’s. And one standing recommendation: no downloading over https:// from the internet in production; that is precisely the problem of article 2/4.

The model size against the emptyDir. By default the artifact lands in an ephemeral volume on the node’s disk. Seventy gigabytes per replica fill a system disk with surprising ease, and then the kubelet evicts pods for disk pressure, including ones that have nothing to do with it. You have to set sizeLimit or use a dedicated pvc://. For multi-node the restriction is harder, with a mandatory ReadWriteMany PVC, which rules out a good part of local block storage and pushes you towards shared file systems, with the consequences covered in storage for AI.

The dependency on Knative and Istio. The real cost of KNative mode: two more control planes to version and correlate with Kubernetes, and two sidecars, queue-proxy and istio-proxy, in every pod, with their CPU and memory consumption and their network hop per request. On a platform that already has a mesh it is a marginal cost; on a cluster dedicated to inference it has to be justified by the capabilities you are actually going to use.

Observability: the name of the metric. The ServingRuntime declares prometheus.kserve.io/port and /path annotations, but the metrics that come out are the engine’s, not KServe’s: vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:gpu_cache_usage_perc. The control plane does not normalise names across runtimes, so a dashboard written against vLLM is no use for Triton, and correlating with DCGM is still your own work. v0.19 raises the HPA/KEDA scaling status into the service conditions and emits events on readiness transitions: exactly what was needed to alert on the control plane and not only on the engine.

Decision map: when KServe, and when it is overkill

The useful comparison is not “KServe versus OME, vLLM Production Stack, Dynamo or llm-d”. Those four are covered in LLM inference operators and compete on a different axis:

Axis	What gets decided	Who competes
Contract and control plane	Which CRD describes a service, which API clients see, how the model is versioned, how the rollout is done, how LLM and non-LLM coexist	KServe, Seldon Core v2
Engine and topology	Which engine runs, how prefill/decode is disaggregated, how routing is done with cache awareness, how multi-node tensor parallel is done	vLLM Production Stack, OME, NVIDIA Dynamo, llm-d

The axes are orthogonal, and LLMInferenceService proves it: KServe did not reimplement distributed scheduling, it leaned on llm-d. Choosing KServe is not choosing instead of an LLM operator: it is choosing the contract layer that operator lives on.

Choose KServe when the catalogue is heterogeneous, LLMs plus embeddings, rerankers, classifiers and vision, which is its decisive advantage and the one none of the four operators has: a service like the one in TEI in production fits into the same vocabulary as the LLM. Also when you need a contract decoupled from the engine because you have clients you do not control; when you need declarative, auditable governance (which artifact, from which origin and with which runtime, in an object versionable in git and subject to admission policies), the piece that makes the rest of this series possible; and when you want a catalogue of platform runtimes separated from the teams’ services.

KServe is overkill when you serve one model, on one node, with up to three replicas: the Deployment from vLLM on Kubernetes is the right answer. When your workload is LLM only and the priority is performance per GPU, because there the value is in the engine. When you are going to use neither scale-to-zero nor canary by revision, because what remains is a Deployment with more YAML. And when you already have an LLM operator and only serve LLMs: putting KServe on top for completeness is adding CRDs without adding decisions.

For an inference factory

First: separate the runtime catalogue from the service catalogue, and do it on day one. The ClusterServingRuntimes belong to the platform; the InferenceServices, to the teams. That boundary turns “serving the thirtieth model” into a five-line change instead of a negotiation about vLLM flags, and it fits with GitOps with Flux: the runtime is reviewed as platform code, the service as product configuration.

Second: accept the two contracts and put the gauge changer where it belongs. V2 for the tensorial world, the OpenAI API for the generative one, and the translation in the gateway, not in every client. And since the OpenAI API does not version in the route, encode the version in the model name and make that name the same identifier that appears in the registry, in the signature and in the traces. That is what makes a request traceable back to the artifact.

Third: measure start-up before promising elasticity. The time from the pod being scheduled to the readiness probe going green, with the real model and from the real origin, decides whether scale-to-zero is a FinOps lever or an incident machine. That number, and not the price of the GPU-hour, is the one to take into the cost conversation.

And here this piece falls short on purpose. KServe gives you the contract, the lifecycle and the rollout, but its whole trust model hangs off a text string: storageUri. The storage initializer downloads whatever is at that destination and puts it in front of the GPU without asking anything else; nothing in the InferenceService guarantees that the artifact is the one that was approved, that it has not changed since the last audit, or that the origin is legitimate. The next article goes there: where the model bytes come from, and why an OCI registry with ORAS, Harbor, MLflow or the Kubeflow Model Registry is the difference between distributing models and scattering them.

Sources

CNCF, KServe becomes a CNCF incubating project (11 de noviembre de 2025) — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
CNCF, KServe project page (incubating, aceptado el 29 de septiembre de 2025) — https://www.cncf.io/projects/kserve/
CNCF TOC, Project Moving Levels Checklist: KServe joining CNCF at Incubation level — https://github.com/cncf/toc/issues/1905
KServe, Releases (v0.15.0, v0.16.0, v0.18.0, v0.19.0) — https://github.com/kserve/kserve/releases
KServe, Inference Protocol V2 (Open Inference Protocol) — https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol
KServe, open-inference-protocol (especificación, adoptantes, gobernanza) — https://github.com/kserve/open-inference-protocol
KServe, open-inference-protocol — generate_rest.yaml — https://github.com/kserve/open-inference-protocol/blob/main/specification/protocol/generate_rest.yaml
open-inference, Vendor neutral fork of kserve/open-inference-protocol — https://github.com/open-inference/open-inference-protocol
KServe, Generative Inference — Runtime Overview (endpoints con prefijo openai/) — https://kserve.github.io/website/docs/model-serving/generative-inference/overview
KServe, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
KServe, Cloud-Native AI Inference at Scale using KServe and llm-d (5 de marzo de 2026) — https://kserve.github.io/website/blog/cloud-native-ai-inference-kserve-llm-d
KServe, Serving Runtime (ServingRuntime y ClusterServingRuntime) — https://kserve.github.io/website/docs/concepts/resources/servingruntime
KServe, Kubernetes Deployment Installation Guide (K8s 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1, sin scale-from-zero en Standard) — https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment
KServe, Canary Rollout Strategy — https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary
KServe, Autoscaling with KEDA — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/keda-autoscaler
KServe, Multi-node/Multi-GPU Inference (PVC RWX, sin autoescalado) — https://kserve.github.io/website/docs/model-serving/generative-inference/multi-node
KServe, Storage Options for Model Artifacts — https://kserve.github.io/website/docs/model-serving/storage/overview
KServe, issue #5335 — Support canaryTrafficPercent for RawDeployment mode via Gateway API HTTPRoute weights (abierta, 2 de abril de 2026) — https://github.com/kserve/kserve/issues/5335
KServe, issue #5062 — Add startupProbe to vLLM main containers (12 de febrero de 2026) — https://github.com/kserve/kserve/issues/5062
KServe, modelmesh-serving (última release v0.12.0 de julio de 2023; repositorio archivado el 14 de abril de 2026) — https://github.com/kserve/modelmesh-serving
Red Hat, Converting ModelMesh and Serverless InferenceServices to RawDeployment (Standard) Mode — https://access.redhat.com/articles/7134025
NVIDIA, Triton Inference Server — Inference Protocols and APIs — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inference_protocols.html
OpenVINO, What is OpenVINO Model Server (API KServe, TFS y endpoints compatibles con OpenAI) — https://docs.openvino.ai/2026/model-server/ovms_what_is_openvino_model_server.html
PyTorch, TorchServe — Notice: Limited Maintenance — https://docs.pytorch.org/serve/
Kubernetes SIG Network, Gateway API Inference Extension (v1.5.0, 19 de abril de 2026) — https://github.com/kubernetes-sigs/gateway-api-inference-extension

How we use our infrastructure to meet defence standards with AI

Thu, 18 Jun 2026 11:00:00 +0200

In defence, the challenge of putting AI into operation is not the model: it is security and compliance. The rules that apply, the allied interoperability standards, the data-centric security rules, the accreditation frameworks and the principles of responsible use, impose concrete requirements on access control, classification, confidentiality, traceability, supply chain and sovereignty. A demo on a laptop connected to the internet ignores all of that; an operable platform has to satisfy it, and has to be able to prove it to an accreditation body.

This article is not a tour of technologies. It is the opposite: we take the requirements imposed by the standards of the sector and, for each one, we show how our infrastructure is used to meet it. The infrastructure is the same real platform we use in other sectors, two RKE2 clusters (Kubernetes v1.35) with Cilium as the network, GitOps with Flux, Keycloak as the identity server, inference on our own GPUs and, in pre-production, scheduling with Volcano, adapted to an environment with stricter security requirements. Every component is here because it solves a regulatory requirement, not because it looks good on a diagram. And, like all infrastructure, it is measured by efficiency and cost, which is what makes compliance sustainable.

It is worth starting with an idea many people are unaware of: standards for this already exist in the sector, and nothing has to be invented. There are allied interoperability standards, there is a defence-specific DevSecOps reference model on Kubernetes, there is a security framework (Zero Trust) and a supplier accreditation scheme, and there is a body of principles for the responsible use of military AI. Let us first lay out that map, and then walk the requirements one by one.

The map of standards in the sector

Four families of standards intersect in a defence AI platform, and understanding what each one covers avoids reinventing wheels.

Allied interoperability. In a coalition, systems have to understand each other. Federated Mission Networking (FMN) defines how mission networks are federated across nations through successive “spirals”; the MIP (Multilateral Interoperability Programme) and its JC3IEDM data model standardise command and control; Link-16 and formats such as NVG (NATO Vector Graphics) cover tactical and situational exchange. They are, in defence, the equivalent of what FHIR and DICOM are in healthcare: the common language without which AI is an isolated system that cannot be integrated into the operation.

Data-centric security: STANAG 4774 and 4778. This is the most distinctive piece of the sector. STANAG 4774 defines the syntax of a confidentiality label (policy, classification, security categories), and STANAG 4778 defines the binding mechanism that ties that label to the data throughout its whole life cycle and between the parties that share it. Instead of protecting only the perimeter or the system, the information object itself is protected: the classification travels with the data and the access decision is taken against the label. Both STANAGs are in the process of ratification by the nations and are moving from recommended practice to a requirement for mission-critical software.

Defence DevSecOps on Kubernetes. To build and deploy defence software there is a very concrete reference model: the DoD Enterprise DevSecOps Reference Design on Kubernetes, embodied in platforms such as Platform One and its Big Bang product line (declarative continuous delivery of hardened packages on a Kubernetes cluster), supported by Iron Bank, the repository of hardened container images compliant with the DISA STIGs, continuously scanned. The associated philosophy is cATO (Continuous Authority To Operate): accreditation stops being a one-off milestone and becomes a state maintained through continuous monitoring. For defence, it is the equivalent of the Kubernetes-native standard that MONAI represents in medical imaging: an agreed way to package and operate software with guarantees about the substrate we already use.

Security and accreditation frameworks. The DoD Zero Trust Reference Architecture (v2.0) sets the guiding principle, do not trust by default, always verify, with adoption targets towards 2027. Supplier accreditation is governed by CMMC 2.0 (whose Phase 2, with mandatory third-party certification for level 2, arrives in November 2026) over the 110 controls of NIST SP 800-171. In Spain and the EU, the Esquema Nacional de Seguridad (the Spanish National Security Framework) at high category and the CCN-STIC guides of the Centro Criptológico Nacional (the Spanish National Cryptologic Centre) play an equivalent role for systems handling sensitive or classified information.

Responsible AI in defence. NATO adopted six Principles of Responsible Use (PRU) for AI in defence: lawfulness, responsibility and accountability, explainability and traceability, reliability, governability and bias mitigation, reinforced by its AI Strategy revised in 2024 and by the DARB (Data and AI Review Board). An important legal nuance: the EU AI Act expressly excludes (art. 2.3) AI systems intended exclusively for military, defence or national security purposes. That is, in defence the responsible AI framework is not the AI Act, but NATO’s PRU, national rules and, as a management system, ISO/IEC 42001.

With this map on the table, let us walk the requirements.

Requirement: coalition interoperability (FMN, MIP/JC3IEDM)

In an allied operation, a system that does not integrate is useless however good it is. The requirement, derived from FMN, from the command and control data model (MIP/JC3IEDM) and from the tactical formats (Link-16, NVG), is that AI must consume and produce information in the common formats of the coalition, not in a dialect of its own.

How we meet it: just as in other sectors we put an interoperability layer at the boundary, here the platform integrates through the agreed gateways and formats, so that AI never talks directly to the mission systems but consumes and delivers information already normalised to the coalition standard. This keeps AI as a component that couples into the federated architecture, respecting its profiles and its data governance, instead of a silo that forces bespoke integrations at every deployment. Interoperability is also a precondition for traceability and labelling: only if the data enters and leaves in a known format can its confidentiality label be attached and preserved.

Requirement: do not trust by default, Zero Trust and verified access (DoD ZT RA, NIST 800-171)

The guiding principle of modern defence security is Zero Trust: no entity, user, service or node, is trusted because of its position in the network; every access is authenticated, authorised and continuously verified. NIST 800-171 details the access and identification controls that underpin it.

How we meet it: a single identity server, Keycloak, governs authorisation with OAuth 2.1 and OpenID Connect, issuing short-lived tokens and binding them to their recipient with audience validation (aud), so that a credential issued for one service is useless against another. Between components handling sensitive information we require mutual TLS (mTLS): it is not only the user or the service that is authenticated, but the workload itself taking part in the exchange. And the network operates in default-deny (next section), so that connectivity is not a right conferred by being inside the cluster, but an explicit permission. Workloads use service accounts with no permissions by default, and administrator access to the Kubernetes API is federated against Keycloak over OIDC with scoped roles: technical access too is authenticated, limited and audited. Zero Trust is not a product you buy; it is a property that emerges from strong identity, least privilege and continuous verification, and that is how the platform is built. Verification, moreover, does not stop at sign-in: tokens are short-lived and are renewed, policies are re-evaluated on every request, and the posture of each workload, signed image, applied security policy, determines whether it can run and communicate. A component that stops complying with its policy stops being able to operate; it is granted no grace period for having been inside.

Requirement: data-centric security and labelling (STANAG 4774/4778)

Unlike other sectors, in defence the classification travels with the data. The requirement, derived from STANAG 4774/4778, is that every information object carries its confidentiality label and that access decisions are taken against that label, not against the system hosting it.

How we meet it: authorisation decisions are not limited to roles, but incorporate attribute-based access control (ABAC) evaluating the classification label of the data against the clearance of the requester. The policy is expressed as code and evaluated at a central decision point (a policy engine of the OPA/Open Policy Agent kind) that the application layer and the gateways consult before serving a resource. The label, in accordance with STANAG 4774, travels with the object, and the binding under STANAG 4778 is preserved across the platform, in the object metadata in storage and in the annotations that travel with the request. For an AI agent, this means that its token gives it access to a classification level, and that data above that level is simply not handed to it, regardless of where it is stored.

This approach embodies the classic no-read-up rule: an entity with a given clearance does not access information above its level, and the decision is taken on the object’s label, not on assumptions made by the system. When a workflow requires moving information to a lower level, a declassification or sanitisation, it is an explicit, audited step, with its own policy, never a side effect of a query. The label is preserved in the object metadata in storage and in the annotations accompanying the request, so that the STANAG 4778 binding is not lost when crossing the components of the platform.

Requirement: segregation by mission and level (multitenancy, NIST 800-171, ENS)

Different missions, different classification levels and different compartments cannot share a common space without controls. The requirement: strong isolation, so that what belongs to one mission or level is not accessible from another.

How we meet it: each mission or compartment lives in its own Kubernetes namespace, with scoped RBAC, ResourceQuota and LimitRange, dedicated storage and databases per operator, not shared behind an application filter, and its own network segmentation. Identity is segregated by realm in Keycloak. The whole definition lives in Git and is reconciled by Flux (GitOps), so that segregation is declarative and auditable through change history. Where the classification level demands it, separation goes beyond the cluster: operation runs on physically distinct clusters or sites, and exchange between classification domains, when necessary, is done through controlled cross-domain solutions, not by opening a hole between namespaces. At the highest levels, each security domain can be an independent cluster with its own chain of trust, its own identities and its own governance, preventing a single control plane from spanning classifications that must never be mixed. The choice between a separate namespace, cluster or site is itself an accreditation decision: it is documented, justified according to the level and compartment, and reflected declaratively in the repository.

Requirement: confidentiality, encryption and exfiltration prevention (NIST 800-171, ENS, Zero Trust)

NIST 800-171 (system and communications protection families) and the ENS require confidentiality, encryption of communications and prevention of information leakage. In defence, exfiltration is not an abstract risk: it is the adversary’s objective.

How we meet it: the network is managed by Cilium (eBPF) with a default-deny model. A pod has no connectivity beyond what a policy grants it explicitly, flow by flow, which turns exfiltration into something blocked in the kernel. A component can only talk to strictly necessary destinations:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: inferencia-egress-minimo
 namespace: mision-alfa
spec:
 endpointSelector:
 matchLabels:
 app: serving-inferencia
 egressDeny:
 - toEntities: [world]  # no egress outside the cluster
 egress:
 - toEndpoints:
 - matchLabels:
 app: gateway-modelos
 - toEndpoints: # internal DNS, without this it does not resolve
 - matchLabels:
 k8s:io.kubernetes.pod.namespace: kube-system
 k8s-app: kube-dns
 toPorts:
 - ports: [{ port: "53", protocol: UDP }]

Two nuances separate a correct policy from a leaky one: you have to explicitly allow DNS and filter by endpoint identity, not by CIDR, because in an identity-based network filtering by IP ranges does not behave the way intuition suggests. On top of this we add flow observability with Hubble and runtime security with Tetragon (eBPF), which watches process execution, file access and connections inside the container, making it possible to detect and block anomalous behaviour; an unexpected process in a sensitive workload is exactly the signal you want to see in real time. Encryption in transit is covered by cert-manager (TLS at the edges) and Cilium’s transparent node-to-node encryption. Added to this are the Pod Security Standards in restricted profile (no privileges, no root) and disciplined management of secrets and keys, which do not live in plain manifests.

Requirement: verifiable software supply chain (Iron Bank/STIG, SBOM, CMMC)

The software supply chain is a first-order attack vector, and the sector treats it as such: images hardened in line with the STIGs, a component inventory (SBOM) and cryptographic verification of provenance. CMMC and NIST 800-171 require it in the supplier flow; Executive Order 14028 popularised the SBOM.

How we meet it: we start from hardened base images (in the spirit of Iron Bank and the DISA STIGs), we generate an SBOM (CycloneDX/SPDX) per artefact and we sign images and SBOMs with Sigstore/Cosign, a graduated CNCF project, with keyless signing and a transparency log. In the cluster, an admission control (Kyverno/policies) rejects any image that is not signed by a trusted identity, so software whose provenance cannot be verified never runs. The rule is expressed as code and verifies the signature of every image before allowing it to run:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: exigir-imagenes-firmadas
spec:
 validationFailureAction: Enforce
 rules:
 - name: verificar-firma-cosign
 match:
 any:
 - resources:
 kinds: [Pod]
 verifyImages:
 - imageReferences:
 - "registro.interno/*"
 attestors:
 - entries:
 - keys:
 publicKeys: |-
 -----BEGIN PUBLIC KEY-----
 ...trusted key...
 -----END PUBLIC KEY-----

An image without a valid signature is rejected at admission, before it runs. Since the whole deployment is GitOps, the chain, from the commit to the signed image to the policy that admits it, is traceable end to end. This satisfies the supply chain control of NIST 800-171 and fits the cATO model: verification is continuous, not a one-off stamp.

Requirement: sovereignty and disconnected or air-gapped operation (classified networks, DDIL)

Defence networks frequently operate isolated from the internet (air-gapped) or under DDIL conditions, degraded, intermittent or low-bandwidth connectivity at the tactical edge. And the data cannot leave to an external API under any circumstances. The requirement: the platform has to work without internet and without taking the data outside.

How we meet it: inference is served locally, on our own GPUs, with a high-performance serving engine (vLLM) and open-weight models that can be hosted entirely inside the perimeter. Everything the platform needs, container images, models, dependencies, is mirrored into internal registries and repositories, so that the GitOps deployment reconciles from internal sources and requires no internet access. A model gateway centralises access and reverses the burden: the default behaviour is local inference, and any egress is an exception that, besides having to be enabled explicitly, would collide with the default-deny network policies. Sovereignty does not rest on a promise, but on several mutually reinforcing controls, gateway, network policy and isolation, so that a configuration slip does not translate into a leak.

One point that cannot be overlooked in defence: no component should “phone home”. Many tools send telemetry, check licences or download updates against external servers by default, and in a classified network that is unacceptable. That is why default-deny is not only a measure against the adversary, but also against the software itself: if an image attempts an undeclared telemetry connection, the policy blocks it and Tetragon logs it. The absence of phone-home is verified, not trusted.

Requirement: resilience and operation at the tactical edge (DDIL)

Much defence operation happens under DDIL conditions, degraded, intermittent or low-bandwidth connectivity, and even in total isolation. The requirement: the platform must remain useful when the link to the rear falls, and reconcile when it returns.

How we meet it: the declarative GitOps model fits disconnected operation naturally. An edge site keeps its own copy of the desired state and reconciles against local registries and repositories, so it operates without depending on a permanent connection; when the link returns, it synchronises. Inference is served locally with models already hosted on the node, with no external calls. Traces and audit events are stored locally and forwarded (store-and-forward) when there is a link, without losing the record. The high-availability control plane and the separation into sites allow controlled degradation instead of collapsing as a block. Resilience is not an add-on: it is a property of the declarative design and of sovereign inference.

Requirement: robustness and protection of the AI system itself (PRU reliability)

NATO’s reliability principle requires AI systems to have explicit and well-tested capabilities, and to be secure against manipulation. This adds a requirement that other sectors treat with less urgency: protecting the model itself, its weights and its behaviour, as a sensitive asset.

How we meet it: model weights are treated as sensitive data, hosted inside the perimeter and with access controlled by the same identity and network policies as everything else. The model gateway is the point where input and output safeguards are applied (validation, limits, filtering) and where a model can be constrained or disconnected. The signed supply chain (SBOM and Cosign) extends to model artefacts, so that only what has verifiable provenance is executed. And traceability feeds continuous evaluation: measuring the model’s behaviour over time is what makes it possible to detect degradation or manipulation before it affects the operation. Robustness is built with the same controls, identity, network, supply chain, traces, now applied to the model as well. Added to this are safeguards against adversarial inputs, prompt injection, manipulated data, at the gateway, which validates and constrains what goes into and out of the model. In defence, assuming the adversary will try to manipulate the AI is not pessimism: it is the design’s starting assumption.

Requirement: reliable execution of mission workloads and efficiency (availability, sustainability)

Serving AI in-house is only sustainable if the GPU, a scarce and expensive resource, is used well, and certain mission workloads must run reliably and to completion. The default Kubernetes scheduler schedules pod by pod, which for distributed workloads produces half-started jobs that hold GPUs without making progress.

How we meet it: GPU allocation is governed by Volcano (a CNCF incubating project), today in pre-production, with gang scheduling (PodGroup with minAvailable: the job starts complete or does not start), per-mission queues with quota and priority, fair-share (DRF) to share out fairly, pre-emption so that an urgent workload can overtake a deferrable one, and topology awareness (NVLink/PCIe) to reduce fragmentation. This raises the effective utilisation of the GPU fleet, which is what makes sovereign inference viable. On the state of the art: Kubernetes 1.34 took DRA (Dynamic Resource Allocation) to general availability, accelerator allocation by attributes instead of by count, and Kueue covers job quota management; both combine with a gang scheduler. It is managed with metrics: effective versus allocated utilisation, fragmentation, throughput per GPU and GPU-hours per mission. At the edge, where the GPU is even scarcer, these same techniques, together with quantised and smaller models, are what make it possible to serve useful inference with the hardware available instead of depending on a datacenter that may not be reachable.

Requirement: explainability, traceability and governability (NATO PRU, NIST AU)

Three of NATO’s six Principles of Responsible Use bear directly on infrastructure: explainability and traceability, governability and responsibility/accountability. Plus the audit (AU) controls of NIST. The requirement: being able to reconstruct every AI decision, keep human control and be accountable.

How we meet it: every inference is recorded as an auditable trace with Langfuse, input, output, model and version, cost, latency and mission, stored in our own components locked behind network policies that prevent it from leaving. That trace provides the traceability required by the PRU and by the NIST AU controls, and is the basis for human oversight, which is exercised at the gateway (review, limits and, crucially, the possibility of intervention and disconnection demanded by the governability principle: an AI system in defence must be able to be deactivated or constrained by the operator). An audit log is only worth anything as evidence if it is intact and ordered in time: that is why events are kept in append-only storage and clocks are kept synchronised by NTP. Instrumentation converges on the OpenTelemetry GenAI semantic conventions.

Accountability also requires being able to attribute a decision: which model, which version, which input and which operator were involved. That is why each trace links the inference to the identity that requested it and to the exact version of the model artefact that answered, so that the chain “who asked what, with which system and with what result” can be reconstructed. Without that attribution, NATO’s responsibility and accountability principle remains a statement of intent.

Requirement: continuous accreditation and reproducible operation (cATO, CMMC, ENS)

The sector has replaced one-off accreditation with cATO: an authorisation that is maintained as long as it is continuously demonstrated that the controls remain in place. The ENS and CMMC likewise require configuration management and traceability of changes.

How we meet it: the entire platform state, deployments, network policies, quotas, RBAC, identity, admission policies, is described in Git and applied by Flux over GitOps. Every change is a revision with an author, a date and a reviewer; every environment is reproducible from its repository. Security policies are code (Kyverno/OPA) evaluated continuously, not a list someone reviews once a year, and compliance monitoring is permanent. This is what cATO looks like in practice: an audit stops being a snapshot and becomes a query against a state that is maintained and verified without interruption, with the ability to reconstruct exactly what was deployed on a given date. A practical benefit is that compliance evidence generates itself: the declarative state in Git, the admission records, the image signatures and the traces are, together, the body of evidence an accreditation needs, produced continuously instead of gathered by hand the week before the inspection.

The piece that ties it all together: ISO/IEC 42001 and NATO’s principles

Everything above consists of technical controls that satisfy concrete requirements. But a management system is needed to order, evaluate and improve them, with defined roles and responsibilities. In defence, that governance has two complementary references: NATO’s Principles of Responsible Use, which set the ethical and operational “what”, and ISO/IEC 42001, the certifiable standard for AI management systems, which supplies the “how” of continuous management. Given that the AI Act excludes defence, this combination, PRU plus ISO 42001, is the governance backbone for responsible, demonstrable military AI. It is, precisely, the content of the AI Management Systems course we will publish shortly.

A view of the sector as of June 2026

It is worth pinning down the snapshot of the moment, because several currents have converged and they condition any platform decision.

An “AI-first” agenda and AI on classified networks. In January 2026, the US Department of Defense set an AI agenda at “war speed”, and through the spring agreements were closed to deploy generative AI models on classified networks. The signal is clear: military AI is moving from pilot to operation, which raises the demands on the infrastructure that supports it.

The substrate is hardened Kubernetes. Platform One, Big Bang and Iron Bank consolidate the pattern: Kubernetes clusters with STIG-compliant images, a signed supply chain and cATO. Defence AI is deployed on that same cloud-native substrate, not on bespoke systems.

Zero Trust with a deadline. Zero Trust adoption targets sit around 2027, which turns strong identity, microsegmentation and continuous verification into requirements with a calendar, not aspirations.

Supplier accreditation is tightening. CMMC Phase 2, in November 2026, ends self-attestation for anyone handling controlled information: third-party certification against the 110 NIST 800-171 controls will be required. The software supply chain, SBOM and Sigstore signing, moves from good practice to a contractual condition.

Data-centric security, operational at last. With STANAG 4774/4778 advancing through ratification, confidentiality labelling bound to the data becomes a real requirement for mission software, and not just a recommendation.

Defence-specific AI governance. NATO’s Principles of Responsible Use and the DARB are maturing as the reference framework, occupying the space the AI Act leaves free by excluding defence. And GPU economics, together with the maturity of open-weight models, pushes compute towards in-house infrastructure, precisely where classified data and disconnected operation demand it.

Europe pushes its own autonomy. In parallel, the EU is strengthening its defence industrial and technological base, through the European Defence Fund and the European Defence Agency, with growing emphasis on technological sovereignty. For a European platform, this reinforces the bet on in-house infrastructure, locally hostable models and minimal dependence on external services: exactly the direction it is built in.

Where we stand in that picture: with the platform on Kubernetes, eBPF as the network and security substrate, a signed supply chain, GPU scheduling maturing in pre-production, sovereign and disconnectable inference by design, traceability in auditable format and the management framework (PRU plus ISO 42001) as the next step. We are not chasing fashion; we are on the line the sector has decided to advance along.

Summary: requirement by requirement

Each rule or standard in the sector imposes a requirement, and each requirement is satisfied by a concrete piece of the infrastructure. Zero Trust and verified access, with Keycloak, mTLS and default-deny. The data-centric security of STANAG 4774/4778, with ABAC over classification labels. Segregation by mission and level, with multitenancy and, where appropriate, physical separation and cross-domain. Confidentiality and leak prevention, with Cilium in default-deny and Tetragon. A verifiable supply chain, with STIG-hardened images, SBOM and Sigstore signing plus admission control. Sovereignty and disconnected operation, with local inference over open models and internal registries. Reliable execution and efficiency, with Volcano. The explainability, traceability and governability of NATO’s PRU, with auditable traces and human control at the gateway. Continuous accreditation (cATO) and reproducible operation, with GitOps and policy as code. And the governance that ties it all together, with NATO’s PRU and ISO 42001.

It is not a PowerPoint: it is infrastructure that operates, that complies and that, being infrastructure, is measured by efficiency and cost. That, the cost control of an AI platform, is what the next article is about.

Sources

How we use our infrastructure to meet healthcare standards with AI

Thu, 18 Jun 2026 10:30:00 +0200

In a healthcare setting, the challenge of putting AI into production is not the model: it is compliance. The rules that apply, both the technical interoperability standards and the law, impose concrete requirements on access control, confidentiality, traceability, data location and governance. A demo on a laptop ignores all of that; a real platform has to satisfy it, and has to be able to prove it to an auditor.

This article is not a tour of technologies. It is the opposite: we take the requirements the rules impose and, for each one, we show how our infrastructure is used to meet it. The infrastructure is real, two RKE2 clusters (Kubernetes v1.35) with Cilium as the network, GitOps with Flux, Keycloak as the identity server, inference on our own GPUs and, in pre-production, scheduling with Volcano, and every component is here because it solves a regulatory requirement, not because it looks good on a diagram. Like all infrastructure, moreover, it is measured by efficiency and cost, which is what makes compliance sustainable instead of an expense nobody can bear.

It is worth starting with an idea many people are unaware of: standards for this already exist in the sector, and nothing has to be invented. There are clinical interoperability standards (HL7, FHIR, DICOM and the IHE profiles that combine them), there is a Kubernetes-native standard for packaging and running medical imaging AI (MONAI Deploy and its MONAI Application Package), and there is an emerging standard for what a cluster must look like to run AI with guarantees (the CNCF Kubernetes AI Conformance). Let us first lay out that map of standards, and then walk the requirements one by one.

The map of standards in the sector

Three families of standards intersect in a healthcare AI platform, and understanding what each one covers avoids reinventing wheels.

Clinical interoperability: HL7, FHIR, DICOM and IHE. The base formats are well known, HL7 v2 for legacy messaging, FHIR for modern clinical data over REST, DICOM/DICOMweb for imaging. What many people overlook is that there is a body, IHE (Integrating the Healthcare Enterprise), which does not invent formats but defines profiles: concrete combinations of those standards to solve a use case in an interoperable way. Profiles such as XDS/XCA (document exchange), PIX/PDQ (patient identification and query) and, crucial for security, ATNA (Audit Trail and Node Authentication), which specifies the four pillars of a secure node: node authentication, user authentication, audit logging and encryption of communications. And it rests on a sibling profile, Consistent Time (CT), which synchronises the clocks of all nodes: without reliable time, an audit log is worthless as evidence, because what happened can be neither ordered nor correlated. ATNA has also been modernised with RESTful ATNA and the BALP (Basic Audit Log Patterns) profile, which express auditing as FHIR AuditEvent resources. IHE is, in practice, the standard that says how FHIR and DICOM are combined securely and auditably; it is the natural reference for designing compliance controls.

Medical AI on Kubernetes: MONAI Deploy and the MAP. For the specific problem of packaging, distributing and running medical imaging AI applications there is a de facto standard: MONAI Deploy, part of the MONAI project (an open source deep learning framework for healthcare imaging, from the PyTorch ecosystem), and its MONAI Application Package (MAP). A MAP is a container image that complies with a specification defined by the MONAI Deploy working group, experts from more than a dozen medical imaging institutions, and, according to that specification, a MAP must support deployment on Kubernetes and integrate with DICOM and FHIR for data exchange. That is, there is a standard format for “a medical AI app” that runs natively on Kubernetes, which is exactly our substrate. This matters for compliance because a MAP encapsulates the model, its dependencies and its input/output contract in a versioned, reproducible form. Around the MAP, MONAI Deploy also supplies the orchestration pieces a clinical environment needs: an Informatics Gateway that receives DICOM studies and triggers inference, and a Workflow Manager that chains the steps of a pipeline on Kubernetes. It is not just a packaging format: it is a standard way to take a medical imaging app from the registry to the PACS without bespoke integration at every hospital.

AI-ready clusters: CNCF Kubernetes AI Conformance. In November 2025 the CNCF launched the Certified Kubernetes AI Conformance Program, which standardises how AI workloads run on Kubernetes through a set of requirements, the KARs (Kubernetes AI Requirements), covering GPU integration, volume management and job-level networking, aligned with Kubernetes v1.35. And what is most relevant for healthcare: the programme announced that during 2026 it will be extended with a Sovereign AI track, focused on hardened sandboxing and data privacy. It is the industry-level formalisation that a cluster running serious AI must meet minimum guarantees, and that data sovereignty is now a first-class category, not a preference.

With this map on the table, let us walk the requirements.

Requirement: interoperability on standards (EHDS, MDR, IHE)

The EHDS requires health data to be exchanged in standard formats, and its profiling is built on FHIR R4; the IHE profiles define how that exchange is done interoperably; the MDR requires data traceability when the software is a medical device. The requirement: the platform cannot speak a dialect of its own.

How we meet it: we put an interoperability layer as the boundary between the clinical domain and the inference domain. A FHIR server (R4, because it is the normative version and the one the EHDS profiles; R5 has marginal adoption and the sector is waiting for R6, in ballot since January 2026) exposes the clinical resources; a DICOMweb node serves imaging through QIDO-RS, WADO-RS and STOW-RS; and a gateway transforms incoming HL7 v2 into FHIR. On that base we apply the relevant IHE profiles: patient identification with PIX/PDQ so identities are not crossed between sources, and document exchange with XDS/XCA where appropriate. The AI model never touches the clinical database directly: it consumes FHIR and DICOMweb through this layer, which is where data is normalised, profiled and access-controlled. When we deploy an imaging AI application, the aim is to package it as a MAP, so that its contract with DICOM/FHIR is the sector standard and not a bespoke integration.

Requirement: AI results return to the clinical workflow in standard form (DICOM, FHIR, MDR)

Clinical AI does not end when it produces an output: that output has to return to the system in a format the clinician and the record understand, and traceably (MDR). Returning a PDF or a proprietary JSON breaks interoperability and leaves the result outside the clinical record, where nobody audits or reuses it.

How we meet it: results from imaging models are re-injected as standard DICOM objects, DICOM SR (Structured Report) for findings and measurements, DICOM SEG for segmentations, or Secondary Capture for visual representations, through STOW-RS, so that the PACS and the viewer treat them like any other series in the study. Results unrelated to imaging are expressed as FHIR resources (Observation, DiagnosticReport) linked to the patient and the originating study. This way the AI result is a first-class citizen of the clinical record, referenceable, versioned and auditable, not a loose artefact on a disk. Packaging the application as a MAP helps here too: the output contract towards DICOM/FHIR is part of the package specification itself, not a different piece of glue script at every deployment.

The GDPR imposes minimisation (art. 5) and reinforced protection of health data as a special category (art. 9); the ENS (Esquema Nacional de Seguridad, the Spanish National Security Framework) requires identification and access control; the EHDS, control over who accesses data and for what; and IHE ATNA requires node authentication and user authentication as pillars of a secure node. Translated into infrastructure: every access to clinical data authenticated, authorised to the minimum and bound to a purpose; and every participating node identified.

How we meet it: a single identity server, Keycloak, governs all authorisation with OAuth 2.1 and OpenID Connect, profiled by SMART on FHIR:

Resource- and purpose-oriented scopes. SMART distinguishes patient/, user/ and system/, and its version 2 separates permissions by operation (create, read, update, delete, search). An AI agent receives the minimum, for example system/ImagingStudy.rs (read and search studies only). It is GDPR minimisation implemented in the token, and the user authentication that ATNA demands.
Service identity without shared secrets. Unattended workloads use private_key_jwt (an assertion signed with a private key).
Audience binding. The token carries the URL of the destination server in aud and that server validates it, preventing lateral reuse of credentials; it relies on FHIR-specific Keycloak extensions.
Node authentication (ATNA) with mTLS. Between nodes handling clinical data we require mutual TLS, so that it is not only the user or the service that is authenticated, but the machine or service itself taking part in the exchange.

DICOMweb is protected with the same token and the same Keycloak. The result is a single identity plane for FHIR, DICOMweb and the MCP servers: one place to demonstrate who can access what.

The same principle of least privilege is applied one level below, to the Kubernetes plane itself, which the ENS also reaches. Operators do not connect with a universal cluster-admin: administrator access to the cluster API is federated against Keycloak over OIDC, with roles scoped by namespace, so that technical access too is authenticated, limited and audited. And workloads use service accounts with no permissions by default, granted only the strict minimum. Whoever administers the platform is subject to the same identity regime as whoever consumes clinical data through it.

The GDPR (art. 32) requires confidentiality, integrity, availability and resilience, with measures such as data separation and encryption; the ENS details segregation and communications protection controls; ATNA requires encryption of communications between nodes. In a multi-customer platform this translates into two things: that one centre’s data is not visible to another, and that a compromised component cannot get data out.

How we meet it for tenant isolation: each tenant lives in its own namespace, with scoped RBAC, ResourceQuota and LimitRange, a dedicated database per operator (CloudNativePG), not a shared database behind an application filter, and its own object storage space. Identity is segregated by realm in Keycloak. Everything is defined in Git and reconciled by Flux (GitOps): separation is declarative and auditable through change history. Faced with “prove to me that centre A cannot see centre B’s data”, the answer is a commit.

How we meet it for network confidentiality and exfiltration prevention: the network is managed by Cilium (eBPF) with a default-deny model. A pod has no connectivity beyond what a policy grants it explicitly, flow by flow, which turns exfiltration into something blocked in the kernel. For example, the traceability component can only talk to its analytical database, its PostgreSQL, the object storage, the cache and DNS:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: trazabilidad-egress-minimo
 namespace: trazabilidad
spec:
 endpointSelector:
 matchLabels:
 app: trazabilidad-web
 egressDeny:
 - toEntities: [world]  # nothing leaves the cluster by default
 egress:
 - toEndpoints:
 - matchLabels:
 app: postgres
 - toEndpoints:
 - matchLabels:
 app: analitica
 - toEndpoints: # DNS, without this the pod does not resolve
 - matchLabels:
 k8s:io.kubernetes.pod.namespace: kube-system
 k8s-app: kube-dns
 toPorts:
 - ports: [{ port: "53", protocol: UDP }]

Two nuances separate a correct policy from a broken or leaky one: you have to explicitly allow DNS (otherwise the pod does not work) and filter by endpoint identity, not by CIDR, because in an identity-based network filtering by IP ranges does not behave the way intuition suggests. On top of this we add flow observability with Hubble and runtime security with Tetragon (eBPF), which watches process execution, file access and connections inside the container, making it possible to detect and block anomalous behaviour in workloads handling sensitive data. For the encryption in transit required by art. 32 and ATNA, cert-manager manages TLS at the ingresses and Cilium can transparently encrypt traffic between nodes.

To this are added two baseline controls that art. 32 and the ENS take for granted. First, the Pod Security Standards in restricted profile, containers without privileges, not running as root and with a read-only filesystem where possible, which reduce the surface of a compromised container. Second, disciplined management of secrets and keys: the private_key_jwt private keys, the certificates and the database credentials do not live in plain manifests in Git, but are managed and rotated, with cert-manager issuing and renewing certificates automatically. A secret leaked into a repository is a potential data breach, and is treated as such from the design stage.

The GDPR does not only limit who accesses data: it limits how much data is processed (minimisation, art. 5) and how long it is kept (storage limitation), and it promotes pseudonymisation as a protection measure (art. 32). The requirement: do not accumulate identifiable clinical data longer than necessary, and de-identify where the use case allows.

How we meet it: the interoperability layer is also the control point for the data life cycle. Pseudonymisation is applied there when the AI workflow does not need direct identity, for example evaluation or tuning over cohorts, keeping the re-identification table separate from the rest of the system and behind its own access policies. One principle we apply strictly: traces and logs contain no clinical data. They record resource identifiers, model, cost, latency and tenant, not the content of the record or the image, so that the observability layer does not become a second store of sensitive data that has to be protected and audited twice over. And retention policies are applied per data type, traces, caches, intermediate copies, with automatic expiry, so that the storage period is a declared, verifiable property, not an oversight accumulating risk on disks nobody reviews.

Processing special-category health data by sending it to a third party’s inference API, outside the controller’s control, is hard to justify under the GDPR and contrary to the spirit of the EHDS. And it is not only a legal requirement: the Sovereign AI track of the CNCF AI Conformance raises it to a first-class technical category for 2026. The implicit requirement is that clinical data must not leave the perimeter.

How we meet it: inference is served locally, on our own GPUs, with a high-performance serving engine (vLLM). Imaging AI applications are packaged, ideally, as MAPs, which run natively in the cluster without taking data outside. In front of them we put a model gateway through which all inference traffic passes: it is the point where it is decided what runs inside and where, so that clinical data does not leave by default and any exception is explicit and logged. Whether this is viable, and not a luxury, depends on efficiency: to serve in-house at a reasonable cost you have to squeeze the hardware, and that is where scheduling comes in. Sovereignty stops being a promise on a slide and becomes a property of how the inference is built.

There is a nuance worth spelling out, because it marks the difference between “we say the data does not leave” and “the data cannot leave without a record”. The gateway reverses the burden: the default behaviour is local inference, and any egress towards an external service is an exception that has to be enabled explicitly, that is logged and that, in addition, collides with the default-deny network policies if the corresponding destination has not been opened. That is, sovereignty does not rest on a design decision alone, but on several mutually reinforcing controls, gateway, network policy and logging, so that a configuration slip does not translate into a silent leak. For special-category data, that difference between trust and guarantee is exactly what the GDPR asks for when it speaks of data protection by design and by default (art. 25).

Requirement: reliable execution of critical workloads and efficiency (ENS, CNCF AI Conformance)

Keeping inference in-house is only sustainable if the GPU is used well, and certain workloads (an overnight batch of image processing) must run reliably and to completion. The CNCF AI Conformance KARs also formalise GPU integration and job-level networking requirements that a serious AI cluster must meet. The default Kubernetes scheduler schedules pod by pod, which for distributed workloads produces half-started jobs that hold GPUs without making progress.

How we meet it: GPU allocation is governed by Volcano (a CNCF incubating project), today in pre-production, with the mechanisms that matter for reliability and efficiency:

Gang scheduling (PodGroup with minAvailable): a distributed job starts complete or does not start, avoiding GPUs blocked while waiting.
Per-tenant queues with quota and priority, which make real the fair sharing between tenants that multitenancy only declares.
Fair-share (DRF) to distribute proportionally and pre-emption so that an urgent workload can overtake a deferrable batch.
Topology awareness (NVLink/PCIe) to reduce fragmentation.

This raises the effective utilisation of the GPU fleet, which is what makes sovereign inference economically viable. And it is managed with metrics, not impressions: effective versus allocated utilisation, percentage of time idle, fragmentation, throughput per GPU (tokens per second) and GPU-hours consumed per tenant. These are the figures that let you decide whether more hardware is needed or whether there is spare capacity badly distributed, and the ones that turn a discussion about cost into a decision with data. On the state of the art: Kubernetes 1.34 took DRA (Dynamic Resource Allocation) to general availability, accelerator allocation by attributes instead of by count, and Kueue covers job quota management; both combine with a gang scheduler such as Volcano. Efficiency is not a FinOps extra: here it is the condition that makes meeting the sovereignty requirement sustainable.

It is also worth reading these components in the light of the CNCF AI Conformance: its requirements (the KARs) verify that a cluster integrates GPUs correctly, manages the volumes AI workloads need and resolves job-level networking. They are not ornaments: they are precisely the capabilities that the inference serving, the model and data storage, and the distributed scheduling we have just described rest on. Aligning with that standard is, in practice, a way to guarantee that the platform runs AI with the same properties the industry has agreed as the minimum acceptable, and to be able to prove it, which in a regulated environment is worth as much as complying.

Requirement: activity logging and human oversight (AI Act, EHDS, IHE ATNA/BALP)

The AI Act, for high-risk systems, and clinical AI almost always is one, requires automatic logging of activity (art. 12) and meaningful human oversight (art. 14); the EHDS imposes traceability over the use of data; and IHE ATNA/BALP defines how that audit log must look, expressing it as FHIR AuditEvent resources. The requirement: being able to reconstruct every access and every decision, in a standard, queryable format.

How we meet it, on two planes. On the data access plane, audit events (who accessed which clinical resource, when, for what purpose) are emitted following the FHIR AuditEvent pattern of BALP, so that auditing is interoperable and not a proprietary log. On the AI decision plane, each inference is recorded as an auditable trace with Langfuse, prompt, response, model and version, cost, latency and tenant, stored in our own components locked behind network policies that prevent it from leaving. The trace satisfies the AI Act logging and the EHDS traceability, and at the same time serves as the basis for continuous evaluation and for human oversight, which is exercised at the gateway (review, limits, possibility of intervention). Instrumentation converges on the OpenTelemetry GenAI semantic conventions, correlatable with the rest of the observability.

An audit log is only worth anything as evidence if it is intact and ordered in time. That is why events are kept in append-only storage (with no possibility of later editing) with their retention defined, and node clocks are kept synchronised, the IHE Consistent Time profile put into practice with NTP, so that the sequence of “who did what and when” is reconstructible and defensible. Without integrity and reliable time, a log is a note, not evidence.

Art. 32 of the GDPR requires not only confidentiality and integrity, but availability and resilience, and the ability to restore access to the data after an incident; the ENS details service continuity measures. A clinical system that goes down and does not recover is a breach of the rules, not just an operational incident.

How we meet it: the platform runs across two sites (site01 and site02), which makes it possible to separate workloads and to have capacity available if a site goes down. The control plane of each cluster is highly available, with several control nodes and replicated etcd. The per-tenant databases are managed by an operator that automates replicas and backups to object storage, and the declarative state in Git allows rebuilding a complete environment from its repository. Recovery does not depend on a particular server or on the memory of whoever set it up, but on re-applying what is versioned. The combination of data copies and configuration reproducibility is, in the terms of art. 32, precisely the ability to restore the service within a reasonable time.

Requirement: reproducible and demonstrable operation (ENS, MDR/ISO 13485, AI Act)

The rules do not only ask that the system do the right thing: they ask that you be able to prove it and reproduce it. The ENS requires configuration management and change traceability; the MDR (via ISO 13485) requires change control and reproducibility; the AI Act, risk management and technical documentation.

How we meet it: the whole platform state, deployments, network policies, quotas, RBAC, identity configuration, is described in Git and applied by Flux over GitOps. Every change is a revision with an author, a date and a reviewer; every environment is reproducible from its repository. Model versions are recorded in the traces, and packaging the AI as a versioned MAP reinforces reproducibility: you know exactly which artefact was executed. A configuration audit thus becomes a query against the history, with the ability to reconstruct what was deployed on a given date.

The piece that ties it all together: ISO/IEC 42001

Everything above consists of technical controls that satisfy concrete requirements. But the rules also require a management system to order, evaluate and continuously improve them, with defined roles and responsibilities. That piece is ISO/IEC 42001, the certifiable standard for AI management systems: the one that turns “we have the controls” into “we have governed, demonstrable AI”. It is, precisely, the content of the AI Management Systems course we will publish shortly, designed to accompany the leap from technical implementation to certifiable governance.

A view of the sector as of June 2026

It is worth pinning down the snapshot of the moment, because several currents have converged right now and they condition any platform decision.

Everything in AI converges on Kubernetes. The CNCF itself described it in 2026 as “the great migration”: AI platforms, previously scattered across proprietary solutions, are consolidating on Kubernetes as a common substrate. The practical consequence is that capabilities once reserved for HPC systems, gang scheduling, topology awareness, integration with the network fabric, are now first-class citizens in Kubernetes.

Accelerator allocation is being standardised. With DRA at general availability (Kubernetes 1.34) the old model of “counting GPUs” gives way to allocation by device attributes, with dynamic partitioning. It is a fundamental change in how expensive hardware is shared out, and our clusters (v1.35) are already on the branch where this is real.

Conformance marks for AI are appearing. The CNCF Kubernetes AI Conformance (launched in November 2025, with the KARs) defines what makes a cluster “AI-ready”, and in 2026 it incorporates the Sovereign AI track: hardened sandboxing and data privacy as formal requirements. For healthcare this is enormous, because it turns data sovereignty, which we had been defending out of legal obligation, into a certifiable property of the platform.

Medical imaging AI has its packaging standard. MONAI Deploy and the MAP are maturing as the standard way to build, validate and run medical imaging applications, with Kubernetes as the target and DICOM/FHIR as the contract. Anyone deploying serious imaging AI in 2026 looks towards the MAP, not towards bespoke integrations.

Regulation enters its application window. The EHDS has been in force since March 2025 and its milestones are approaching: data access bodies towards 2027, cross-border primary use towards 2029, all profiled on FHIR R4. The AI Act saw its high-risk deadlines postponed by the Digital Omnibus (Annex III to December 2027, Annex I to August 2028), but its obligations, risk management, logging, human oversight, do not move. And the FHIR R4 to R6 transition is being planned skipping R5. The picture is of a sector where the technology (Kubernetes, eBPF, DRA, MAP) and the rules (EHDS, AI Act, IHE) are finally advancing at the same time.

GPU economics pushes in-house. The resource remains scarce and expensive, and that has two mutually reinforcing effects. On one hand, the pressure for efficiency, utilisation, fragmentation, density, is at its peak, which is why scheduling (Volcano) and attribute-based allocation (DRA) are at the centre of the debate. On the other, the maturing of open-weight models, which perform better and better on modest hardware, has made it viable to serve quality inference locally without depending on an external API. The combination, competent open models plus efficiency tools on Kubernetes, is exactly what turns data sovereignty, in healthcare, from a costly aspiration into a reasonable option. The 2026 trend is not only “AI in the cloud”; it is also a pragmatic return of compute to in-house infrastructure where the data and the cost demand it.

Where we stand in that picture: with the platform on Kubernetes, eBPF as the network and security substrate, GPU scheduling maturing in pre-production, sovereign inference by design, traceability in auditable format and the management framework (ISO 42001) as the next step. We are not chasing fashion; we are exactly on the line the sector has decided to advance along.

Summary: requirement by requirement

The thesis is simple: each rule or standard in the sector imposes a requirement, and each requirement is satisfied by a concrete piece of the infrastructure. The interoperability of the EHDS and the IHE profiles, with a FHIR R4 plus DICOMweb layer and, for imaging AI, MAP packaging. The minimal, authorised access of the GDPR and the ENS and the authentication of ATNA, with Keycloak, SMART on FHIR and mTLS. The confidentiality and leak prevention of art. 32, with multitenancy, Cilium in default-deny and Tetragon. Data sovereignty (GDPR/EHDS and CNCF Sovereign AI), with local inference made viable by Volcano’s efficiency. The logging and oversight of the AI Act and the auditing of ATNA/BALP, with traceability in FHIR AuditEvent format. The reproducible operation of the ENS and the MDR, with GitOps and versioned MAPs. And the governance that ties it all together, with ISO 42001.

It is not a PowerPoint: it is infrastructure that runs, that complies and that, being infrastructure, is measured by efficiency and cost. That, the cost control of a healthcare AI platform, is what the next article is about.

Sources

Volcano and Kueue: gang scheduling, queues and GPU quotas for distributed workloads on Kubernetes

Tue, 16 Jun 2026 13:00:00 +0200

TL;DR

Volcano (volcano-sh, CNCF incubating) is a complete batch scheduler that replaces or complements the kube-scheduler: it places pods with gang semantics (all-or-nothing via PodGroup/minMember), manages queues with priority, DRF fair-share and preemption between queues, and understands network and NUMA topology.

Kueue (kubernetes-sigs/kueue) is a queue and quota manager at the Job level: it does NOT place pods (it delegates to the kube-scheduler or to Volcano), but it decides when a workload can be admitted according to available quota (ClusterQueue/LocalQueue/Cohort), with fair sharing, borrowing between teams and preemption by priority. It natively integrates Job, JobSet, RayJob, all the Kubeflow operators and more.

The winning combination in production for multi-tenant GPU workloads is: Kueue for quota and queues + Volcano (or the sig-scheduler coscheduling plugin) for the job’s gang.

The analogy

Imagine a club with limited capacity and a dance floor inside.

Kueue is the doorman and the booking manager: it checks whether the team’s quota (its reserved capacity) allows the group in, applies the fair waiting list, borrows capacity from other teams if any are idle, and takes the space back when the owner needs it. But the doorman does not decide where each person sits inside the venue.

Volcano is the head waiter: once the group has permission to come in, he decides which tables they sit at, makes sure the whole group sits down at once or nobody comes in (gang), picks the tables according to topology (who needs to talk to whom) and throws out lower-priority groups to make room if necessary.

Without a doorman (Kueue), the head waiter does not know how many groups he can take at once or whether a team is exceeding its capacity. Without a head waiter (Volcano), the doorman lets the group in but its members scatter across the available tables on their own, and the party of 8 that needs to sit together never manages it.

The problem neither solves by default: the kube-scheduler

Kubernetes’ kube-scheduler is a pod scheduler, not a job scheduler. It assigns pods one by one to the most suitable node according to available resources and affinity constraints. For a distributed training workload that needs, say, 8 pods with 4 GPUs each (32 GPUs in total across 8 nodes of 4×H100), the standard scheduler does the following:

It looks for a node with 4 available GPUs. It finds one. It schedules pod 1.
It looks for another node with 4 GPUs. It finds one. It schedules pod 2.
It carries on until it reaches pod 6 and it turns out there are no longer any nodes with 4 free GPUs: the cluster has exactly 32 GPUs and other workloads are using some of them.
Pods 1–5 are Running. Pods 6–8 are Pending.
Pods 1–5 can do nothing without the others: a distributed PyTorch job needs all the workers to start before the torchrun process can begin. It waits with the resources occupied. Deadlock.

This is not a bug, it is the design: the kube-scheduler has no concept of “schedule this group of pods only if you can schedule all of them”. As a result:

The resources of pods 1–5 are locked up without producing work.
Other jobs that could run with the partial resources also wait.
If several jobs are in this situation, the cluster can end up with fragmented resources, no job running and everyone in circular deadlock.

On top of that, the kube-scheduler has no notion of:

Queues per team or project with relative priority.
Quotas of resources per team with the ability to borrow from idle teams.
Fair-share: if team A has spent weeks using 80 % of the cluster, it should wait longer than team B, which has spent weeks idle.
Inter-queue preemption: evicting a lower-priority job from another team to make room for this team’s urgent job.

Solving any of these problems requires adding a layer on top of the scheduler. Volcano and Kueue are the two dominant OSS solutions in 2026, with complementary architectural approaches.

Volcano: the batch scheduler

What it is and what it replaces

Volcano (volcano-sh) is a Kubernetes-native batch scheduler accepted by the CNCF as its first and only official container batch scheduling project (volcano.sh/en/docs). At version v1.15.x as of June 2026, with CNCF incubating status.

Volcano is not an addon to the kube-scheduler: it is an alternative scheduler (or a complementary one, depending on the configuration) that places pods. It is installed as a deployment, and jobs that want to benefit from its capabilities must use the volcano scheduler class in their pod spec (schedulerName: volcano) or use the VolcanoJob CRD.

Volcano’s central value proposition is that it treats groups of pods as atomic scheduling units, not individual pods. That is what makes it possible to solve the deadlock described above.

Gang scheduling via PodGroup

Volcano’s central mechanism is the PodGroup: a CRD that groups a job’s pods and defines how many must be schedulable before Volcano starts any of them.

# PodGroup for a distributed PyTorch training job: 8 workers, minimum 8
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: pytorch-train-pg
 namespace: ml-training
spec:
 minMember: 8 # all-or-nothing: if there is no room for 8, none starts
 minResources:
 nvidia.com/gpu: "32" # 8 pods × 4 GPUs = 32 GPUs minimum in the cluster
 queue: team-datos  # which Volcano queue this job is assigned to
 priorityClassName: high-priority

The minMember parameter implements all-or-nothing semantics: Volcano only assigns nodes to the group’s pods when it can assign at least minMember pods simultaneously. If the cluster does not have capacity for 8 GPU pods at this moment, no pod in the group moves out of Pending. Nothing is blocked, nothing is fragmented.

minMember can be lower than the job’s total pod count: this allows elastic gang scheduling, where the job can start with fewer workers and scale up, useful for jobs that tolerate a reduced worker count.

For a job’s pods to be associated with the PodGroup, they carry the annotation:

# Pod spec of the PyTorch worker
metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: pytorch-train-pg
spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: pytorch/pytorch:2.5-cuda12.4
 resources:
 limits:
 nvidia.com/gpu: "4"

Queue: queues with quotas and priority

Volcano introduces the Queue CRD to manage multiple tenants with independent quotas:

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4 # relative weight for fair-share between queues (proportion plugin)
 capability: # absolute ceiling of resources this queue can use
 nvidia.com/gpu: "16"
 guarantee: # guaranteed resource, never lent to other queues
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true # if true, others can reclaim the resources it lends when they need them

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "8"
 guarantee:
 resource:
 nvidia.com/gpu: "4"
 reclaimable: true

The weight field feeds the proportion plugin: queues compete for the cluster’s available resources in proportion to their weight. A cluster with 32 GPUs and two queues of weight 4 and 2 splits the GPUs in a 4:2 ratio (≈21 and 11 GPUs respectively) when both are saturated.

Scheduler plugins: DRF, binpack, topology-aware

Volcano implements its scheduling logic as a pipeline of actions and plugins:

Actions (what the scheduler does in each cycle):

enqueue: moves jobs from the waiting queue to schedulable when quota is available.
allocate: assigns nodes to schedulable pods.
preempt: evicts lower-priority pods to make room for higher-priority ones within the same queue.
reclaim: evicts pods from other queues that are using more than their guarantee to return resources to the owner.
backfill: fills idle resources with best-effort jobs that do not interfere with the rest.

Plugins relevant to GPU workloads:

Plugin	What it does
`gang`	Implements the PodGroup’s all-or-nothing semantics
`proportion`	Fair-share by queue weight (proportional quota)
`capacity`	Quotas with `guarantee`/`capability` and reclaim; a more expressive alternative to proportion
`drf`	Dominant Resource Fairness: multi-dimensional fair-share (CPU, memory, GPU)
`binpack`	Packs pods onto the fullest nodes; reduces GPU fragmentation
`priority`	Orders jobs by priority within the same queue
`nodeorder`	Node scoring according to multiple criteria (affinity, resources, spread)
`task-topology`	Affinity between pods of the same job (inter-GPU communication)
`numa-aware`	NUMA affinity: aligns pods with the node’s NUMA socket to reduce memory latency

Network and NUMA topology (v1.11+)

Volcano v1.11 (February 2025) introduced Network Topology Aware Scheduling as a first-class feature (CNCF blog, March 2025). Distributed training jobs in a datacenter with a hierarchical network structure (spine/leaf, blocks of nodes with NVSwitch) can declare topology constraints:

# VolcanoJob with a network topology constraint
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: llm-pretrain
 namespace: ml-training
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 plugins:
 ssh: []
 env: []
 svc: []
 networkTopology:
 mode: hard # hard: the job MUST satisfy the constraint
 highestTierAllowed: block  # pods cannot span beyond a network "block"
 tasks:
 - replicas: 8
 name: worker
 template:
 spec:
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The highestTierAllowed: block semantics instruct Volcano to place the 8 pods within the same network block (for example, all the nodes under the same access switch), minimising the inter-block traffic that degrades a distributed all-reduce.

NUMA awareness works in a similar way: with the numa-aware plugin, pods request a NUMA policy (single-numa-node, restricted, best-effort) and Volcano selects nodes where the requested CPU, memory and GPU resources are in the same NUMA domain, avoiding the remote memory access overhead (NUMA-crossing) that can degrade training throughput by 15-30 % on multi-socket nodes.

GPU virtualization in v1.11+

Volcano v1.11 also introduces support for dynamic MIG and vCUDA: instead of declaring nvidia.com/gpu: 1 for a whole GPU, workloads can declare nvidia.com/gpu-memory: 20Gi and Volcano (with the corresponding device plugin) dynamically provisions the MIG instance or the vCUDA partition. This is [project marketing with no independent benchmarks published as of June 2026], but the feature’s architecture is documented in the code.

What it replaces or adds to the default scheduler

Capability	kube-scheduler	Volcano
Placing pods on nodes	Yes	Yes (it replaces it for workloads marked with `schedulerName: volcano`)
Gang scheduling (all-or-nothing)	No	Yes (PodGroup + minMember)
Queues with priority	No	Yes (Queue CRD)
Inter-queue fair-share	No	Yes (DRF, proportion, capacity plugins)
Inter-queue preemption	No	Yes (reclaim action)
Topology-aware (NUMA, network)	Partial (node affinity)	Yes (dedicated plugins)
Elastic gang (minMember < total)	No	Yes
Best-effort backfill	No	Yes
Framework integrations	Partial	MPI, PyTorch, Ray, TensorFlow, Spark, Flink, Horovod

Kueue: the queue and quota manager

What it is and what it does NOT do

Kueue (kubernetes-sigs/kueue) is a Kubernetes-native system that manages quotas and how jobs consume them (kueue.sigs.k8s.io). Kueue decides when a job must wait, when it should be admitted (pods can be created) and when it should be evicted (active pods must be deleted).

Kueue’s central design principle is explicit in its documentation: avoid duplicating mature functionality of Kubernetes components. Autoscaling is the cluster-autoscaler’s responsibility. Pod-to-node scheduling is the kube-scheduler’s responsibility. Job lifecycle management is the kube-controller-manager’s responsibility. Kueue replaces none of them: it sits on top as a layer of admission control and quota management at the Job level.

This is the fundamental distinction: Kueue does not place pods on nodes. When Kueue admits a workload, it simply allows the corresponding job controller to create the pods, and those pods are scheduled by the kube-scheduler (or by Volcano, if it is configured as the scheduler).

The four core objects

ResourceFlavor: maps abstract resources to concrete groups of physical nodes.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 node-pool: gpu-training
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"

ClusterQueue: defines the resource quota per flavor for a tenant. A cluster-scoped object.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform  # cohort it belongs to (can lend/borrow)
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16 # GPUs guaranteed for this team
 borrowingLimit: 8 # can take up to 8 additional GPUs from the cohort
 lendingLimit: 8 # can lend up to 8 of its 16 nominal GPUs
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority  # reclaims lent quota by evicting lower-priority jobs
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority

LocalQueue: the namespace-scoped entry point for a team’s workloads. Jobs point to their LocalQueue; Kueue maps them to the corresponding ClusterQueue.

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos

Cohort: groups ClusterQueues that can lend quota to each other. It is not a standalone CRD; it is declared as a field in the ClusterQueue (spec.cohort: name). Kueue aggregates the available quota of all the ClusterQueues in the cohort and lets any of them borrow what the others are not using, respecting the borrowingLimit and lendingLimit.

Kueue implements Fair Sharing as the ordering policy for the queue of pending workloads (kueue.sigs.k8s.io/docs/concepts/fair_sharing): when several workloads compete for quota in the cohort, those belonging to ClusterQueues with higher accumulated historical usage have lower admission priority. This implements equitable sharing without permanently blocking any team.

Preemption in Kueue operates along two dimensions:

reclaimWithinCohort: the ClusterQueue that lends quota reclaims it by evicting workloads that are using it on loan, according to a priority policy.
withinClusterQueue: within the same ClusterQueue, lower-priority workloads are evicted to make way for higher-priority workloads from the same team.

Gang semantics in Kueue: all-or-nothing with ready Pods

Kueue provides gang admission at the Job level: it admits the complete workload only when all the necessary quota is available. If a RayJob needs 8 GPUs (1 head + 7 workers), Kueue does not admit the workload until 8 GPUs are available in the ClusterQueue (or borrowed from the cohort). waitForPodsReady with a timeout adds a second guarantee: if the created pods do not become Ready within the configured time, Kueue re-queues the workload and releases the quota (kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready).

This is gang semantics at the admission level, not at the pod placement level. It guarantees that the quota is available before creating the pods, but it does not guarantee that the kube-scheduler can place them all on concrete nodes at the same time. For that second guarantee you need Volcano or the coscheduling plugin.

Topology-Aware Scheduling (TAS)

Kueue v0.10+ introduces Topology-Aware Scheduling (kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling): it allows node topologies (blocks, subblocks, hosts) to be defined and workloads to request co-location levels. Kueue only admits the workload when it can satisfy the topological constraint, and it adds node selectors and taints at admission time so that the scheduler places the pods in the right topology.

TAS is configured with the Topology CRD:

apiVersion: kueue.x-k8s.io/v1beta1
kind: Topology
metadata:
 name: datacenter-topology
spec:
 levels:
 - nodeLabel: "topology.kubernetes.io/block"
 - nodeLabel: "topology.kubernetes.io/rack"
 - nodeLabel: "kubernetes.io/hostname"

And the ResourceFlavor references the topology:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 topologyName: datacenter-topology

Framework integrations

Kueue has built-in integration (no additional code) for the following workload types, activated with an annotation on the job:

metadata:
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # points to the team's LocalQueue

The supported workload types include: batch/Job, JobSet, RayJob, RayCluster, PytorchJob, TFJob, MPIJob, JAXJob, PaddleJob, XGBoostJob, TrainJob, AppWrapper, LeaderWorkerSet, Deployment, StatefulSet, and plain Pod/PodGroup.

For LLM workloads, the directly relevant cases:

RayJob (distributed training with Ray Train): Kueue admits the RayJob when there is quota for the whole Ray cluster (head + workers). Documented at docs.ray.io.
PyTorchJob (Kubeflow Training Operator): gang admission of the complete job.
JobSet: for coordinated multi-replica jobs (LWS, multi-step pipelines).
Deployment/StatefulSet: for continuous inference, allowing inference GPU quota to be managed the same way as training quota.

The key distinction: Volcano places, Kueue admits

This table sums up the fundamental architectural difference:

Dimension	Volcano	Kueue
Main role	Scheduler (places pods on nodes)	Admission controller + quota manager (decides when to create pods)
Gang scheduling	Yes, at the placement level (PodGroup/minMember)	Yes, at the admission level (all-or-nothing on quota)
Multi-tenant quotas	Yes (Queue with capability/guarantee)	Yes (ClusterQueue with nominalQuota/borrowingLimit)
Cohorts / borrowing	Partial (reclaimable between queues)	Yes (Cohort with explicit lendingLimit/borrowingLimit)
Fair sharing	Yes (DRF plugin)	Yes (Fair Sharing based on historical usage)
Preemption	Yes (preempt + reclaim actions)	Yes (reclaimWithinCohort, withinClusterQueue)
Topology/NUMA	Yes (dedicated plugins, network topology, NUMA-aware)	Yes (TAS, topology levels in ResourceFlavor)
Framework integrations	Volcano Job (MPI, PyTorch, Ray, TF, Spark, Flink)	Native: Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet
Who places the pods	Volcano	kube-scheduler (or Volcano if configured)
Installation footprint	Medium-high (its own scheduler, CRDs, webhook, metrics)	Light (controller, CRDs, webhook; does not replace the scheduler)
Maturity / status	CNCF incubating; v1.15 (June 2026); production at Huawei, Baidu, DiDi	kubernetes-sigs; API v1beta2; production at Google GKE, Red Hat OpenShift 4.20, Runway ML
Adoption curve	Steeper (requires changing schedulerName or using the VolcanoJob CRD)	Gentler (adds labels to existing jobs; does not change the scheduler)

How they coexist: the production pattern

In production, Kueue and Volcano are complementary, not mutually exclusive. The most common pattern in 2026 for multi-tenant GPU clusters is:

Kueue manages the global quota: which team can use how many GPUs, how much it can borrow, when a job goes into a queue versus being admitted.
Volcano does the gang scheduling at the pod level: once Kueue admits the job (the quota is available), Volcano places the pods, making sure they are all placed simultaneously on compatible nodes.

The integration is configured by specifying schedulerName: volcano in the pod specs of the workloads managed by Kueue. Kueue sees the Job/RayJob/PyTorchJob and manages its quota; when it admits it, the pods are created and Volcano places them with gang semantics. Volcano’s PodGroups are created automatically by the Volcano Job controller or by the Kubeflow Training Operator itself when it detects that the scheduler is Volcano.

# PyTorchJob managed by Kueue (quota) + Volcano (gang placement)
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: llm-finetune-70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the quota
 annotations:
 scheduling.volcano.sh/queue-name: team-datos  # Volcano uses its own Queue
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano  # Volcano does the placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The third way: the sig-scheduler coscheduling plugin

If you do not want to deploy a complete alternative scheduler but you need gang scheduling, there is the coscheduling plugin from kubernetes-sigs/scheduler-plugins. This plugin extends the kube-scheduler with a PodGroup mechanism similar to Volcano’s, implemented as a scheduling framework plugin (permit plugin). The advantage is that it does not replace the scheduler; the disadvantage is that it has less functionality than Volcano (no DRF, no Queue/fair-share, no network topology). It is the right option for simple clusters that only need gang and do not want Volcano’s operational complexity. Kueue can also work alongside this plugin.

Complete YAML examples

Volcano: Queue + PodGroup + VolcanoJob

# 1. Queue for the data team (16 nominal GPUs, ceiling at 24)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4
 capability:
 nvidia.com/gpu: "24"
 cpu: "192"
 memory: "768Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "16"
 reclaimable: true
---
# 2. Queue for the AI team (8 nominal GPUs, ceiling at 16)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "16"
 cpu: "64"
 memory: "256Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true
---
# 3. PodGroup: 70B fine-tuning job, 8 workers × 4 GPUs = 32 GPUs
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: finetune-70b-pg
 namespace: ns-datos
spec:
 minMember: 8
 minResources:
 nvidia.com/gpu: "32"
 queue: team-datos
 priorityClassName: training-high
---
# 4. VolcanoJob (wrapper that Volcano understands natively)
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: finetune-70b
 namespace: ns-datos
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 priorityClassName: training-high
 plugins:
 env: []
 svc: []
 policies:
 - event: PodEvicted
 action: RestartJob
 tasks:
 - replicas: 8
 name: worker
 policies:
 - event: TaskCompleted
 action: CompleteJob
 template:
 metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: finetune-70b-pg
 spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 command: ["torchrun", "--nproc_per_node=4", "--nnodes=8",
 "--node_rank=$(RANK)", "--master_addr=$(MASTER_ADDR)",
 "--master_port=23456", "train.py"]
 env:
 - name: NCCL_DEBUG
 value: "INFO"
 resources:
 requests:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 restartPolicy: Never

Kueue: ResourceFlavor + ClusterQueue + LocalQueue + annotated Job

# 1. ResourceFlavor: nodes with H100 SXM
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"
---
# 2. ClusterQueue for the data team: 16 nominal GPUs, can borrow 8
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16
 borrowingLimit: 8
 lendingLimit: 8
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority
---
# 3. ClusterQueue for the AI team: 8 nominal GPUs
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-ia
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 8
 borrowingLimit: 8
 lendingLimit: 4
 - name: "cpu"
 nominalQuota: "64"
 - name: "memory"
 nominalQuota: "256Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# 4. LocalQueue in the data team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos
---
# 5. LocalQueue in the AI team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-ia
 namespace: ns-ia
spec:
 clusterQueue: cq-team-ia
---
# 6. Batch inference RayJob managed by Kueue
apiVersion: ray.io/v1
kind: RayJob
metadata:
 name: batch-eval-llama70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the admission
spec:
 entrypoint: "python batch_eval.py --model /models/llama-70b"
 rayClusterSpec:
 headGroupSpec:
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-head
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 workerGroupSpecs:
 - replicas: 3
 minReplicas: 3 # gang: Kueue does not admit if there is no quota for 3 workers
 maxReplicas: 3
 groupName: gpu-worker
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-worker
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

Kueue + Volcano together: PyTorchJob with Kueue quota and Volcano gang placement

# PyTorchJob: Kueue controls the quota, Volcano does the gang placement
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: distributed-finetune
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos # Kueue: quota and admission
 annotations:
 # Volcano automatically creates the PodGroup when schedulerName=volcano
 scheduling.volcano.sh/queue-name: team-datos
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano # Volcano: gang placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

For LLM workloads: training, fine-tuning and batch inference

Distributed multi-GPU training and fine-tuning

Gang scheduling is essential for any distributed training job that uses NCCL all-reduce (PyTorch DDP, FSDP, DeepSpeed ZeRO). If a single worker in the group does not start, the torchrun coordinator waits indefinitely; with the standard kube-scheduler this scenario happens every time the cluster is under contention.

In a generic cluster of 4 nodes of 4×H100 SXM (16 GPUs in total), a fine-tuning job for a 70B model typically requires 8 GPUs in tensor-parallel 8 (TP=8) or 16 GPUs in TP=4 plus data-parallel 4. With Volcano, the PodGroup with minMember: 8 guarantees that either the 8 pods are placed at once or none blocks resources. With Kueue on top, the quota guarantees that the team does not exceed its 16 nominal GPUs and that other teams with available quota are not blocked by a waiting job.

The cross-link with inference capacity planning is direct: the model’s VRAM budget (weights plus KV-cache) determines the minimum TP and therefore the PodGroup’s minMember.

Batch inference and evaluations (evals)

Batch inference jobs, generating responses for an evaluation dataset, processing embeddings in bulk, offline re-ranking, are naturally parallel workloads that do not necessarily require strict gang scheduling (each request is independent), but they do benefit from quota and fair-share.

For these workloads, Kueue alone is enough: a batch/Job with multiple independent pods is managed with the ClusterQueue’s quota with no need for Volcano. If several teams are submitting evaluation jobs simultaneously, Kueue orders the admission by fair-share and priority, and with cohort borrowing the jobs of teams with free quota do not have to wait for the quota of busy teams.

The chargeback of these workloads connects directly with what is described in GPU chargeback and showback: the ClusterQueue’s nominalQuota is the expression of the GPU budget in Kubernetes, and OpenCost can attribute the cost per namespace or label for the monthly report.

Multi-tenant GPU quota and chargeback

The alignment between Kueue and the chargeback system is direct:

FinOps concept	Kueue mechanism
Guaranteed GPU budget	`nominalQuota` per ClusterQueue
Maximum spending ceiling	`nominalQuota + borrowingLimit`
Lending idle capacity	Cohort + `lendingLimit`
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`
Fair-share between teams	Fair Sharing policy in the ClusterQueue
Chargeback of the loan	borrowed GPU hours × cost per GPU-hour (OpenCost)

For the utilisation dimension as a FinOps lever, see GPU utilisation as a FinOps lever: Kueue and Volcano together make it possible to maximise utilisation without sacrificing the quota guarantees, which is exactly the FinOps objective.

Managing MIG partitions within this system (declaring nvidia.com/mig-4g.40gb as a resource in the ClusterQueue) integrates naturally: the ResourceFlavor can map to nodes with a specific MIG profile, as explained in sharing a GPU: time-slicing, MPS and MIG.

Diagram: the flow of a training job with Kueue + Volcano

Full comparison table

Criterion	Volcano	Kueue	Coscheduling plugin
Role	Scheduler (placement)	Admission + quota (no placement)	kube-scheduler plugin (placement)
Gang scheduling	Yes, pod level (PodGroup/minMember)	Yes, admission level (quota gang)	Yes, pod level (PodGroup)
Multi-tenant quotas	Yes (Queue capability/guarantee)	Yes (ClusterQueue nominalQuota)	No
Cohorts / borrowing	Limited (reclaimable)	Yes (Cohort with borrowingLimit/lendingLimit)	No
Fair-share	Yes (DRF, proportion)	Yes (Fair Sharing by historical usage)	No
Inter-queue preemption	Yes (reclaim action)	Yes (reclaimWithinCohort)	No
Network / NUMA topology	Yes (v1.11+, dedicated plugins)	Yes (TAS, Topology CRD)	No
Native integrations	MPI, PyTorch, Ray, TF, Spark, Flink, Horovod	Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet	Any job with a PodGroup
Who places the pods	Volcano	kube-scheduler (or Volcano)	kube-scheduler
Elastic gang	Yes (minMember < total replicas)	Partial (partial admission in batch/Job)	Limited
Footprint	Medium-high (its own scheduler)	Light (an additional controller)	Minimal (a scheduler plugin)
Compatibility with Kueue	Yes (as the scheduler under Kueue)	—	Yes (a complementary option)
CNCF status / maturity	CNCF incubating, v1.15	kubernetes-sigs, v1beta2, adopted in GKE/OpenShift	kubernetes-sigs/scheduler-plugins, experimental
When to choose it	HPC-like distributed training, NUMA, network topology, MPI	Multi-tenancy with flexible quota, heterogeneous workloads, inference + batch together	Simple clusters that only need gang without their own scheduler

Operational pitfalls and honest scepticism

1. Deadlock from a badly configured gang

The most frequent scenario: minMember set equal to the total replica count in a cluster where several jobs compete for the same resources. If two jobs of 8 pods each try to use a cluster with 8 GPU nodes and job A has 4 pods placed (not 8, so Volcano holds them pending), and job B also has 4 pods held, nobody moves forward. Volcano is doing its job properly: it places none of them until there is room for all 8. But if the queues’ nominalQuota values are badly sized against the cluster’s real capacity, this produces indefinite waits.

Solution: size the queue quotas so that the sum of guarantee does not exceed the real capacity, and so that the minMember values of the active jobs fit inside the available quota. Node autoscaling with ProvisioningRequest (Kueue + cluster-autoscaler) helps, but it introduces provisioning latency that has to be taken into account in the job’s SLA.

2. Quota versus real capacity: the silent drift

Kueue’s nominalQuota and Volcano’s guarantee are administrative declarations. They do not guarantee that the nodes with those GPUs are available, healthy or that the device plugin has registered them correctly. A node in NotReady with 4 GPUs reduces the real capacity without Kueue knowing: the ClusterQueue will carry on admitting workloads that then cannot be placed.

Recommended monitoring: cross Kueue’s metrics (kueue_admitted_workloads_total, kueue_pending_workloads) with the cluster’s real capacity metrics (GPUs registered in the device plugin) to detect the drift. Kueue exposes native Prometheus metrics; so does Volcano.

3. GPU resource naming: MIG, time-slicing and ResourceFlavor

If the cluster uses MIG, the resources in the pod specs change from nvidia.com/gpu to nvidia.com/mig-Xg.Ygb (for example, nvidia.com/mig-3g.40gb). Kueue’s ResourceFlavors and Volcano’s Queues must declare the right resource, or the quota will not match the pods. With time-slicing, the resource is still nvidia.com/gpu but the device plugin advertises more instances than there are physical GPUs; the quota is expressed in virtual replicas, which can lead to over-admission if the VRAM budget is not taken into account (see sharing a GPU: time-slicing, MPS and MIG).

4. Preemption in production: the evicted workload loses progress

When Kueue or Volcano evicts a training job that has been running for hours, that job loses its progress if it has no checkpointing configured. The preemption is correct from the quota point of view, but it destroys work if the job is not prepared for it. Before enabling aggressive preemption, verify that all training jobs have periodic checkpointing with automatic restoration. PyTorch plus Torchrun have native support; so does DeepSpeed. Stateless batch inference jobs do not have this problem.

5. Volcano as the sole scheduler versus coexistence with the kube-scheduler

Volcano can be configured as the cluster’s default scheduler (all pods go through it) or as an alternative scheduler (only pods with schedulerName: volcano). The first option simplifies the configuration but breaks system pods that assume kube-scheduler behaviour. The second, the recommended one, requires ML jobs to explicitly set schedulerName: volcano, which can be a non-trivial operator or chart change for existing workloads. Kueue solves this more transparently: it only requires a label on the job, with no change of scheduler.

6. Real operational complexity in 2026

Running Kueue plus Volcano plus the Training Operator plus the GPU Operator in production means four components with their own CRDs, webhooks, versions and release cycles. A Kubernetes upgrade may require updating all four in sequence. The operational debt is real. For a small team without the capacity to maintain this stack, a managed Kubernetes provider (GKE with native Kueue, OpenShift with the Red Hat build of Kueue) may be more pragmatic than assembling the full stack from scratch.

The sig-scheduler coscheduling plugin is a deliberately simpler option when all you need is gang: fewer features, less complexity, fewer things to maintain.

Sources

Volcano — Introducción y features (CNCF incubating, v1.15) — https://volcano.sh/en/docs/
Volcano — Volcano v1.11 Released: A New Era of Cloud-Native Scheduling (CNCF blog, marzo 2025) — https://www.cncf.io/blog/2025/03/05/volcano-v1-11-released-a-new-era-of-cloud-native-scheduling-for-ai-and-big-data/
Volcano — Release v1.11.0 (GitHub) — https://github.com/volcano-sh/volcano/releases/tag/v1.11.0
Volcano — Network Topology Aware Scheduling design doc — https://github.com/volcano-sh/volcano/blob/master/docs/design/Network%20Topology%20Aware%20Scheduling.md
Volcano — NUMA-aware scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/numa-aware.md
Volcano — Unified Scheduling (docs oficiales) — https://volcano.sh/en/docs/unified_scheduling/
Volcano — Capacity scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/capacity-scheduling.md
NVIDIA Technical Blog — Practical Tips for Preventing GPU Fragmentation for Volcano Scheduler — https://developer.nvidia.com/blog/practical-tips-for-preventing-gpu-fragmentation-for-volcano-scheduler/
Kueue — Overview (kueue.sigs.k8s.io, actualizado febrero 2026) — https://kueue.sigs.k8s.io/docs/overview/
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Topology Aware Scheduling — https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/
Kueue — Preemption — https://kueue.sigs.k8s.io/docs/concepts/preemption/
Kueue — Setup All-or-nothing with ready Pods — https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready/
Kueue — GitHub (kubernetes-sigs/kueue) — https://github.com/kubernetes-sigs/kueue
Ray — Gang Scheduling with RayJob and Kueue — https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-kueue-gang-scheduling.html
Kubeflow — Volcano scheduler integration — https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/volcano/
kubernetes-sigs/scheduler-plugins — Coscheduling plugin README — https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/coscheduling/README.md
kubernetes/enhancements — KEP-583 Coscheduling (sig-scheduling) — https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling
InfraCloud — Batch Scheduling on Kubernetes: Comparing YuniKorn, Volcano and Kueue — https://www.infracloud.io/blogs/batch-scheduling-on-kubernetes/
Red Hat — Red Hat build of Kueue (OpenShift 4.20) — https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/ai_workloads/red-hat-build-of-kueue
AceCloud — Multi GPU Orchestration in Kubernetes 2026: Kueue, Volcano, DRA — https://acecloud.ai/blog/multi-gpu-orchestration-kubernetes/
CloudOptimo — Kubernetes AI Infrastructure in 2026: GPU Scheduling and Production Realities — https://www.cloudoptimo.com/blog/kubernetes-ai-infrastructure-in-2026-gpu-scheduling-and-production-realities/

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

Sun, 14 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the energy track (C3). C1 made the conceptual inventory of the tools; here we get practical: how the measurement stack is deployed on Kubernetes, where each piece takes its figures from, how accurate it is against a power meter, how much overhead it adds, and how energy is attributed per pod and per MIG instance. It is the equivalent of B2/B3 for energy: to stop talking about what each tool measures and start building the bench that measures it. With commands; no recommendations, only the mechanics.

The measurement stack on Kubernetes

To measure energy per workload on a GPU cluster, the practical stack has four pieces, and the good news is that three of them you already have if you do observability:

The new piece compared with a normal observability stack is Kepler, which adds energy attribution per pod. DCGM already gives GPU power; Prometheus stores it; Grafana draws it. It is reused infrastructure, not a new system, in line with GPU observability via DCGM.

DCGM exporter: the base of GPU power

GPU power, the dominant source in LLM inference, comes from NVIDIA’s dcgm-exporter (part of the GPU Operator), which exposes DCGM metrics to Prometheus. The key metric is DCGM_FI_DEV_POWER_USAGE (instantaneous power in W), alongside utilisation (DCGM_FI_DEV_GPU_UTIL) and memory. With that, energy is the integral of power over time across the workload’s window. DCGM gives power at card level; to split it per pod or per MIG instance you need a layer on top, which is Kepler.

Kepler in depth: eBPF and the sources

Kepler (Kubernetes-based Efficient Power Level Exporter) is a Prometheus exporter that measures energy at container, pod and node level, using eBPF to keep overhead low and scientific methods to improve accuracy (Kepler · GitHub). Its strength is that it combines several power sources:

Source	What it covers
RAPL	CPU and DRAM
NVML	NVIDIA GPU power
ACPI	platform power management
Redfish / IPMI	platform power (BMC)
Regression models	estimation when there is no real metric

For the GPU, the dcgmEndpoint parameter points at the dcgm-exporter metrics URL, which enables power attribution per MIG instance (Kepler · configuration). That is: DCGM gives the card’s power, Kepler splits it between the pods (and MIG partitions) that use it, via eBPF and models. The result is per-pod energy metrics in Prometheus, ready for Grafana, usable for scheduling or scaling, consumption reporting and visualisation (Kepler · GitHub).

Deployment: a DaemonSet on top of Prometheus

The deployment is a DaemonSet (one Kepler per node), pointing at the existing Prometheus and at the dcgm-exporter:

# Helm, pointing at the dcgm-exporter for GPU attribution
helm install kepler kepler/kepler \
 --namespace kepler --create-namespace \
 --set dcgmEndpoint="http://dcgm-exporter:9400/metrics"

Prerequisites: a kernel with eBPF, the dcgm-exporter running (from the GPU Operator) and Prometheus scraping. Once deployed, Kepler exposes metrics such as kepler_container_joules_total per pod, queried with PromQL and drawn in Grafana. Energy per workload stops being a manual estimate and becomes a continuous time series, just like cost with OpenCost.

PromQL queries: energy per pod

With Kepler exporting, these are the queries that build the panels. The exact names vary by version; check your /metrics:

# Cumulative energy per pod (joules)
kepler_container_joules_total

# Mean power per namespace over the last hour (W)
sum by (namespace) (rate(kepler_container_joules_total[1h]))

# GPU power per node (from DCGM, W)
sum by (node) (DCGM_FI_DEV_POWER_USAGE)

# Energy per token: power ÷ throughput (with a gateway/engine metric)
sum(rate(kepler_container_joules_total[5m])) / sum(rate(vllm_generation_tokens_total[5m]))

The last one closes the circle with the cost track: it crosses Kepler’s energy with the engine/gateway tokens to give J/token in real time, per workload. With that in Grafana you have energy per pod, per namespace and per token, continuously, the energy equivalent of the OpenCost cost report.

A calibration procedure

Since Kepler estimates, it is worth calibrating once per node type. A simple procedure:

Power meter at the node input (or a PDU/BMC reading), measuring the node’s real power.
Controlled load: a stable sweep at a known power (the engine at 80 % GPU, say).
Compare: the node’s sum of kepler_container_joules_total plus idle against the meter reading over the same window.
Adjust Kepler’s factor or model if the deviation is noticeable, and document the residual error.
Repeat for each node type (calibration is not transferable between different hardware).

With that residual error documented, Kepler’s continuous figure has a known error bar, which is exactly what a CSRD auditor or a technical committee will ask for. Measuring without calibrating gives a number with no credibility; calibrating once turns the continuous estimate into a defensible figure.

Accuracy: estimation vs power meter

The data honesty of the energy track: Kepler estimates (eBPF + models + counters), it does not measure with a power meter. Accuracy depends on the source: with NVML/DCGM and RAPL available, the figure is fairly faithful (it reads hardware counters); when it falls back on regression models (with no real metric), it is an estimate with more error. Good practice: calibrate against a physical power meter at least once per node type, adjust, and use Kepler as a continuous proxy. The meter is the point-in-time ground truth; Kepler, the continuous series calibrated against it. For a CSRD report that an auditor may challenge, that initial calibration is what gives the continuous figure credibility.

Overhead: the cost of measuring

Measuring energy has its own compute cost, worth budgeting for. In a Kubernetes environment, the overhead of the monitoring system is split between the Prometheus server and the targets: the Prometheus server consumes ~150 % of CPU (1.5 cores) and ~15 GB of RAM, while the targets (node exporter and dcgm-exporter) together use ~35 % of CPU and ~300 MB of RAM (arXiv · Container-level Energy Observability). Kepler uses eBPF precisely to keep its own overhead low. The reading: the bulk of the cost of measuring is in Prometheus (which you probably already have for observability), not in the energy exporters. Adding Kepler to an existing observability stack is marginal; standing up Prometheus just for this would be expensive. That is why measuring energy is nearly free if you already observe, and a project if you do not.

Zeus: for the bench and for optimisation, not for continuous production

Zeus (ml.energy) plays a different role: it is not a continuous production exporter, it is the bench tool for measuring and optimising the energy of one specific workload (a benchmark, a training run), with power capping and frequency selection. It fits into the benchmarking flow (alongside GuideLLM/AIPerf): during a sweep, Zeus measures the energy per configuration and lets you test the effect of power capping on performance. The practical division: Kepler/DCGM in production (continuous per-pod attribution), Zeus on the bench (fine measurement + optimisation of one workload), CodeCarbon if you want carbon context integrated into an ML pipeline.

CodeCarbon and Scaphandre in practice

The other two pieces, with their specific role:

CodeCarbon integrates inside the code of an ML pipeline (a decorator or a context manager wrapping the training or the inference), and returns the energy and the CO₂eq of that block, using nvidia-smi for the GPU and estimating CPU/RAM. It fits when you want the carbon figure per experiment or per ML job, integrated into the data flow, not at infrastructure level. It is the closest to the power meter among the Python tools.
Scaphandre is a metrology agent in Rust that reads RAPL and gives process/VM power with raw hardware accuracy. For CPU it is very accurate; for GPU it has to be combined with NVML. It fits when you need the most faithful CPU metrology, or a lightweight agent independent of Kubernetes.

The division with Kepler: Kepler attributes energy per pod in the cluster (the infrastructure view); CodeCarbon gives carbon per ML job (the pipeline view); Scaphandre gives raw CPU metrology. They do not compete; they cover different layers, and a mature platform uses whichever matches the question.

Choosing the stack by objective

Objective	Tool	Why
Continuous per-pod attribution (production)	Kepler + DCGM	eBPF, per pod/MIG, into Prometheus
Fine measurement + optimisation (bench)	Zeus	measures and optimises (power capping)
Carbon in an ML pipeline	CodeCarbon	integrated CO₂eq, close to the power meter
Raw CPU accuracy (RAPL)	Scaphandre	hardware metrology
Calibration / ground truth	power meter	the standard everything is calibrated against

It is not “one tool”, it is a stack by objective: continuous production (Kepler/DCGM), bench (Zeus), reporting (CodeCarbon), calibration (power meter). Mixing them according to the phase is what gives a complete and credible energy measurement.

From figure to action

Measuring is the start; the value is in acting. With energy per pod in Prometheus:

Reporting (CSRD): the carbon footprint per workload, calibrated, ready for regulatory reporting.
Energy/carbon-aware scheduling: move deferrable load to hours or nodes of lower intensity (connects with the hourly carbon of the Spanish grid).
Power capping: with Zeus, trim peak power with little performance loss, lowering the W in the numerator of J/token.
Optimisation: identify pods with poor energy efficiency (a lot of energy per token) and attack them with quantisation or batching.

The measurement stack is not a panel to look at: it is the input to scheduling, optimisation and reporting decisions. Just as OpenCost closes the cost loop, Kepler+DCGM closes the energy one.

From energy to carbon: the final layer

The energy figure (kWh) becomes carbon by multiplying by the grid intensity, and here the practical stack connects with the hourly source. In Spain, the intensity is published by Red Eléctrica (esios) and aggregated by ElectricityMaps, hour by hour. The pattern:

# Carbon per workload: energy (kWh) × grid intensity (gCO2/kWh)
(sum by (namespace) (increase(kepler_container_joules_total[1h])) / 3.6e6)
 * on() group_left() grid_carbon_intensity_gco2_kwh

where grid_carbon_intensity_gco2_kwh is a metric you feed from esios/ElectricityMaps (an exporter or a periodic push). With that, the Grafana panel shows gCO₂ per namespace and per token, hour by hour, not an annual average. And since Spanish intensity varies a lot (from ~80 in solar hours to ~250 on gas-heavy nights), that hourly resolution is what enables carbon-aware scheduling: moving the deferrable work to the clean hours lowers the measured carbon. The measurement stack does not only report carbon; with the hourly figure, it allows it to be reduced.

Energy efficiency KPIs to watch

KPI	What it indicates	Where from
J/token per model	energy efficiency	Kepler ÷ engine tokens
Mean power per pod	consumption per workload	`rate(kepler_container_joules_total)`
gCO₂/token (hourly)	real footprint	energy × grid intensity
Idle energy	waste	power of pods with no traffic
Effective PUE	DC overhead	total energy ÷ compute energy

These five turn the energy panel into a decision tool: which model is more efficient, which pod wastes, when it pays to schedule. Without them the stack is a pretty panel; with them, it is the input to optimisation and reporting.

Example: energy attribution on a multi-tenant cluster

With Kepler splitting per pod, an energy report per team on the example node (8×H100, 5.6 kW of boards, ~7.84 kW with PUE), over one month:

Team	Energy/month (kWh)	gCO₂ (ES grid ~160)	Mean J/token
A · chat-prod	~3,100	~496 kg	~2.2 J/tok
B · batch	~1,400	~224 kg	~2.8 J/tok
C · experimentation	~900	~144 kg	~6.5 J/tok
idle	~250	~40 kg	—

What it reveals: A is efficient (FP8, high utilisation → 2.2 J/token); C is inefficient (6.5 J/token, underused GPU), just as it dragged the idle cost in the FinOps track. The carbon column takes the figure into the per-team CSRD report. And the idle row (~250 kWh, ~40 kg CO₂) is pure wasted energy. Without Kepler, that split does not exist: you would only see the node’s total consumption, with no idea whose each watt is. With it, energy is attributable and actionable, exactly like cost with OpenCost. In fact, the two panels together (OpenCost cost + Kepler energy) give the complete picture of efficiency per team.

Energy and carbon-aware scheduling

The highest-return action the stack enables: scheduling by energy and carbon. With energy per pod and hourly grid intensity in Prometheus, two policies:

By carbon: move the deferrable load (training, ingestion, overnight batch) to the hours of lowest grid intensity. In Spain, the hours of peak solar (~80 gCO₂/kWh) against gas-heavy nights (~250). The same workload emits ~3× less when scheduled at the right hour.
By efficiency: consolidate workloads to eliminate idle (a powered GPU with no work is wasted energy), and direct each workload to the hardware most efficient for it.

Both use the same data (energy per pod + hourly intensity) that the stack already exposes, and they connect with cluster scheduling (Kueue/Volcano). It is the difference between measuring carbon (and reporting it) and reducing it (by scheduling). The C3 stack enables both.

Deployment checklist

Step	Action	Verification
1	dcgm-exporter (GPU Operator)	`DCGM_FI_DEV_POWER_USAGE` in Prometheus
2	Kepler DaemonSet, kernel with eBPF	`kepler_container_joules_total` per pod
3	`dcgmEndpoint` pointing at the exporter	GPU/MIG attribution works
4	Grid intensity metric (esios)	`grid_carbon_intensity` in Prometheus
5	Calibration vs power meter (once per node type)	error bar documented
6	Grafana panels (J/token, carbon, idle)	KPIs visible

Step 5 (calibration) is the one most often skipped and the one that gives the figure credibility; step 3 (dcgmEndpoint) is the one most often forgotten, and without it the GPU is not attributed.

Energy by phase: prefill vs decode

A level of detail the stack allows and that guides optimisation: power is not constant during a request. Prefill (processing the prompt) is compute-bound and draws a lot of power in a burst; decode (generating tokens) is memory-bound and draws a lower peak but for longer. With fine DCGM sampling that bimodal profile is visible, and it has consequences:

A workload with long prompts (a lot of prefill) has a different power profile from one with long generation (a lot of decode), even if they produce the same tokens.
Power capping affects each phase differently: trimming peak power mostly touches the prefill.

For most cases the mean J/token is enough; to optimise energy efficiency thoroughly, separating prefill and decode consumption (aligning the DCGM window with each phase) reveals where the spend is. It is the energy equivalent of separating TTFT and TPOT in benchmarking, the same principle, measuring the two phases separately because they behave differently.

State of the art: the LLM resolution gap

An honest note on the limits of current tooling. The general-purpose tools (Kepler, CodeCarbon, Scaphandre) measure well at machine, process and pod level, but they lack LLM-specific resolution: out of the box they do not distinguish prefill from decode energy, nor energy per token by phase, nor the energy cost of a reasoning block against a normal response. It is an open gap in the field: per-pod attribution exists, but attribution per token and per inference phase has to be built by crossing power (DCGM/Kepler) with engine metrics (vLLM exposes tokens per phase). The 2026 state of the art is a mature stack for infrastructure and an emerging one for LLM-specific resolution, exactly the crossing that the reproducible harness (S4) has to build by hand. Acknowledging it avoids promising a per-token accuracy that the general-purpose tools do not deliver on their own.

Limits and traps (data-driven)

Uncalibrated estimation. Kepler estimates; without calibrating against a power meter at least once, the figure can carry a noticeable error, especially when it falls back on regression models.
Forgetting the dcgmEndpoint. Without pointing Kepler at the dcgm-exporter, GPU (and MIG) attribution does not work; it is the key parameter for GPU.
Accounting for overhead badly. The bulk of the cost of measuring is in Prometheus; if you stand it up just for energy, budget for it (1.5 cores, 15 GB).
Confusing bench and production. Zeus measures one workload finely; Kepler attributes continuously in production. Using one for the other gives the wrong figure.
Misaligned windows. For energy per token, the power window and the token window must coincide; if they do not, the J/token does not correspond.

With the measurement stack in place, the energy track has its instrumentation; the following articles (C6 carbon, C8 TCO and regulation) build on this data. The foundation is this: a continuous per-pod measurement, calibrated, on top of the observability infrastructure you already have.

Closing

Measuring the energy of an AI cluster in production does not require a new system: it requires adding Kepler, per-pod attribution via eBPF, to the DCGM, Prometheus and Grafana stack you already have for observability. That is the central idea of C3: the cost of measuring is almost entirely in the Prometheus you already pay for, and Kepler turns it from “card power” into “energy per pod, per team and per token”. With two cautions that separate a credible figure from a decorative panel: calibrate once against a power meter (to have an error bar that survives an audit) and point the dcgmEndpoint (without which the GPU is not attributed). Done that way, energy stops being the soft axis of sustainability and becomes an actionable time series: the J/token per model that exposes inefficiency, the hourly carbon that enables clean scheduling, the idle that gets recovered, and the CSRD report that holds up. For the sovereign platform, the Kepler+DCGM stack closes the energy loop just as OpenCost closes the cost one, and together, with GuideLLM’s goodput, they give the three faces of the dashboard measured on the same iron, in euros, joules and grams of CO₂.

Sources

Kepler · GitHub (CNCF, eBPF, por pod/nodo) — https://github.com/sustainable-computing-io/kepler
Kepler · configuración (fuentes, dcgmEndpoint para MIG) — https://github.com/sustainable-computing-io/kepler/blob/main/docs/user/configuration.md
CNCF · explorando Kepler (consumo de aplicaciones cloud) — https://www.cncf.io/blog/2023/10/11/exploring-keplers-potentials-unveiling-cloud-application-power-consumption/
Red Hat · Kepler: monitorización eficiente de energía para Kubernetes — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
arXiv · Container-level Energy Observability in Kubernetes (overhead) — https://arxiv.org/pdf/2504.10702
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Zeus Project (ml.energy) — https://ml.energy/zeus/

OpenCost in depth: how GPU cost is allocated in Kubernetes

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. European references (OVH is among the supported providers). The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Second article in the FinOps track (A2), and the first tool deep dive. The FinOps introduction established that OpenCost is the CNCF standard for cost allocation; here the box gets opened: how it allocates cost internally, where it gets its data from, how on-prem prices are configured in euros, what its API exposes, and where the traps are. Understanding the mechanics matters because it defines what any tool on top (Kubecost included) can and cannot do, and because a badly set base price invalidates the whole split. No recommendations; only the mechanics and the methodology.

What OpenCost is

OpenCost is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project). It is a cost allocation layer for Kubernetes: it reads resource usage from the cluster, joins it with a price, and splits the cost across Kubernetes dimensions down to the container (OpenCost · GitHub). It supports dynamic pricing via the billing APIs of AWS, Azure and GCP, and custom prices for on-prem clusters (OpenCost · configuration).

What it is not: it is not an optimiser, nor a governance system, nor a product unit economics tool. It is the piece that answers “how much does each namespace/pod/team cost?”, and leaves the rest to the layers above. That is why it is the foundation of the FinOps track: without correct allocation, there is no cost per token and no TCO model worth anything.

The cost model: node-level allocation

The key to OpenCost is that the model works at node level. It starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit per-resource prices, OpenCost uses the ratio of a set of base prices (marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem).

This is what makes it work on-prem: you declare the node cost (depreciated capex + opex), and OpenCost splits it without needing a cloud invoice. The normalisation guarantees that no cost is “lost” or “invented”: what you pay for the node is exactly what gets split across its resources.

Where the data comes from: Prometheus

OpenCost instruments nothing of its own: it reads from Prometheus, which is a prerequisite of the installation (OpenCost · GitHub). The sources:

Source	What it provides
`kube-state-metrics`	state of K8s objects (pods, requests/limits)
`node-exporter`	node resources and capacity
`cAdvisor`	real CPU/memory usage per container
`dcgm-exporter` (NVIDIA)	GPU usage, memory and power

With those series, OpenCost automates the join between Kubernetes resource usage and the provider price, reducing the custom PromQL you would otherwise write by hand (Grafana · manage costs). The GPU signal comes from DCGM via the dcgm-exporter (part of the NVIDIA GPU Operator), the same base as GPU observability. Without Prometheus and without DCGM exporting, OpenCost does not see the GPU.

Architecture: the pieces

OpenCost is not a monolith; it is three responsibilities worth telling apart:

Piece	Function
Cost model	resolves the node price and splits/normalises it across resources
Allocation API	serves the allocated cost per dimension (`/allocation` and endpoints)
Exporter	publishes cost metrics on `/metrics` for Prometheus/Grafana

The data flow: Prometheus holds the usage series (from kube-state-metrics, cAdvisor, dcgm-exporter); the cost model joins them with the resolved price; the Allocation API serves them aggregated by whatever dimension you ask for; and the exporter returns them to Prometheus so that Grafana can plot them. It is a cycle: metrics come in from Prometheus and cost goes back to Prometheus enriched. That symmetry is what lets you reuse your observability stack without standing up a new database: OpenCost lives on top of Prometheus, not beside it.

An operational note: historical analysis depends on Prometheus retention. For cost reports spanning months, a long-term backend behind it (Thanos, Mimir, VictoriaMetrics) is advisable, or periodically exporting the allocations to a store of your own.

On-prem prices in euros: the piece that decides everything

On-prem, the most important figure, and the most neglected, is the node price. It is configured per provider; the options are alibaba, aws, azure, gcp, oracle, ovh or default (on-prem) (OpenCost · configuration). For on-prem you use default, supplying base prices in a default.json (or by overriding the Helm values.yaml); anything you do not override uses the chart’s value (OpenCost · on-prem).

To set the node price in euros, the calculation is the TCO model one: depreciated capex + opex. An example of an 8×H100 node:

Component	Calculation	€/hour
Capex (node ~240,000 €, 36 months)	240,000 ÷ 26,280 h	~9.13
Energy (5.6 kW × PUE 1.4 × 0.058 €/kWh, France)	~0.455 €/h	~0.46
Maintenance, networking, operations	estimate	~1.5
Node total		~11.1 €/h

That ~11 €/h is the number you put into OpenCost, and everything is split from it. And here is the documented trap: with the default pricing configuration, OpenCost under-prices CPU and GPU on-prem (issue #3781). If you do not adjust the base prices to your real cost, the split will be systematically low and the cost per token you report will be unreal. Configuring the node price is not optional: it is the figure everything else hangs from.

Configuring the price: an example

The base price is given in the Helm values.yaml (or a default.json), in euros per hour and per unit of resource, derived from the node’s total cost:

# values.yaml — on-prem prices in €/hour (8xH100 node, France)
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 8xH100 node"
 CPU: "0.030" # €/CPU-hour
 RAM: "0.004" # €/GB-hour
 GPU: "1.30" # €/GPU-hour ← the figure that moves cost/token the most
 storage: "0.0002" # €/GB-hour

OpenCost normalises those values so that, multiplied by the node capacity, they add up to the total price you declared (~11 €/h in the example). The number to be most careful with is the GPU one: at 1.30 €/GPU-hour, the 8 cards are ~10.4 €/h, the bulk of the node’s cost. A badly set GPU base shifts the whole cost per token. Always review the resolved prices in /allNodePricing after configuring, because the default under-pricing bug bites exactly here.

Allocation: by usage, not by presence

The behaviour that has changed most and that matters most for GPU: in recent versions, the cost allocated to a GPU is determined by its usage, not its presence. If a GPU costing 100 €/month is only half used, the allocated cost is ~50 € ([search]). This aligns allocation with reality, you pay for what you use, but it has a consequence: the other half does not disappear, it is idle that somebody keeps paying for. OpenCost exposes it, and that is where the optimisation lever comes from.

OpenCost allocates to any Kubernetes dimension: cluster, node, namespace, controller (deployment, statefulset and so on), service, pod and container, and by labels (team, product). Access is through the Allocation API and several endpoints that expose the mechanics:

Endpoint	What it returns
`/allocation`	cost allocated by the dimension you ask for
`/costDataModel`	the resolved node price
`/allNodePricing`	hourly prices per node
`/pricingSourceSummary`	summary of the pricing source
`/metrics`	cost metrics for Prometheus

The /metrics endpoint is what turns OpenCost into a Prometheus exporter: once there, you can write PromQL to calculate the cost and efficiency of any Kubernetes concept and build panels in Grafana (OpenCost · exporter).

Allocation API parameters

/allocation is controlled by a handful of parameters worth knowing, because they define which number you get:

Parameter	What it controls	Example
`window`	time range	`7d`, `today`, specific dates
`aggregate`	aggregation dimension	`namespace`, `label:team`, `pod`
`accumulate`	sum the range or break it per interval	`true` (total) / `false` (series)
`idle`	include idle cost or not	`true` / `false` / `separate`
`filter`	filter by namespace, label and so on	`namespace:"llm-prod"`

The idle parameter is the most revealing: with idle=separate, OpenCost returns the idle cost as a separate row, which lets you see at a glance how much is being paid for unused capacity. A query with aggregate=label:team and idle=separate over window=30d is, literally, the monthly chargeback report with the waste highlighted.

GPU in depth: DCGM, usage, MIG and idle

The GPU is the expensive resource, so its allocation deserves detail:

Usage signal: dcgm-exporter gives DCGM_FI_DEV_GPU_UTIL (utilisation), DCGM_FI_DEV_FB_USED (memory), and power. OpenCost uses utilisation to allocate by usage.
MIG (Multi-Instance GPU): an A100 is split into up to 7 isolated instances, which allows cutting idle from 50 % to almost 0 % by sharing one card across small workloads ([search]; see sharing the GPU: time-slicing, MPS and MIG). The cost allocation of each MIG instance follows the same usage logic.
Idle: the reliable detection pattern combines three pieces, DCGM metrics in Prometheus, an alert that fires when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, and routing of that alert to the team that owns the namespace ([search]).

An attribution nuance worth knowing: MIG creates isolated instances that Kubernetes sees as distinct resources, so OpenCost allocates them cleanly, each one to its pod. Time-slicing, by contrast, shares the same physical GPU across several pods with no compute isolation: there, per-pod utilisation is harder to separate, and cost attribution becomes approximate (the GPU is split across the pods sharing it, according to their measured usage). The practical rule: if you need exact per-team chargeback on a shared GPU, MIG gives cleaner attribution than time-slicing, at the price of rigid fixed partitions. It is a trade-off between cost accuracy and flexibility that is better decided before, not after, building the split.

Efficiency: the number that triggers action

With cost allocated by usage, OpenCost lets you calculate efficiency: how much of the allocated cost corresponds to useful work against the total. A GPU at 30 % utilisation has an efficiency of 30 %: the remaining 70 % is idle cost that somebody pays without receiving work. That figure, per team and per namespace, is what turns a dashboard into a chargeback conversation: “your namespace has 5 GPUs allocated at 25 % efficiency; either you raise utilisation or you release three cards”. It connects with FinOps and multi-tenancy: allocation locates the waste; scheduling and co-residency recover it.

Worked example: the allocation report

On the ~11 €/h node (8×H100), three teams share the cluster for a month. What a /allocation?aggregate=label:team&idle=separate&window=30d query returns:

Team	GPU-hours	Avg. util.	Allocated cost (€)	Efficiency
A · chat-prod	2,880 (4 GPU)	78 %	~3,744	high
B · batch	1,440 (2 GPU)	55 %	~1,872	medium
C · experimentation	1,440 (2 GPU)	22 %	~1,872	low
idle (separate)	—	—	~1,100	—

What the report reveals: the cost allocated to A, B and C comes out of their GPU-hours and the node price; but the idle row (~1,100 €/month) is capacity nobody used and everyone pays for in the pro rata. And the efficiency column points at C: 2 GPUs at 22 % is ~1,560 € a month of which only ~340 € is useful work. That is the actionable figure: C does not have a model problem, it has a utilisation problem. Without OpenCost, those ~1,100 € of idle and C’s inefficiency stay diluted in an aggregate invoice nobody questions. With it, they are a named row and a number in euros.

Useful PromQL queries

As a Prometheus exporter, OpenCost lets you build the panels by hand. Some reference queries (exact metric names vary by version; check in your /metrics):

# Cost per node and hour (resolved price)
node_total_hourly_cost

# GPU cost allocated per namespace (€/hour)
sum by (namespace) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
)

# Average GPU utilisation per namespace (for efficiency)
avg by (namespace) (DCGM_FI_DEV_GPU_UTIL)

# Idle GPUs: utilisation < 10 % sustained
DCGM_FI_DEV_GPU_UTIL < 10

The last one, wrapped in an alert with for: 15m and routed to the namespace owner, is the standard idle detection pattern. With these series in Grafana you have cost per team, efficiency and waste on one dashboard, the “Inform report” of the FinOps phase in real time.

Dynamic cloud prices vs custom on-prem prices

The difference is worth understanding, because it changes how reliable the figure is:

Cloud (aws/azure/gcp/oracle/ovh): OpenCost obtains the price dynamically from the provider’s billing API, so the allocated cost reflects the real rate (including discounts, spot and so on) without you declaring anything.
On-prem (default/custom): you declare the price, derived from your TCO model. The advantage is total control; the risk is that a badly calculated price skews everything. That is why the TCO model (depreciated capex + opex) is not an academic exercise: it is literally OpenCost’s input.

The ovh provider among those supported is relevant for a European platform: if part of the workload goes to a European sovereign cloud, OpenCost can allocate its cost with dynamic pricing, and mix it with on-prem in the same allocation view.

From pod cost to cost per token

OpenCost gets as far as “this vLLM pod cost X €/hour”. The cost per token, the metric that compares on-prem against cloud, needs one more piece: the gateway (LiteLLM) that counts tokens per request and per team, as covered in the FinOps introduction. The full chain is: OpenCost gives the pod cost by usage → the gateway gives the tokens per team → dividing them gives the cost per token per team. OpenCost is the hardware half of that equation; without it, you would know the tokens but not their real cost. That is why A2 (allocation) is the foundation the whole cost per token of the proposal hangs from.

Deployment: Helm + Prometheus

The typical deployment is via Helm, with Prometheus as a prerequisite (for scraping and series storage). Conceptual steps:

Have Prometheus with kube-state-metrics, node-exporter, cAdvisor and dcgm-exporter.
Install OpenCost with Helm, pointing it at your Prometheus.
Configure the node price in euros (default.json / values.yaml), the step most often neglected.
Expose /metrics and build panels in Grafana with PromQL over the cost metrics.

As a Prometheus exporter, OpenCost lets you write PromQL for the cost and efficiency of any concept and create bespoke dashboards (OpenCost · exporter). It reuses infrastructure a GPU cluster already has for observability (DCGM, Prometheus, Grafana): OpenCost only adds the pricing and allocation layer.

OpenCost vs Kubecost (just the essentials)

OpenCost is the free base; Kubecost (IBM) is the commercial product built on it. Briefly, what the commercial one adds:

Capability	OpenCost	Kubecost
Cost allocation (CPU/GPU/mem/PV)	✓	✓
GPU via DCGM	✓	✓ (3.0)
Long historical retention	depends on your Prometheus	included
Automatic rightsizing	—	✓ (Turbonomic)
Governance, alerts, enterprise RBAC	basic	✓
Commercial support	community	✓ (IBM)

For most on-prem clusters with a platform team, OpenCost covers allocation; Kubecost adds value when you want automated optimisation and enterprise features without operating it yourself. The detail of the comparison, and when paying is worth it, is article A3.

Implementation checklist

For OpenCost to give a defensible per-team cost, order matters:

Step	Action	Verification
1	Prometheus with kube-state-metrics, node-exporter, cAdvisor	series present
2	NVIDIA GPU Operator + dcgm-exporter	`DCGM_FI_DEV_GPU_UTIL` in Prometheus
3	Install OpenCost (Helm) pointing at Prometheus	`/allocation` responds
4	Configure the node price in € (capex+opex)	`/allNodePricing` shows the real one
5	Label pods/namespaces by team and product	`aggregate=label:team` splits correctly
6	Grafana panels for cost, efficiency and idle	the waste is visible
7	Idle alert (`util<10` `for: 15m`) to the owner	the alert reaches the team
8	(optional) long-retention backend	months of history available

Step 4 is the one most often skipped and the one that most invalidates the result. If you only do one thing right, make it that one: without the correct node price in euros, steps 5 to 7 split a wrong cost with great precision.

Limits and traps (data-driven)

Default under-pricing. With the default pricing config, OpenCost underestimates on-prem CPU and GPU (#3781). Adjust the base prices to your real cost or the whole split will come out low.
It depends on Prometheus and DCGM. Without the right series (especially dcgm-exporter), there is no GPU allocation. Input data quality rules.
Allocation by usage ≠ total cost. A GPU at 50 % charges half; the other half is idle that still costs. Do not confuse “allocated cost” with “cost paid”.
Prometheus retention. Historical cost analysis depends on how much your Prometheus retains; for long series, plan a long-term backend.
Allocation, not per-token measurement. OpenCost reaches the pod; cost per token needs the gateway (LiteLLM) above it, as covered in the FinOps introduction.

With OpenCost’s mechanics clear, the next article (A3) compares what Kubecost and the commercial alternatives add, to decide which FinOps stack to adopt. But the foundation, correct allocation with the node price properly set in euros, is this one.

Closing

OpenCost looks like “installing a cost tool” and is, in reality, declaring what your hardware is worth and letting it split itself. That simplicity is its strength, it lives on top of the Prometheus and DCGM you already have, with no new database, and also its trap: the split is only as good as the node price you declare, and the default value under-prices on-prem GPU. Properly configured in euros, OpenCost turns the cluster’s aggregate invoice into a split per team, per product and per GPU, separates used cost from idle, and exposes the efficiency that triggers chargeback conversations. It is the hardware half of the cost per token equation, the other half, the tokens, is supplied by the gateway, and without this half there is no TCO model or on-prem vs cloud comparison that holds up. For a sovereign architecture proposal, OpenCost is the piece that makes cost stop being an intuition and become a table in euros, reproducible and auditable, with the idle GPU named and identified. The Inform phase of GPU FinOps starts here, and everything else hangs on this number being right.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · configuración (proveedores: aws/azure/gcp/oracle/ovh/default) — https://opencost.io/docs/configuration/
OpenCost · configuración on-prem (precios personalizados) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exportador de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
OpenCost · issue #3781 (infra-precio CPU/GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Grafana · manage costs (OpenCost + Kubernetes) — https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/manage-costs/

GitOps for the inference stack with Flux: operating the assistant as code

Thu, 11 Jun 2026 10:20:00 +0000

This post is part of the operational series on how to squeeze a generic on-premise 4×H100 SXM 80 GB LLM cluster. The sibling pieces: the document ingestion pipeline for RAG that fills the vector store we deploy here, the embeddings and reranker service with TEI that is one of the services GitOps manages, and the secrets hardening of the sovereign stack, which goes deep into the secrets problem we only state here. The complete end-to-end assistant (LibreChat + LiteLLM + RAG) that orchestrates all of this will get its own post.

TL;DR

An LLM assistant in production is a distributed system, not a binary. At a minimum it has an inference engine (vLLM or similar), an L7 gateway that routes per model and applies rate limiting, a chat front end, a vector store for RAG, an embeddings/reranker service, and an observability stack. Six substantial services, spread across three or four namespaces, with start-up dependencies (the front end is useless if the gateway is not there, the gateway is useless if the engine has not loaded the model, RAG does not answer if the vector store is empty). Operating it by hand, a kubectl apply here and a helm upgrade there, produces a cluster whose real state nobody can reconstruct: there is no record of what was applied, or when, or why. GitOps solves this with four principles (OpenGitOps): the desired state is declarative, it is versioned and immutable in git, it is applied automatically by an agent (nobody SSHes in to deploy) and it is continuously reconciled by comparing the actual state against the declared one. Flux is that agent: six controllers (source, kustomize, helm, notification, image-reflector, image-automation) that clone the repo, render Kustomize/Helm, apply to the cluster, correct drift and, optionally, write back to git to bump the tag of a new image. The unit of work is the Kustomization or the HelmRelease, with interval (how often it reconciles), prune (deletes what is no longer in git), dependsOn (orders the start-up) and health checks. The interval sets the MTTR of drift: with interval=1m, a manual edit to the cluster is reverted in ≤ 1 min on average. Secrets cannot sit in the clear in git, a chicken-and-egg problem, and it is solved with SOPS/age, sealed-secrets or External Secrets + Vault. Rollback is a git revert. GitOps is not free: the learning curve is real, and debugging why a Kustomization is not reconciling is a new skill.

The analogy: the master plan and the foreman who does not negotiate

Picture a large building site with a signed and filed master plan, and a foreman who has a single order: the site must be identical to the plan, at all times. The foreman does not improvise. Every so often he walks the site with the plan in hand and compares: if a wall is where the plan says, he leaves it; if someone moved a partition overnight without updating the plan, he puts it back; if the plan says there is a column and it does not exist, he builds it; if a column exists but no longer appears on the plan, he knocks it down. The plan is the single source of truth: to change the site you do not touch the site, you change the plan, and the foreman takes care of the rest on his next round.

That is exactly the mechanics of GitOps. Git is the master plan: declarative (it describes the end state, not the steps), versioned (every change is signed into the history, with author and reason), immutable (a commit is not rewritten). Flux is the foreman: every interval it walks the cluster, compares against git and converges. And here is the lesson that separates GitOps from “keeping the YAML in a repo”: the foreman undoes manual changes. If an operator comes in with kubectl edit and raises the inference engine replicas from 2 to 4 at three in the morning to put out a fire, on the next round Flux takes it back to 2, because the plan says 2. This enrages anyone coming from the imperative world, and it is precisely the point: if you want 4 replicas permanently, you update the plan. The cluster stops being a system with its own opaque memory and becomes a reproducible projection of git. Delete the entire cluster, point a fresh Flux at the same repo, and the site is rebuilt identically.

The analogy also marks the boundary: the plan describes the building, not who holds the keys to the materials store. Secrets (passwords, tokens, keys) cannot go into the public plan. That is the chicken-and-egg problem we cover further down and that the hardening piece develops in full.

The problem: an assistant is six services, not one

Before Flux, it is worth fixing the size of the problem. A minimally serious sovereign LLM assistant, deployed on the generic reference cluster, has this service topology:

Service	Function	Typical namespace	State
Inference engine (vLLM)	Serves the tokens of the general LLM and the code one	`llm-serving`	Stateless (model from object store)
Embeddings + reranker (TEI)	Vectorises queries and reorders candidates for RAG	`llm-serving`	Stateless
Vector store (Qdrant)	Stores and searches document embeddings	`data`	Stateful (PVC)
L7 gateway (Envoy AI Gateway / LiteLLM)	Per-model routing, rate limiting, auth	`gateway`	Semi-stateless
Chat front end	Assistant UI	`apps`	Stateless
Observability (Langfuse, OTel Collector, dashboards)	LLM-aware traces, metrics and logs	`observability`	Stateful (Langfuse Postgres)

Six services, four or five namespaces, two components with persistent state, and a real dependency graph: the front end calls the gateway, the gateway calls the engine and the embeddings service, RAG depends on the vector store being full, and everything emits traces to observability. Deploying this by hand means remembering the order, the values of each helm install, the ConfigMaps, the Secrets, the nodeSelectors that pin the GPU pods to the right nodes. Do it twice (a staging environment and a production one) and the two diverge within days. That is exactly what GitOps eliminates.

The six Flux controllers

Flux is not a monolithic binary but a set of cooperating controllers, the GitOps Toolkit. A default installation brings four; the two image automation ones are added with --components-extra (Flux installation):

source-controller: clones and keeps the sources up to date, whether git repos (GitRepository), Helm repos (HelmRepository), buckets or OCI artifacts. It is the one “holding the plan”: it exposes the repo contents as an internal artifact that the others consume.
kustomize-controller: takes a source artifact, renders Kustomize (bases + overlays) and applies the result to the cluster. It is responsible for prune, dependsOn and the health checks of the Kustomizations.
helm-controller: reconciles HelmRelease objects, installing and updating Helm charts declaratively, without anybody running helm from a terminal (helm-controller).
notification-controller: the bridge to the outside world in both directions. It receives webhooks (to reconcile instantly on every push, instead of waiting for the interval) and emits events and alerts to Slack, an internal chat or an incident system.
image-reflector-controller: scans container registries and stores the tags it finds in an internal database. It is the “eyes” of image automation.
image-automation-controller: uses what the eyes see to write back to git, committing the new image tag into the manifests when a version appears that satisfies the policy.

A cluster without image automation does not need the last two. But for an inference stack where the model server is updated fairly often, they are what automates version promotion without touching anything by hand.

The diagram has three paths. The grey one is the main flow: git → source-controller → kustomize/helm-controller → cluster. The dashed red one is the reconciliation loop that runs every interval: it compares actual state against desired, fixes drift and prunes what is left over. The purple one is image automation: the reflector sees a new tag in the registry, the policy decides whether it qualifies, and the automation-controller writes back to git. That last path is the one that closes the circle and turns git into a system that updates itself.

Repo structure: clusters, infrastructure, apps

The most widespread convention separates three levels of responsibility. It is not the only one, but it ages well:

gitops-repo/
├── clusters/
│ └── prod/
│ ├── infrastructure.yaml # Kustomization → ./infrastructure
│ └── apps.yaml # Kustomization → ./apps (dependsOn infrastructure)
├── infrastructure/
│ ├── controllers/ # ingress, cert-manager, GPU operator, KEDA...
│ └── configs/ # ClusterIssuer, RuntimeClass, StorageClass...
└── apps/
 ├── base/ # manifests common to all environments
 │ ├── llm-engine/ # vLLM HelmRelease
 │ ├── gateway/ # L7 gateway HelmRelease
 │ ├── chat-front/ # Deployment + Service + Ingress
 │ ├── vector-store/ # Qdrant HelmRelease (+ PVC)
 │ └── observability/ # Langfuse + OTel HelmRelease
 ├── staging/ # overlay: low replicas, small model
 └── prod/ # overlay: high replicas, large model, MIG

The key piece is Kustomize’s base / overlay separation. The base/ describes the service once; each overlay (staging/, prod/) applies a patch with the environment’s differences: number of replicas, model size, MIG profile, nodeSelector, --gpu-memory-utilization. That avoids duplicating complete manifests per environment. The clusters/ folder is the entry point Flux reconciles first: it holds the root Kustomizations that point to infrastructure/ and apps/, with a dependsOn that guarantees the infrastructure (CRDs, operators, storage classes) is ready before the applications.

How many manifests: the calculation that justifies overlays

This is where the numbers make the argument. Without overlays, each service needs a complete set of manifests per environment. With $N$ services and $M$ environments, the cost in files to maintain is:

$$ \text{files}_{\text{naïve}} = N \times M \times k $$

where $k$ is the average number of manifests per service (Deployment/HelmRelease + Service + ConfigMap + PVC + Ingress ≈ 5). For our assistant, $N=6$ services and $M=3$ environments (dev, staging, prod), with $k=5$:

$$ 6 \times 3 \times 5 = 90 \text{ complete files, each maintained separately.} $$

With the base/overlay structure, the base is written once and each overlay contains only the patch with the differences (typically 1 short file per service per environment):

$$ \text{files}_{\text{overlay}} = \underbrace{N \times k}_{\text{base}} + \underbrace{N \times M}_{\text{patches}} = 6 \times 5 + 6 \times 3 = 30 + 18 = 48 $$

It is not just that there are fewer files (48 against 90): it is that the bulk of a change happens in a single place. Changing the engine’s memory limit for every environment is one edit in base/, not three synchronised edits. The saving factor grows with $M$: for 5 environments, the naïve version is 150 files and the overlay one is 60. Duplication is the enemy of auditability, and overlays attack it at the root.

The reconciliation loop: desired vs actual

The heart of Flux is a control loop that never ends: every interval it reads the desired state (git), reads the actual state (cluster), computes the difference and applies it. It is the same principle as a thermostat: it reads the target temperature (git), reads the actual temperature (cluster) and switches the boiler on or off until they match. It does not “deploy once”; it converges forever.

A Kustomization manifest for the applications, with the pieces that matter (Kustomization reference):

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: apps
 namespace: flux-system
spec:
 interval: 1m # reconciles every minute: sets the drift MTTR
 path: ./apps/prod  # production overlay
 prune: true # deletes from the cluster whatever is removed from git
 sourceRef:
 kind: GitRepository
 name: gitops-repo
 dependsOn:
 - name: infrastructure  # does not apply apps until infra is Ready
 wait: true # waits for the resources to be healthy
 timeout: 5m  # fails the reconciliation if it does not converge in 5 min
 healthChecks:
 - apiVersion: apps/v1
 kind: Deployment
 name: gateway
 namespace: gateway

Four fields do the heavy lifting:

interval defines how often the foreman’s round runs. It is the direct determinant of the drift MTTR (next section).
prune: true switches on garbage collection: objects that were applied before but are no longer in the current git revision are deleted from the cluster automatically. Without prune, a service withdrawn from the repo keeps running, and the plan and the site diverge silently.
dependsOn orders the graph: Flux does not apply the manifests of a Kustomization until every referenced one is in state Ready: True. It is the mechanism that guarantees data → gateway → front end.
wait + healthChecks: with wait: true, Flux monitors every applied resource and waits for them to be ready before marking the reconciliation as successful; healthChecks lets you tune exactly which resources to watch. That is what makes a dependsOn mean something: the dependency is not satisfied until it is healthy, not merely applied.

MTTR and drift detection as a function of the interval

The interval is the only parameter the operator chooses to govern the speed of the loop, and it translates directly into operational metrics. If a deviation (drift) happens at a random instant within the reconciliation period $T$, the waiting time until Flux detects it is uniformly distributed over $[0, T]$, with mean:

$$ \mathbb{E}[t_{\text{detection}}] = \frac{T}{2} $$

To that you add the correction time $t_c$ (render, diff, apply), normally a few seconds. The drift MTTR comes out as:

$$ \text{MTTR}_{\text{drift}} = \frac{T}{2} + t_c $$

With interval=1m and $t_c \approx 10\text{s}$, drift is corrected in $\frac{60}{2} + 10 = 40$ s on average, with a worst case of $60 + 10 = 70$ s. With interval=10m, the mean rises to $5\text{min}\;10\text{s}$ and the worst case to more than 10 min: a window in which a manual change stays active. The temptation is to set interval=10s and forget about it, but there is a cost: every reconciliation consumes CPU and, above all, makes requests to the cluster API and to the registry. With dozens of Kustomizations reconciling every 10 s, the kube-apiserver and the image-reflector start to feel the pressure. The practical rule: a short interval (1m) for critical applications whose drift hurts, a long interval (10–30m) for stable infrastructure that almost never changes, and notification-controller webhooks for instant reconciliation on every push, so that the interval governs only drift and not deployment latency.

Helm as code: the inference engine HelmRelease

For the engine, the gateway and the front end, the usual approach is to package them as Helm charts and declare them with HelmRelease. The helm-controller reconciles them without anyone ever running helm:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: llm-engine
 namespace: llm-serving
spec:
 interval: 10m
 chart:
 spec:
 chart: vllm
 sourceRef:
 kind: HelmRepository
 name: vllm-charts
 values:
 image:
 tag: "0.8.4" # managed by image automation (see below)
 replicaCount: 2
 nodeSelector:
 nvidia.com/gpu.product: H100-SXM
 resources:
 limits:
 nvidia.com/gpu: 1
 extraArgs:
 - "--gpu-memory-utilization=0.90"
 - "--tensor-parallel-size=1"

Everything that in the imperative world would be a vllm serve flag, --gpu-memory-utilization or --tensor-parallel-size, now lives as versioned data. Changing the fraction of VRAM the engine reserves is a commit, reviewable in a PR, with history. The relationship with sharing a GPU is direct: the MIG profile and the --gpu-memory-utilization are deployment parameters, and here they are code.

Image automation: bumping the tag without touching anything by hand

The inference engine is updated fairly often (vLLM patches, security fixes). Without automation, every new version requires somebody to edit the tag in the HelmRelease. Flux image automation does it on its own, with three objects (automate image updates):

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
 name: vllm
 namespace: flux-system
spec:
 image: registry.example.local/inference/vllm
 interval: 5m  # scans the registry tags every 5 min
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
 name: vllm
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: vllm
 policy:
 semver:
 range: ">=0.8.0 <0.9.0" # only patches and minors within 0.8.x–0.8.x
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
 name: vllm
 namespace: flux-system
spec:
 interval: 5m
 sourceRef:
 kind: GitRepository
 name: gitops-repo
 git:
 commit:
 author:
 name: fluxbot
 email: fluxbot@example.local
 messageTemplate: "auto: bump vllm to {{ .NewTag }}"

The division of roles: ImageRepository scans all the tags of the image repository and stores them in the reflector’s internal database. ImagePolicy reads those tags and picks the “latest” according to the policy; the policy field is mandatory and defines how the selection is made (ImagePolicy). ImageUpdateAutomation takes the chosen tag, edits the manifest in the repo (wherever there is a # {"$imagepolicy": "flux-system:vllm"} marker) and commits (ImageUpdateAutomation).

The choice of policy matters:

semver: interprets tags as semantic versions and picks the highest one that satisfies the range (range: ">=0.8.0 <0.9.0"). By default it excludes prereleases (0.8.0-rc.1 does not qualify) unless they are asked for explicitly. It is the right option for production: you stay inside a tested version range and do not jump to a new major by accident.
regex / alphabetical / numerical: for tag schemes that are not pure semver, for example main-<sha>-<timestamp> from an internal pipeline. More flexible, but it forces you to trust that tag order reflects version order, which is fragile.

A safety guideline: instead of having ImageUpdateAutomation push directly to main, configure it to write to a branch and open a PR. That way the version bump goes through human review or a validation pipeline before reaching the cluster. Automation proposes; the human (or an automatic gate) disposes. And for production, the version bump is not the deployment: it is the trigger for the progressive rollout described in canary, blue-green and shadow.

Secrets in GitOps: the chicken-and-egg problem

GitOps demands that all the desired state is in git. But secrets, the Langfuse Postgres password, the registry token, the gateway API keys, cannot go into a repo in the clear, not even a private one: git history is immutable, and a secret committed once stays there forever. This is the fundamental tension: GitOps wants everything in git; security forbids secrets in git. There are three families of solution, all with the same underlying idea, that only the encrypted secret goes into git and the cluster holds the key to decrypt it:

SOPS + age/KMS: you encrypt the sensitive values of a manifest with SOPS (it leaves the keys readable and encrypts only the values). The encrypted YAML goes to git; the kustomize-controller carries an age key or reaches a KMS to decrypt it at apply time. Simple, with no extra components in the cluster.
sealed-secrets: a controller in the cluster holds a private key. You encrypt the secret against its public key (with kubeseal), the encrypted SealedSecret goes to git, and the controller decrypts it into a normal Secret inside the cluster. The private key never leaves the cluster.
External Secrets + Vault: not even the encrypted secret goes into git, only a reference (ExternalSecret) saying “the value of this key is in Vault, at this path”. The External Secrets operator resolves it at apply time. It is the cleanest pattern for many secrets and frequent rotation, at the cost of operating a Vault.

The choice depends on the volume of secrets and on whether you already have a central manager. For a small stack, SOPS+age is enough and has no dependencies. The sibling hardening piece goes into the detail of each option, key rotation and the threat model.

Promotion and rollback: git revert is the panic button

The most elegant consequence of having the cluster as a projection of git is that rollback is a git revert. If a deployment breaks production, the new engine gives worse latency or the gateway starts returning 5xx, there is no need to remember which version was there before or to rebuild the state by hand. You revert the commit that introduced the change, and on the next reconciliation (≤ interval, or instantly with a webhook) Flux returns the cluster to the previous state. The git history is, literally, the deployment history: every commit is a restore point with author, date and reason.

This fits with the promotion strategies of the canary, blue-green and shadow post: Flux manages what is declared, and the progressive rollout tool manages how traffic moves between the old version and the new one. The git revert is the last-resort rollback (it returns everything to a known state); the canary is the mechanism that avoids needing it. For services where the version change is delicate, the right approach is for image automation to do the bump on a branch, for the canary to validate the regression gates, and only then to promote the commit to main.

The seams: GitOps is not free

It would be dishonest to sell GitOps as magic. It has real costs that any team pays:

Learning curve. The declarative model is a change of mindset. The operator who has spent years fixing incidents with kubectl edit has to unlearn the reflex: now the cluster reverts their changes and that, at first, feels like Flux “fighting them”. Understanding that the change is made in git, not in the cluster, takes weeks of discomfort.

Debugging the reconciler. When a Kustomization does not apply, the error is not in the pod; it is in the reconciliation chain. Did the source-controller clone the right revision? Did the kustomize-controller render the overlay properly? Did a dependsOn health check fail? Did the timeout fire before the large model finished loading? Diagnosing this needs flux get, flux logs, reading the status.conditions of the Flux objects and understanding which link got stuck. It is a new skill, different from debugging plain Kubernetes.

Timeouts and slow loads. The timeout of a Kustomization with wait: true has to be longer than the start-up time of the slowest service. An inference engine that takes 4 minutes to load a large model from the object store will fail a Kustomization with timeout: 2m, even though everything is fine. Calibrating timeouts per service is fine-grained work.

The drift you did want. Sometimes the operator needs an urgent temporary change and Flux reverts it. The correct answer, suspending reconciliation of that Kustomization with flux suspend, making the change, then reflecting it in git and resuming, is discipline that has to be built. Without that discipline, people end up fighting the foreman at 3 in the morning.

None of these costs cancels the benefit. But a team adopting GitOps expecting “everything to be easier from day one” gets frustrated. It is easier from month two onwards, once auditability, reproducibility and trivial rollback have paid off the curve.

Applied to the generic 4×H100 cluster

On the reference cluster (4×H100 SXM 80 GB per GPU node, NVLink, plus CPU and control nodes), the assistant stack is managed entirely by GitOps. Hardware decisions become data in the overlays.

nodeSelectors and MIG profiles as code. The prod overlay pins each service to the right node and declares the GPU profile. The general engine uses a whole GPU; embeddings and the small LLM fit in MIG slices, exactly the split from the sharing a GPU post, but now versioned:

# apps/prod/llm-engine-patch.yaml
spec:
 values:
 nodeSelector:
 nvidia.com/gpu.product: H100-SXM
 extraArgs:
 - "--gpu-memory-utilization=0.92"
 - "--tensor-parallel-size=1"
---
# apps/prod/embeddings-patch.yaml — on a MIG slice
spec:
 values:
 nodeSelector:
 nvidia.com/mig.config: "3g.40gb"
 resources:
 limits:
 nvidia.com/mig-3g.40gb: 1

The MIG profile (3g.40gb), the VRAM fraction (0.92), the tensor-parallel setting: all of it is text in a PR. Changing the GPU split between services is a reviewable commit, not a hand-run nvidia-smi mig session that nobody records.

Start-up order with dependsOn. The assistant’s dependency graph is encoded with chained dependsOn. The right order is data → embeddings/engine → gateway → front end:

# apps/prod/gateway.yaml
spec:
 dependsOn:
 - name: llm-engine  # the gateway is useless without the engine
 - name: embeddings
 # ...
---
# apps/prod/chat-front.yaml
spec:
 dependsOn:
 - name: gateway  # the front end is useless without the gateway

With wait: true on each Kustomization, Flux does not mark llm-engine as Ready until the engine pod passes its health check, which in the case of vLLM means model loaded and endpoint responding, not just pod started. Only then does it begin applying the gateway. This avoids the classic cascade of failures of manual deployment: bring up the front end first, see a 502 because the gateway is not there, bring up the gateway, see a 503 because the engine is still loading the model. With dependsOn + wait, the order is guaranteed by the reconciler, not by the operator’s memory.

The result: deleting the assistant’s entire namespace and letting Flux rebuild it from git produces exactly the same stack, in the same order, with the same GPU configuration. The 4×H100 cluster stops having a configuration known only to whoever built it, and becomes a reproducible projection of a repo that anyone can audit.

Conclusion

GitOps is not a tool, it is an inversion of the direction of control: instead of pushing changes to the cluster, you declare the state in git and let an agent pull the cluster towards it. For an LLM assistant (six services, several namespaces, start-up dependencies, delicate GPU configuration) that inversion turns a fragile and opaque system into a reproducible and auditable one. Flux is the foreman who keeps the site identical to the plan and undoes any change that has not gone through the plan first. The price is a real learning curve and a new debugging skill. The prize is that the cluster stops keeping secrets about itself: everything it is, is written down, signed and reproducible in git.

References

OpenGitOps — GitOps principles — opengitops.dev
Flux installation — fluxcd.io/flux/installation
Flux Kustomization — fluxcd.io/flux/components/kustomize/kustomizations
helm-controller — github.com/fluxcd/helm-controller
Flux Image Policies — fluxcd.io/flux/components/image/imagepolicies
Flux Image Update Automations — fluxcd.io/flux/components/image/imageupdateautomations
Automate image updates to Git — fluxcd.io/flux/guides/image-update
SOPS — github.com/getsops/sops

Sharing one GPU between several workloads: time-slicing, MPS and MIG

Thu, 11 Jun 2026 02:00:00 +0000

This post opens an operational series on how to squeeze a generic on-premise 4×H100 SXM LLM cluster. The sibling pieces: Serving several models on one GPU: swap and sleep (what to do when the models do not fit at the same time and have to take turns in memory), RAG on CPU: separating the data plane from generation (moving retrieval off the GPU to free it up) and Sovereign end-to-end assistant with LibreChat, LiteLLM and RAG, the final assembly, a fourth instalment in preparation. Here we start with the most basic question: you have one GPU and you want to put several workloads on it. How do you split it?

TL;DR

You have one GPU, or a few, and several workloads that want to run on it: a chat model, an embeddings service, a reranker, a queue of dev jobs. The GPU is underused if it only runs one thing, but piling several on carelessly causes contention, OOM or cascading failures. There are three mechanisms and they share out different things. Time-slicing (replicas from the NVIDIA k8s device-plugin) multiplexes in time: it advertises the GPU as “N GPUs” and the processes take turns at the compute, but they share the full physical VRAM, with no memory isolation, no fault isolation and no QoS. Its trap is an OOM that does not show up in the Kubernetes scheduler but at run time, when the sum of VRAM allocations exceeds the real memory. MPS (Multi-Process Service) multiplexes in space: it shares out the SMs between processes that execute kernels concurrently, reduces context-switch overhead and allows SM and memory limits per process. It raises throughput when there are many small kernels, but fault isolation is still weak. MIG (Multi-Instance GPU) partitions in hardware: it cuts the Hopper GPU into up to seven instances with dedicated SMs, L2, memory and bandwidth, with real memory, fault and performance isolation; datacenter only (A100/H100/H200/B200), never on an RTX 5090. The rule: real isolation / multi-tenant / compliance → MIG (if it is Hopper); many small concurrent kernels and trust between workloads → MPS; dev, bursts, consumer GPU or no need to isolate → time-slicing. This post works it through with numbers: the VRAM budget of four vLLM instances on an H100 advertised as four replicas, and what fits into a 10 GB MIG instance.

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Imagine you have a single professional hob and three orders to cook at the same time. There are three ways to organise it, and they are exactly the three mechanisms.

Time-slicing is one hob taken in turns, with no larder of your own. Each cook comes in, cooks their dish, leaves, and the next one comes in. The split is temporal: nobody cooks at the same time, they take turns. The problem is not the hob, which gets shared out fine, but the common larder: the ingredients sit in a single shared cupboard and nobody has their own. If the three cooks reserve more flour than there is in total, it is not that they wait their turn: it is that there is no flour. The service goes down for everyone. And if one cook leaves a pan burning and starts a fire, it burns the whole kitchen, not just their corner.

MPS is several cooks coordinated on the same worktop. Now they do cook at the same time, sharing out the space on the worktop (the SMs). A head chef (the MPS daemon) coordinates so they do not collide and so the worktop does not sit empty while one of them waits for water to boil. You can assign each cook a percentage of the worktop and a larder limit. They work faster as a group because the worktop does not sit idle between small tasks. But they still share the kitchen: if one starts a serious fire, the others notice.

MIG is several independent kitchens in the same building. A concrete wall separates each kitchen: its own hob, its own larder, its own door and its own electrical panel. What happens in kitchen 3, a fire, an empty larder, a slow cook, does not touch kitchen 1. It is the only split with genuine isolation. The price: you have to decide in advance how many kitchens and of what size, the walls are fixed, and only expensive buildings (datacenter) come ready to put them up.

The rest of the post is, essentially, when you want cheap turns, when you want coordinated cooks and when you need concrete walls.

An H100 SXM 80 GB does not fill up with just any workload. A bge-reranker-v2-m3 reranker takes a few hundred MB and saturates a handful of SMs; a bge-m3 embeddings service is just as small; a 1B guardrail model in INT4 fits in a couple of GB. Dedicating 80 GB of HBM3 and 132 SMs to serving embeddings is using a hydraulic press to hammer in a drawing pin, the same argument as in mixed environments, but now inside the GPU instead of moving the workload to different silicon.

The aim of sharing is to raise the useful utilisation of fixed capital. But sharing badly introduces three pathologies:

Compute contention: two workloads fight over the same SMs and both run slowly, with unpredictable latency jitter.
Memory contention: the sum of requested VRAM exceeds the physical amount and something dies with a CUDA out of memory.
Cascading failure: a workload that blows up (an illegal kernel, an OOM) can drag its neighbours down with it if they share a context.

The three mechanisms attack these pathologies to different depths. None of them solves all three except MIG, and MIG costs specific hardware. Let us look at them one by one.

Time-slicing: compute turns, shared larder

Time-slicing is temporal multiplexing in software. On Kubernetes, the NVIDIA GPU Operator configures the device-plugin to advertise N replicas of each physical GPU. An H100 declared with replicas: 4 appears to the scheduler as four nvidia.com/gpu resources, and Kubernetes can place four pods on it. Internally, the GPU scheduler hands out compute turns to each process: it runs a bit of process A, switches to B, to C, to D, then back to A. It is the same time-sharing an operating system does with the CPU.

The key idea, and the one that causes most confusion, is this: a replica is NOT a fraction of the GPU. It is a compute turn. NVIDIA’s documentation is explicit: unlike MIG, there is no memory or fault isolation between replicas. The four replicas of the H100 see the full 80 GB of VRAM, unpartitioned. There is no 20 GB per replica. There are 80 GB for all four, handed out in cudaMalloc arrival order.

This has three consequences you need to internalise:

It does not isolate memory. If the sum of what the four processes reserve exceeds 80 GB, the fourth cudaMalloc fails with OOM. The Kubernetes scheduler does not see it coming: it counted four available nvidia.com/gpu resources and happily placed four pods. The OOM appears at run time, not at scheduling time. This is trap number one of time-slicing.
It does not isolate faults. A process that triggers an unrecoverable CUDA error can leave the GPU context in a state that affects its neighbours. They share the same device with no barriers.
It gives no compute QoS. Under contention, the turn-taking does not guarantee a minimum fraction to anyone. Each workload’s latency suffers jitter proportional to how many active replicas are fighting over the GPU at that instant. A latency-sensitive workload (an interactive chat) can see its TTFT dance around depending on what its neighbours are doing.

What is it good for, then? For dev, bursts and low utilisation. If you have four developers who touch the GPU sporadically, advertising four replicas lets all four have access without fighting almost ever (they rarely overlap while active). For batch workloads that tolerate jitter. And, a decisive advantage, it works on consumer GPUs: an RTX 5090 32 GB does not support MIG, but it does support time-slicing. It is the only “Kubernetes-native” way to share a 5090 between several pods.

The VRAM budget in time-slicing (the calculation that avoids the OOM)

Here is the maths you have to do before deploying, because Kubernetes will not do it for you. Suppose an H100 80 GB advertised as 4 replicas and we want to run four vLLM instances on it, one per replica.

vLLM reserves memory with the --gpu-memory-utilization parameter, which is the fraction of the total physical VRAM that each instance keeps (for model weights plus KV-cache). The detail that kills: that fraction is computed over the 80 physical GB, not over a supposed “20 GB of my replica”, because the replica does not have 20 GB. Remember, there is no memory partitioning. Each vLLM sees the 80 GB and reserves its fraction of them.

The no-OOM constraint is therefore that the sum of fractions be less than 1:

$$\sum_{i=1}^{N} g_i < 1 \quad\Longleftrightarrow\quad \sum_{i=1}^{N} g_i \cdot V_{\text{HBM}} < V_{\text{HBM}}$$

where $g_i$ is the --gpu-memory-utilization of instance $i$ and $V_{\text{HBM}} = 80$ GB. It is wise to leave headroom (runtime overhead, fragmentation, CUDA context), so in practice you aim for the sum to stay comfortably below 1, say $\le 0.9$.

A case that works. Four vLLM instances at $g_i = 0.20$:

$$\sum_{i=1}^{4} 0.20 = 0.80 \quad\Rightarrow\quad 0.80 \times 80\ \text{GB} = 64\ \text{GB} < 80\ \text{GB} \quad\checkmark$$

Each instance reserves $0.20 \times 80 = 16$ GB. Four instances add up to 64 GB, leaving a 16 GB cushion. There is no OOM. Each vLLM has 16 GB for weights plus KV-cache: enough for a 7B–8B model in FP8/INT4 with a modest KV-cache.

A case that blows up. The same four instances, but somebody raises $g_i = 0.30$ thinking “I have four replicas, I can give each one more”:

$$\sum_{i=1}^{4} 0.30 = 1.20 \quad\Rightarrow\quad 1.20 \times 80\ \text{GB} = 96\ \text{GB} > 80\ \text{GB} \quad\times$$

The first instances start up and reserve $0.30 \times 80 = 24$ GB each. Three instances are already at $72$ GB. The fourth tries to reserve another 24 GB, there is nothing left, and it dies with CUDA out of memory. And worse: Kubernetes will reschedule it on the same GPU (it still sees four replicas), where it will die again, in a CrashLoopBackOff that makes no sense if you only look at the pod manifest.

The operational rule is brutally simple: in time-slicing, you manage the VRAM budget by hand, adding up the --gpu-memory-utilization values. The replica count controls how many pods fit in compute turns, but it does not reserve a single byte of memory. Confusing the two is the recurring mistake.

MPS: coordinated cooks on the same worktop

The Multi-Process Service (MPS) attacks a different problem. By default, when several processes use the same GPU without MPS, each one has its own CUDA context, and the GPU alternates between contexts (time-slicing at driver level): they do not execute kernels at the same time, they take turns, with context-switch overhead. If your kernels are small and do not fill the GPU on their own, this leaves SMs idle: process A uses 30 % of the SMs during its turn and the other 70 % is wasted.

MPS introduces a daemon that shares a single CUDA context between processes, so that their kernels can execute concurrently occupying different SMs at the same time. It is spatial sharing of compute: instead of taking turns at the whole worktop, each cook occupies a part and they work in parallel. This reduces context-switch overhead and raises throughput when there are many small concurrent kernels that individually do not saturate the GPU.

And, unlike pure time-slicing, MPS allows per-process limits, which gives you a form of QoS:

CUDA_MPS_ACTIVE_THREAD_PERCENTAGE limits the percentage of SMs an MPS client can use. By default each client gets $100 / \text{MaxSharedClients}$. Pinning it to, say, 40 % caps that process’s compute ceiling (MPS docs).
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT imposes a memory cap per client (valid from CUDA 11.5). This is what time-slicing does not have: a per-process VRAM limit that the runtime enforces.

These two limits turn MPS into a resource provisioning mechanism that mitigates the noisy neighbour: you can guarantee that a process does not eat more than X % of the SMs or more than Y GB. The combination gives reasonable QoS, not perfect but real.

The limitation MPS does not solve: fault isolation is weak. Since clients share the daemon’s CUDA context, a fatal error in one client can affect the daemon and therefore the other clients (historically, a client dying dirtily could require restarting the daemon). It is better than time-slicing in this respect, but a long way from hardware isolation. That is why MPS fits when there is trust between the workloads, processes from your own team, not third-party tenants.

The canonical use case: many small concurrent inference requests that individually leave the GPU half empty. MPS overlaps them and raises aggregate throughput. Serving several small models, or several light replicas of the same model, on a datacenter GPU where you trust all the workloads, is MPS territory.

MIG: concrete walls

The Multi-Instance GPU (MIG) is the only one of the three that gives genuine isolation, because it cuts the GPU in hardware. Available on modern datacenter GPUs, A100 (Ampere), H100/H200 (Hopper), B200 (Blackwell), and never on consumer ones: an RTX 5090 (consumer Blackwell) does not support MIG, nor do GeForce cards in general.

MIG divides the GPU into up to seven instances (GPU Instances), and each instance gets a dedicated portion of:

SMs (compute slices): compute is split into 7 slices, each one roughly 1/7 of the SMs.
L2 cache and memory: each instance has its slab of HBM and its portion of L2 cache.
Memory bandwidth: dedicated, not shared.
Data paths and engines: with fault barriers between instances.

The result is that a MIG instance behaves like a smaller, independent GPU: whatever happens in one, an OOM, a kernel blowing up, a workload saturating its compute, does not affect its neighbours. Memory, fault and performance (QoS) isolation, the three things time-slicing does not give and MPS only half gives.

The H100 80GB profiles

MIG does not allow arbitrary sizes: it has fixed profiles. On the H100 80GB, the profile catalogue (notation <compute>g.<memory>gb) is:

Profile	Compute (slices)	Memory	Max. instances
`1g.10gb`	1/7	10 GB	7
`1g.20gb`	1/7	20 GB	4
`2g.20gb`	2/7	20 GB	3
`3g.40gb`	3/7	40 GB	2
`4g.40gb`	4/7	40 GB	1
`7g.80gb`	7/7	80 GB	1 (whole GPU)

(There is also 1g.10gb+me, a variant with media engines for video encoding.) The base memory unit on the H100 80GB is 10 GB per slice (80 GB / 8, with one slice reserved), and the compute unit is 1/7 of the SMs. The profiles combine these units. Note 1g.20gb: the same compute fraction as 1g.10gb (1/7 of the SMs) but twice the memory, useful when a workload needs more VRAM than compute.

One important detail: MIG partitions do not mix freely. The GPU is divided following a valid geometry (the profiles fit together like pieces in a grid), and the profiles are fixed when you configure the GPU; changing them requires draining and repartitioning. They are concrete walls: solid, but they do not move while hot.

The calculation: 7×1g.10gb against 1×7g.80gb

Let us compare the two extremes. On the left, seven 1g.10gb instances: seven isolated 10 GB GPUs. On the right, a single 7g.80gb: the whole H100, unpartitioned.

The operational question is what fits in 10 GB. An instance’s VRAM budget is split between model weights and KV-cache:

$$V_{\text{inst}} = V_{\text{weights}} + V_{\text{KV}} + V_{\text{overhead}}$$

Take a model of 7B parameters in FP8 (1 byte/parameter):

$$V_{\text{weights}} \approx 7 \times 10^9 \times 1\ \text{byte} = 7\ \text{GB}$$

On a 1g.10gb instance (10 GB), after the 7 GB of weights and subtracting roughly 0.5–1 GB of runtime overhead, about 2 GB are left for KV-cache. That is enough for a modest context window and low concurrency, fine for a guardrail service, a classifier or an extraction model that handles short prompts one at a time. A 7B in INT4 (~3.5 GB of weights) leaves ~5.5 GB of KV-cache, much roomier. But a 13B model in FP8 (~13 GB of weights) does not fit in a 10 GB instance: not even the weights get in. For that you need 1g.20gb, 2g.20gb or larger.

Against this, the 7g.80gb (whole GPU) gives you the 80 GB for one large model: a 70B in FP8 (~70 GB of weights) fits with a tight KV-cache, or a 70B with more room spread across several whole GPUs in tensor-parallel (see TP against replicas: one big one versus N small ones).

The reading is clear: fine partitioning (7×1g.10gb) maximises the number of small isolated workloads; not partitioning (1×7g.80gb) maximises the model size that fits. The KV-cache available per instance shrinks proportionally as you partition, so fine-grained MIG serves many light isolated services, not one large model chopped up. If your workload is a single large model, MIG is not for you: use the whole GPU or several in TP.

The decision tree

The three questions, in order:

Do you need REAL isolation?
(multi-tenant, compliance, one workload failing must not touch another)
│
┌────┴────┐
YES NO
│ │
Is it Hopper/ Many SMALL concurrent kernels
Ampere/ AND you trust every workload?
Blackwell? │
│ ┌────┴────┐
┌──┴──┐ YES NO
YES NO │ │
│ │ MPS Dev / bursts / consumer GPU
MIG │ (spatial, / no need to isolate?
│ │ per-process │
│ no real QoS) YES
│ isolation: │
│ rethink TIME-SLICING
│ (move to (temporal, cheap,
│ CPU, another works on a 5090)
│ GPU, or accept
│ the risk with
│ time-slicing)

And each branch in one sentence:

MIG when isolation is a requirement (compliance, multi-tenant, hard SLA) and you have datacenter hardware that supports it. Concrete walls cost money, but if you need them there is no substitute.
MPS when you have many small concurrent workloads that leave the GPU half empty and you trust all of them (same team, not third-party tenants). You raise throughput with reasonable QoS, accepting imperfect fault isolation.
Time-slicing when it is dev, bursts, low utilisation, a consumer GPU, or you simply do not need to isolate anything. Cheap and universal, but manage the VRAM budget by hand.

One nuance the recent documentation records: they can be combined. You can do time-slicing on top of a MIG instance (hardware isolation at the instance boundary, software turns inside), or use MPS inside a MIG instance. The layers are not mutually exclusive; the tree picks the primary strategy.

Applied to the generic 4×H100 cluster

Let us get down to numbers with a generic on-premise cluster of 4×H100 SXM 80 GB with NVLink. It is common to have a heterogeneous menu of workloads: a large chat model, light services (embeddings, reranker, guardrails) and a dev/experimentation queue. Each type calls for a different mechanism. A reasoned split:

GPU 0 and GPU 1, large model in tensor-parallel (not shared). A 70B model in FP8 takes ~70 GB of weights; served comfortably with a generous KV-cache it needs more than one H100. We spread it in tensor-parallel over two whole H100s joined by NVLink (intra-node bandwidth is what makes TP viable; the detail is in TP against replicas). Here we do not share: these two GPUs belong to the large model, full stop. Total isolation by dedication.

$$V_{\text{available}} = 2 \times 80 = 160\ \text{GB};\quad V_{\text{weights}} \approx 70\ \text{GB};\quad V_{\text{KV}} \approx 80\ \text{GB of KV-cache}$$

There is memory to spare for a long request queue and high concurrency.

GPU 2, split with MIG into small instances for light services. Embeddings (bge-m3), reranker (bge-reranker-v2-m3) and a couple of guardrail models (1B–3B) are different workloads, potentially from different teams, and you want a failure or a spike in one not to touch the others. Light multi-tenancy with isolation → MIG. A reasonable partitioning of the H100:

$$\underbrace{3 \times \texttt{1g.10gb}}_{\text{30 GB, 3/7 SMs}} \;+\; \underbrace{1 \times \texttt{4g.40gb}}_{\text{40 GB, 4/7 SMs}}$$

The three 1g.10gb instances (10 GB, 1/7 of the SMs each) host embeddings, reranker and a 1B INT4 guardrail, each isolated, with no noisy neighbour. The 4g.40gb (40 GB, 4/7 of the SMs) hosts an intermediate 7B–13B model with a decent KV-cache for a support service. Each service has its larder and its wall; if the reranker blows up, the chat never finds out.

GPU 3, time-slicing for dev and bursts. The developers touch the GPU sporadically: experiments, short fine-tunes, model trials. They do not need isolation (it is the same team) and they rarely overlap while active. We advertise it as 4 replicas via the device-plugin. Four dev pods fit in turns. VRAM budget with the formula above: if each dev brings up a vLLM at --gpu-memory-utilization 0.20, they add up to $4 \times 16 = 64$ GB < 80 GB, with no OOM. If somebody needs more, lower the replica count or coordinate with the team. The cost of flexibility is manual discipline.

Summary of the split:

Resource	Mechanism	Workload	Isolation
GPU 0 + GPU 1	Dedication (TP)	70B chat in tensor-parallel	total (dedicated)
GPU 2	MIG (3×1g.10gb + 1×4g.40gb)	embeddings, reranker, guardrails, 7B–13B	real hardware
GPU 3	Time-slicing (4 replicas)	dev, bursts, experiments	none (trust)

The logic is always the same: spend isolation (MIG) where you need it, spend cheap concurrency (time-slicing) where you do not, and reserve whole GPUs for what genuinely fills them. An H100 serving embeddings on a 7g.80gb would be as absurd as an RTX 5090 trying MIG: the tool does not match the workload.

What we have not covered

What happens when they do not even fit at the same time: if you have more models than VRAM and they have to take turns in memory (loading/unloading weights, not just taking turns at compute), you enter swap and sleep territory, the sibling piece Serving several models on one GPU.
NUMA-aware scheduling: on multi-socket nodes, which GPU goes with which CPU/memory matters for latency; see Kubelet resource managers on RKE2.
Autoscaling the replicas: how many instances to bring up according to real load, with KEDA and queue metrics; see Autoscaling LLMs on Kubernetes with KEDA.
A jitter benchmark under contention: how much TTFT really wobbles in time-slicing with 4 active replicas compared with MIG, material that deserves its own measurement, not an estimate.

References

NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview. https://docs.nvidia.com/deploy/mps/index.html
NVIDIA — MPS: Tools and Interface Reference (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, CUDA_MPS_PINNED_DEVICE_MEM_LIMIT). https://docs.nvidia.com/deploy/mps/appendix-tools-and-interface-reference.html
NVIDIA — Multi-Instance GPU (MIG) User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (H100 80GB catalogue). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — k8s-device-plugin (time-slicing replicas). https://github.com/NVIDIA/k8s-device-plugin
vLLM — Engine Arguments (--gpu-memory-utilization). https://docs.vllm.ai/en/latest/serving/engine_args.html

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

Tue, 09 Jun 2026 17:00:00 +0200

Operational companion to The contractor with the master key. That post explains the why and the where: the threat model, the five isolation families, which domain uses each. This one is the how, with commands. If you have not read it, read it first: here I take for granted what the blast radius is, why bwrap runs without root and what Tetragon watches. The procedure comes in two independent tracks, client and cluster, because, as the sibling post argues, the control extrapolates but the primitive gets rewritten.

TL;DR

Two reproducible procedures. Client (workstation): install ai-jail (it wraps bubblewrap), generate the per-project .ai-jail, audit with --dry-run, fix the allowlists with --bootstrap, use --lockdown for anything you do not trust, and leave the agent without git push permission. Cluster (RKE2 with Cilium + Tetragon): put the pod baseline in place (unprivileged securityContext, seccomp: RuntimeDefault, default-deny NetworkPolicy), put the untrusted agent’s pod in a microVM with runtimeClassName: kata, and deploy the Tetragon TracingPolicy files in two phases: observe with action: Post to raise the baseline, then promote to action: Sigkill on tcp_connect (egress) and security_file_open (secret paths). The golden rule of the Tetragon phase: adopt first, block later; never put a Sigkill into production without having seen the events in observation mode first.

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

ai-jail wraps the sandbox; on Linux it needs bubblewrap separately, on macOS it needs no extra dependency.

# ai-jail (macOS and Linux)
brew tap akitaonrails/tap && brew install ai-jail
# or, with cargo:
cargo install ai-jail
# or, with mise:
mise use -g ubi:akitaonrails/ai-jail

# bubblewrap on Linux (pick your distro)
sudo pacman -S bubblewrap # Arch
sudo apt install bubblewrap # Debian / Ubuntu
sudo dnf install bubblewrap # Fedora

Check the binary is there and that bwrap runs without root (it must not ask for sudo):

ai-jail --version
bwrap --ro-bind / / --unshare-all echo "bwrap ok without root"

If bwrap fails asking for privileges, your kernel has unprivileged user namespaces disabled; enable them (sysctl kernel.unprivileged_userns_clone=1 on older Debian/Ubuntu) before continuing.

A1 — The per-project .ai-jail file

On the first launch inside the project, ai-jail creates a .ai-jail (TOML) that is committable to the repo: any colleague who clones inherits the same policy.

cd ~/Projects/mi-app
ai-jail claude # creates .ai-jail and launches Claude Code inside the sandbox

The generated file:

# .ai-jail — sandbox configuration (commit it to the repo)
command = ["claude"]
rw_maps = [] # extra writable directories
ro_maps = [] # extra read-only directories

Before trusting the sandbox, audit it. --dry-run --verbose prints every mount point, every isolation flag and the full bwrap command, without executing anything:

ai-jail --dry-run --verbose claude

Read the output and confirm three things: that $HOME is mounted as a tmpfs (not the real one), that ~/.ssh, ~/.aws and ~/.gnupg do not appear among the mounts, and that the only writable directory is the project one. If you need an extra directory:

ai-jail --rw-map ~/Projects/shared-lib claude # extra, writable
ai-jail --map /opt/datasets claude # extra, read-only

Other agents, same binary:

ai-jail codex
ai-jail opencode
ai-jail bash # bare shell to debug the sandbox
ai-jail -- python script.py # any command

A2 — Permission allowlists with –bootstrap

--bootstrap generates each agent’s permission configuration, with sensible allow/deny/ask, and takes a backup before overwriting:

ai-jail --bootstrap

What it produces, in summary:

Agent	File	Base policy
Claude Code	`~/.claude/settings.json`	allow: `git status/diff/log`, `ls`, `grep`, `cargo`, `npm`, `python`, `docker compose` · ask: `git push`, `rm`, `docker run` · deny: `rm -rf`, `sudo`, `chmod 777`, `git push --force`
Codex	`~/.codex/config.toml`	`approval_policy = "on-request"`
OpenCode	`~/.config/opencode/opencode.json`	permissions for `bash`, `edit`, `write`

The operational key: git push is under ask, not allow, and git push --force under deny. The agent can commit, branch and rebase locally as much as it wants; none of that touches the remote. (If you use the Claude Code /sandbox, also set "allowUnsandboxedCommands": false to close the dangerouslyDisableSandbox escape hatch, which is opt-out out of the box.)

A3 — Lockdown for anything you do not trust

To audit third-party code or run an agent over a project you do not know, --lockdown goes further: project mounted read-only, GPU/Docker/display disabled, --rw-map/--map ignored, $HOME a pure tmpfs with no host dotfiles, network cut with --unshare-net and the environment wiped with --clearenv.

ai-jail --lockdown bash

It is the most restrictive sandbox possible short of a VM. Use it as your mental default for anything that is not your own code on your own machine.

A4 — The safety net: git with no push

It is not a flag, it is a property of the environment that changes the risk arithmetic. If the project is in git with a remote, and the agent does not have push permission, the worst case, that it corrupts every file in the project, is reverted with:

git checkout . # back to the last commit
# and if it touched .git (unlikely): delete the dir and re-clone

The remote was never touched. Sandbox for the filesystem + git for the code + manual push is already a reasonable level for daily use: ai-jail protects your data and the system, git protects the code, and the decision to publish is still yours.

Track B — Cluster (RKE2 with Cilium + Tetragon)

The untrusted agent, or the inference that executes generated code, runs as a pod. The same principle as on the client, different primitives. We assume a generic RKE2 cluster with Cilium as CNI and Tetragon already deployed (the eBPF agent DaemonSet on every node).

B0 — The pod baseline

Before any eBPF, the standard kit. Unprivileged securityContext, read-only root, default seccomp:

apiVersion: v1
kind: Pod
metadata:
 name: ai-agent
 namespace: agentes
 labels:
 app: ai-agent
spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 10001
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: agent
 image: registry.interno/ai-agent:pinned
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop: ["ALL"]
 volumeMounts:
 - { name: work, mountPath: /work }  # the only writable one
 volumes:
 - name: work
 emptyDir: {}

And the default egress cut, the cluster twin of --unshare-net. A default-deny egress NetworkPolicy in the namespace, opening only DNS and the bare essentials:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-egress
 namespace: agentes
spec:
 podSelector: {}
 policyTypes: ["Egress"]
 egress:
 - to:
 - namespaceSelector:
 matchLabels: { kubernetes.io/metadata.name: kube-system }
 ports:
 - { protocol: UDP, port: 53 }
 - { protocol: TCP, port: 53 }

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

For genuinely untrusted code, take it out of the shared kernel. With Kata deployed there is a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
 name: kata
handler: kata

And the pod asks for it with one line, runtimeClassName: kata, running in its own microVM with a dedicated kernel instead of sharing the node’s:

spec:
 runtimeClassName: kata  # ← the pod runs in a microVM, not on the node's kernel
 # ...the rest the same as B0

It is the cluster twin of isolation by construction: a kernel exploit inside the pod does not reach the node.

B2 — Tetragon, observation phase (Post)

Now the layer that separates a platform with runtime visibility from one without. Observe first, never kill from the outset. A TracingPolicyNamespaced, scoped to the namespace and to the agent’s label, that reports (does not kill) three things: process executions, network connections and opens of sensitive paths. action: Post only emits the event.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-observa
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- outbound connections ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchActions:
 - action: Post
 # --- opens of sensitive files ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets"
 - "/work/.git/config"
 matchActions:
 - action: Post

(Process executions need no kprobe: Tetragon emits process_exec/process_exit natively.) Deploy it and watch the events live from the node’s Tetragon pod:

kubectl apply -f agente-observa.yaml
# readable events, filtered by namespace:
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
 tetra getevents -o compact --namespace agentes

Leave this running for a typical working day of the agent. Note which destinations it really connects to (your internal registry, your HF mirror, your vLLM endpoint) and which paths it opens. That is your baseline: the list of what is legitimate. Without this step, a Sigkill kills good work and generates an availability incident, precisely what ENS asks you to avoid.

B3 — Tetragon, enforcement phase (Sigkill)

With the baseline in hand, promote to blocking. Two rules. The first: kill any connection whose destination is not on the allowlist — NotDAddr inverts the match: it fires for everything that is not those networks. The second: kill any attempt to open a secrets path.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-enforce
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- egress: kill everything that is NOT on the allowlist ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr"
 values:
 - "127.0.0.1"
 - "10.0.0.0/8" # internal cluster network
 - "172.16.10.20" # internal registry (example)
 matchActions:
 - action: Sigkill
 # --- reading secrets: kill the process ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets/kubernetes.io/serviceaccount/token"
 - "/work/.ssh"
 matchActions:
 - action: Sigkill

kubectl apply -f agente-enforce.yaml

Now the agent can do whatever it likes inside the pod, but the instant it tries to connect to a disallowed destination or read the service account token, Tetragon kills it in the kernel, before the packet leaves or the read returns bytes. It is the cluster twin of the curl blocklist and the unmounted ~/.ssh, but applied at runtime and over any binary, not just the ones you know about.

Operational warning. Enforcement with Sigkill requires a recent kernel with eBPF support for the action (5.10+ is safe). Deploy agente-enforce in a test namespace first, and keep agente-observa active in parallel: if the block fires, the Post event tells you exactly what triggered it. Adopt first, block later.

The client ↔ cluster equivalence table

The same vector, the two primitives. This is “extrapolating the technology” made explicit:

Threat vector	Client (workstation)	Cluster (RKE2)
`$HOME` / writable root	`$HOME` as ephemeral tmpfs (`bwrap`)	`readOnlyRootFilesystem: true` + `emptyDir`
Arbitrary egress	`curl`/`wget` blocklist · `--unshare-net`	default-deny NetworkPolicy + Tetragon `NotDAddr`→`Sigkill`
Reading secrets	`~/.ssh`/`~/.aws`/`~/.gnupg` not mounted	secrets outside the pod + Tetragon `security_file_open`→`Sigkill`
Kernel escape	Landlock (2nd VFS barrier)	`runtimeClassName: kata` (microVM, own kernel)
No escape hatch	process inside `bwrap`, no way out	no `privileged`, `drop ALL`, `allowPrivilegeEscalation:false`
Damage to the code	git remote with no `push` → `git checkout .`	GitOps + PR review, the agent does not apply to `main`
Visibility	`--dry-run --verbose` (static, pre-run)	Tetragon `tetra getevents` (dynamic, at runtime)

Checklist of gotchas

Do not put in a Sigkill without passing through Post. The observation baseline is not optional: it is what separates “blocking a C2” from “killing your own fine-tuning job”.
The .ai-jail gets committed; secrets do not. The TOML is policy, not credentials. Check you are not putting paths holding sensitive data into rw_maps.
readOnlyRootFilesystem breaks apps that write to /tmp. Mount an emptyDir at /tmp as well as the work one.
A NetworkPolicy with no DNS rule leaves the pod blind. Open port 53 to kube-system or nothing resolves.
Kata is not free. It adds startup latency and not every workload with special devices (GPU passthrough) fits; reserve it for the untrusted, not for everything.
The Claude Code /sandbox does not cover MCP or hooks unless you enable sandbox-runtime. If your agent uses MCP servers, assume they run with full permissions until you do.
NotDAddr with literal IPs ages badly. Document the allowlist and review it when the registry or the inference endpoint changes; consider stable internal CIDRs instead of loose IPs.

References

ai-jail (Fabio Akita), GPL-3.0: https://github.com/akitaonrails/ai-jail
bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io
Tetragon — TracingPolicy: https://tetragon.io/docs/concepts/tracing-policy/
Tetragon — enforcement (Sigkill/Override): https://tetragon.io/docs/concepts/enforcement/
Kata Containers — Kubernetes RuntimeClass: https://katacontainers.io
Kubernetes — Pod Security & seccomp: https://kubernetes.io/docs/tutorials/security/seccomp/
Kubernetes — Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Cilium: https://cilium.io

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

Sat, 06 Jun 2026 12:00:00 +0200

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

Disaggregated serving. The KV-cache that travels between the prefill pool and the decode pool moves over RDMA. These are large transfers, sensitive to latency and bandwidth, which in multi-node setups go out through the NIC.
Multi-node NCCL collectives. When tensor/pipeline parallelism crosses the node boundary, the all-reduce/all-gather operations no longer go over NVLink but over GPUDirect RDMA on the NIC.

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Three kernel facts the analogy compresses:

The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 piece	What it does	What Cilium eBPF puts in
`kube-proxy` (iptables/IPVS)	Service balancing	LB in eBPF; with `kubeProxyReplacement=true`, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)	VXLAN overlay + NetworkPolicy	native datapath (`routingMode=native`), L3/L4 and L7 NetworkPolicy in eBPF
veth per pod	namespace interface pair	netkit (kernel ≥6.8): namespace overhead ~0
host iptables traversal	netfilter hooks	eBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus = 2-31,34-63 # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33 # kubelet housekeeping (post 3)
IRQ affinity = 0-1,32-33 # NIC IRQs → housekeeping ONLY (this post)
 # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
 --namespace kube-system \
 --set routingMode=native \
 --set bpf.datapathMode=netkit \  # namespace overhead ~0 (kernel >=6.8)
 --set bpf.masquerade=true \
 --set kubeProxyReplacement=true \  # replaces RKE2's kube-proxy
 --set enableIPv4BIGTCP=true \  # 192k super-packets (mlx5/ice NIC)
 --set enableIPv6BIGTCP=true \
 --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
 --set bpf.mapDynamicSizeRatio=0.08 \
 --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
 name: gpu-nic-numa-aligned
spec:
 spec:
 devices:
 requests:
 - name: rdma-nic
 deviceClassName: dra.net  # NICs published by DRANET
 constraints:
 - requests: ["rdma-nic"]
 matchAttribute: "dra.net/pcieRoot" # same PCIe root as the GPU
 # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

References

Cilium, Tuning Guide (netkit, host-routing, BIG TCP, XDP, pin the IRQs and kill irqbalance): https://docs.cilium.io/en/stable/operations/performance/tuning/.
Cilium 1.19 (February 2026), Cilium at Ten Years — hardening of encryption, policies and observability: https://www.infoq.com/news/2026/02/cilium-119/.
Isovalent, Cilium 1.18 (IPv6, encrypted overlay, ingress bandwidth, policy perf): https://isovalent.com/blog/post/cilium-1-18/.
RKE2, Network Options (Canal by default; Cilium with kube-proxy replacement): https://docs.rke2.io/networking/basic_network_options.
Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/.
Kubernetes blog, v1.36: More Drivers, New Features, and the Next Era of DRA (May 2026): https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/.
DRANET (kubernetes-sigs), the network DRA driver and the paper The Kubernetes Network Driver Model (+59.6% all_gather / +58.1% all_reduce): https://github.com/kubernetes-sigs/dranet.
AKS Engineering, Optimizing RDMA performance for AI workloads on AKS with DRANET (April 2026): https://blog.aks.azure.com/2026/04/01/dranet-rdma-optimization-for-ai-on-aks.
Linux network tuning — IRQ affinity, RSS/RPS/RFS and NUMA softirq: https://andreaskaris.github.io/blog/networking/rss-irq-affinity-and-rps/.

The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

Sat, 06 Jun 2026 08:00:00 +0200

The close of the “under the engine” series. We saw the cable between GPUs and the host: NUMA, hugepages and CPU isolation done by hand. Here is the piece that makes it declarative and at scale: how the RKE2 kubelet pins each vLLM pod to the right NUMA node without a single script.

TL;DR

Pinning NUMA with numactl/isolcpus/taskset, the subject of the previous post, does not scale to a cluster where pods come and go and there are dozens of nodes. The kubelet automates it with three components that work as Hint Providers for a central coordinator, the Topology Manager: the CPU Manager (assigns exclusive CPUs to containers of Guaranteed pods with whole CPUs), the Memory Manager (NUMA-local memory and hugepages) and the Device Manager/GPU plugin (which knows which GPU is on which NUMA node). With the single-numa-node policy, the Topology Manager only admits the pod if its CPUs, its memory and its GPU fit in the same NUMA domain; if they do not fit, it rejects the pod, strict admission, like the maître d’ who will not seat a party of eight unless there is a table for eight. On RKE2 all of this is configured with kubelet-arg in /etc/rancher/rke2/config.yaml. This post explains the mechanism, gives the 10 knobs and takes apart the gotchas that break pinning silently: the cpu_manager_state file you have to delete when changing policy, the QoS that has to be exactly Guaranteed, and the reserved-cpus that must match the host’s isolcpus. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the orchestration that materialises the host

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

A restaurant has tables of different sizes and, house rule, tables are never pushed together. A party of eight arrives. The maître d’ looks for a single table where all eight fit. If there is one, he seats them; if only tables for four are left, he does not accept them. He would rather turn the booking away than seat the group split across two separate tables, because he knows a split dinner goes badly.

That maître d’ is the Topology Manager under the single-numa-node policy. The “party” is an inference pod asking for CPUs, memory and a GPU. The “table” is a NUMA node. The maître d’ asks three assistants (are there free CPUs on any node? the CPU Manager; is there free memory? the Memory Manager; is there a free GPU? the Device Manager) and only admits the pod if all three resources fit on the same node. If not, he rejects it (the pod ends up Failed with TopologyAffinityError), and the scheduler will try another node.

The difference from the previous post: there you were the maître d’, seating each process by hand with numactl. Here the maître d’ is the kubelet, and it does it for every pod, on every node, automatically, rejecting whatever does not fit. That is what turns artisanal pinning into a declarative property of the cluster.

The mechanism: Hint Providers and the coordinator

The Topology Manager does not assign resources; it coordinates the ones that do. The flow, when a Guaranteed pod arrives at a node:

The three managers are Hint Providers: each one tells the Topology Manager on which NUMA node(s) it could satisfy its part. The Topology Manager computes the intersection and, depending on the policy, decides:

none (default): no coordination; each manager does its own thing without aligning. No NUMA guarantee.
best-effort: it tries to align on one node; if it cannot, it admits anyway (on whichever node). Better than nothing, no guarantee.
restricted: if it fails to align, it rejects the pod. Strict, but it allows multi-node affinity if the intersection provides it.
single-numa-node: it requires everything to fit on a single NUMA node, or it rejects. The strictest one, and the one that genuinely guarantees the locality of the previous post.

And two preconditions without which none of this kicks in:

The pod has to be QoS Guaranteed: requests == limits on CPU and memory, and whole CPUs (not 500m). Only then does the CPU Manager assign exclusive CPUs.
The CPU Manager has to be on the static policy (not none).

Without those two, the Topology Manager has nothing to align and pinning does not happen, even with the policy in place. That is gotcha no. 1.

How it is configured on RKE2

RKE2 passes arguments to the kubelet with the kubelet-arg key in /etc/rancher/rke2/config.yaml. The reference configuration for GPU inference nodes:

# /etc/rancher/rke2/config.yaml (on every agent node with GPUs)
kubelet-arg:
 - "cpu-manager-policy=static"
 - "topology-manager-policy=single-numa-node"
 - "topology-manager-scope=pod"
 - "memory-manager-policy=Static"
 - "reserved-cpus=0-1,64-65" # housekeeping; must match the host
 - "system-reserved=memory=8Gi"
 - "kube-reserved=memory=4Gi"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi" # required by Memory Manager Static
node-label:
 - "fibercli.local/numa-pinned=true"

After deploying it: systemctl restart rke2-agent. Critical gotcha: if the node has already run with cpu-manager-policy=none, there is a state file /var/lib/kubelet/cpu_manager_state that pins the old policy; changing the arg without deleting that file makes the kubelet fail to start or ignore the new policy. You have to stop the agent, rm /var/lib/kubelet/cpu_manager_state, and start again. (The same applies to memory_manager_state.)

And the vLLM pod, to be eligible, Guaranteed with whole CPUs:

resources:
 requests:
 cpu: "16" # whole, not a weird fractional "16000m"
 memory: "200Gi"
 nvidia.com/gpu: "2" # TP=2 → 2 GPUs on the same NUMA node
 hugepages-1Gi: "16Gi"
 limits:
 cpu: "16" # == requests → Guaranteed QoS
 memory: "200Gi"
 nvidia.com/gpu: "2"
 hugepages-1Gi: "16Gi"

With this, on a node with the config above, the kubelet assigns 16 exclusive CPUs from the NUMA node where the 2 requested GPUs live, its local memory and the hugepages, or it rejects the pod if they do not fit together. The artisanal pinning of the previous post, now declarative.

The 10 knobs to reach for

Ordered by dependency (the first ones are a precondition for the later ones). The canonical reference is the Kubernetes Topology Manager documentation and the RKE2 configuration.

Knob 1 — `cpu-manager-policy=static`: the foundation

Without this there are no exclusive CPUs and nothing else kicks in.

kubelet-arg: [ "cpu-manager-policy=static" ]

Gotcha: changing it requires deleting /var/lib/kubelet/cpu_manager_state and restarting the kubelet, or startup fails. It is cause no. 1 of “I set the policy and it does not pin”.

Knob 2 — `Guaranteed` QoS + whole CPUs: the pod’s precondition

This is not node config, it is pod config, but without it knob 1 does nothing for that pod. requests == limits on CPU and memory, and whole CPUs. A cpu: 500m or a requests != limits downgrades the pod to Burstable and it loses the pinning. Plenty of people set the node policy and forget the pod’s QoS.

Knob 3 — `topology-manager-policy=single-numa-node`: strict admission

kubelet-arg: [ "topology-manager-policy=single-numa-node" ]

The strict maître d’. For GPU inference it is the right policy: it guarantees that CPU+memory+GPU share a node. best-effort guarantees nothing (it admits misaligned); restricted allows multi-node affinity. Start with single-numa-node and drop to restricted only if you have admission problems.

Knob 4 — `topology-manager-scope=pod`: group the whole pod

kubelet-arg: [ "topology-manager-scope=pod" ]

With container scope (the default), each container is aligned separately; with pod scope, the whole pod goes to the same node. For a vLLM pod with sidecars (metrics, proxy), pod scope stops the sidecar from dragging the main container onto another node. Recommended for inference.

Knob 5 — `memory-manager-policy=Static` + `reserved-memory`: NUMA-local memory

kubelet-arg:
 - "memory-manager-policy=Static"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi"

Memory Manager Static forces cpuset.mems so that the pod’s memory comes out of the right node (and so do the hugepages). It requires declaring reserved-memory per node, or the kubelet does not start. It is the declarative equivalent of the previous post’s --membind.

Knob 6 — `reserved-cpus`: the housekeeping cores (must match `isolcpus`)

kubelet-arg: [ "reserved-cpus=0-1,64-65" ]

It reserves cores for the system and the daemons; the rest are left for exclusive pods. Key point of the series: these reserved-cpus must be the same cores you left out of isolcpus on the host (previous post). If the host isolates 2-31 but RKE2 reserves 0-3, there is a mismatch: isolated cores that the kubelet assigns to pods without them actually being quiet. Coordinate the two layers.

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

The Device Manager can only give a correct NUMA hint if the GPU plugin exposes which node each GPU is on. The NVIDIA device plugin / GPU Operator does this, but you have to verify that the topology information arrives (in some versions it requires flags). Without a GPU hint, the Topology Manager aligns CPU and memory but not the GPU, and GPU locality is precisely the one that matters most.

Knob 8 — hugepages as a pod resource

resources:
 limits:
 hugepages-1Gi: "16Gi" # the node must have them pre-reserved (post 2, knob 4)

The hugepages you reserved at host boot (previous post) are requested as a resource. The Memory Manager assigns them NUMA-local. If you request them without having reserved them on the node, the pod does not get scheduled.

Knob 9 — `system-reserved` / `kube-reserved`: do not oversubscribe

kubelet-arg:
 - "system-reserved=cpu=500m,memory=8Gi"
 - "kube-reserved=cpu=500m,memory=4Gi"

It reserves resources for the system and the K8s components so the node does not run out of air under load. Badly calibrated, either the node chokes (too little reserved) or you waste capacity (too much). It must be consistent with reserved-cpus.

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

# GPU node: taint to repel anything that does not need a GPU
node-taint: [ "nvidia.com/gpu=present:NoSchedule" ]
node-label: [ "fibercli.local/numa-pinned=true" ]

Keep the NUMA-pinned nodes for inference and push out whatever does not need it (databases, the Langfuse backend, runners). A ClickHouse stealing memory bandwidth from a carefully pinned vLLM pod throws away all the work of the previous nine knobs. Workload isolation is the closing move.

Summary table

#	Knob	Where	Function
1	`cpu-manager-policy=static`	kubelet-arg	exclusive CPUs (foundation)
2	`Guaranteed` QoS + whole CPUs	pod spec	pinning precondition
3	`topology-manager-policy=single-numa-node`	kubelet-arg	strict NUMA admission
4	`topology-manager-scope=pod`	kubelet-arg	group the whole pod
5	`memory-manager-policy=Static`	kubelet-arg	NUMA-local memory/hugepages
6	`reserved-cpus`	kubelet-arg	housekeeping (match isolcpus)
7	GPU plugin with topology	GPU Operator	NUMA hint for the GPU
8	`hugepages-1Gi`	pod spec	hugepages as a resource
9	`system/kube-reserved`	kubelet-arg	do not oversubscribe
10	taints + labels	node config	isolate GPU workloads

Verifying that the pinning really happened

Do not trust that the config “is in place”. Check:

# Is the active policy the one you set?
cat /var/lib/kubelet/cpu_manager_state | jq .policyName # "static"

# Which exclusive CPUs does the container have?
kubectl exec <pod> -- cat /sys/fs/cgroup/cpuset.cpus.effective

# Inside the pod: is the assigned GPU local to those cores?
kubectl exec <pod> -- nvidia-smi topo -m

# Were there topology rejections?
kubectl describe pod <pod> | grep -i TopologyAffinityError

A pod in Failed with TopologyAffinityError is not a bug: it is the maître d’ doing his job. That node did not have a table where CPU+memory+GPU fitted together. The answer is to review the sizing of the pod or the node, not to relax the policy casually.

How it connects with the rest of the stack

With the host (previous post). This post is the declarative automation of that one. cpu-manager-policy=static materialises the taskset; memory-manager-policy=Static materialises the --membind; reserved-cpus must match isolcpus. The two layers are one single decision seen from two places: the host executes it, the kubelet declares it. Uncoordinating them (isolcpus 2-31 vs reserved-cpus 0-3) breaks both.

With the interconnect (post 1). The Topology Manager pins the right GPU to the pod, but if you ask for 2 GPUs for TP=2, you will want those two to share NVLink. The NUMA policy guarantees they are on the same socket; that they are NVLink-connected is guaranteed by the baseboard hardware (post 1, knob 1). The two things together are what makes TP=2 perform.

With autoscaling. When KEDA scales vLLM pods, each new replica goes through Topology Manager admission. If the node has no free “table”, the pod stays pending. Pod autoscaling and node autoscaling (cluster-autoscaler) have to account for NUMA granularity, not just aggregate CPU/memory.

With capacity planning. Sizing changes: it is not “128 vCPU per machine”, it is “128 minus the reserved-cpus, in blocks that fit per NUMA node”. A machine with 2 sockets × 64 cores does not serve a pod asking for 80 cores under single-numa-node: they do not fit at one table. Planning has to reason per NUMA node, not per machine.

With service coexistence. The knob 10 taint is what keeps Langfuse, databases and runners off the inference nodes. Without that boundary, all the fine-grained pinning gets eaten by a noisy neighbour. Observability goes on its own nodes; inference, pinned, on its own.

Traps and things that are not what they seem

Policy in place, QoS forgotten. The most common mistake: cpu-manager-policy=static on the node but the pod is Burstable (requests != limits or fractional CPU). Pinning does not happen and nobody warns you. Guaranteed QoS with whole CPUs is a necessary condition.

Fossilised cpu_manager_state. Changing policy without deleting /var/lib/kubelet/cpu_manager_state (and memory_manager_state) makes the kubelet fail or ignore the change. Stop agent → delete file → start.

reserved-cpus ≠ isolcpus. If the host isolates some cores and RKE2 reserves others, the managers assign cores to pods that are not really quiet, or they leave isolated cores idle. The two lists have to be consistent. It is the coordination failure between the previous post and this one.

GPU plugin without NUMA topology. If the device plugin does not expose each GPU’s NUMA node, the Topology Manager aligns CPU and memory but leaves the GPU to chance, and GPU locality is the one that weighs most. Verify that the GPU Operator publishes the topology.

single-numa-node that rejects too much. If the pods ask for more resources than fit on one node (for example, more cores than a socket has), rejection is constant. The answer is not to drop to best-effort (which silences the problem by serving misaligned), but to size the pod so it fits at one table, or to accept restricted with full knowledge of what you are doing.

Believing best-effort “is nearly the same”. best-effort admits the pod even if it fails to align: it gives you a false sense of NUMA-awareness while you serve from the wrong socket. For inference with a tail SLO, single-numa-node or restricted; best-effort only if the alternative is scheduling nothing at all.

Conclusion

The previous post pinned by hand; this one does it at scale and with a guarantee that artisanal numactl did not give: strict admission. The kubelet, via CPU Manager, Memory Manager and Topology Manager, acts as a maître d’ who only seats the pod if its CPUs, its memory and its GPU fit at the same NUMA table, and who rejects what does not fit instead of serving a split dinner. Of the ten knobs, the first two, cpu-manager-policy=static and Guaranteed QoS with whole CPUs, are the precondition without which the other eight do nothing, and they are exactly the ones most often forgotten; the rest tune the policy, the memory, the hugepages and the coexistence. The thread that closes the series: the inference performance that looked like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a cable problem (NVLink not used), a host problem (remote NUMA, jitter) or an orchestration problem (pinning that never happened because the QoS was wrong). Going down a level is not infrastructure snobbery: it is where the root causes live that no application-layer dashboard is ever going to point out for you.

References

Kubernetes, Control Topology Management Policies on a node: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/.
Kubernetes, Control CPU Management Policies on the Node: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/.
Kubernetes, Control Memory Management Policies on a Node: https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/.
RKE2, Configuration Options (kubelet-arg in config.yaml): https://docs.rke2.io/install/configuration.
RKE2, Advanced Options and Configuration: https://docs.rke2.io/advanced.
rancher/rke2, discussion #3034 CPU Management Policies for RKE2 (the cpu_manager_state gotcha): https://github.com/rancher/rke2/discussions/3034.

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

Mon, 01 Jun 2026 16:00:00 +0200

This post complements those on GPU observability for LLM inference (where the metrics that feed the HPA come from), Capacity planning (what ceiling and what head-room the autoscaler assumes) and Continuous batching (which explains why num_requests_waiting is the primary metric).

TL;DR

Classic Kubernetes autoscaling, HPA on cpu or memory, is no use for LLM inference. The reason: the vLLM pod consumes little CPU (the GPU does the work) and the process RSS is flat; both metrics can sit at 30 % while the GPU is saturated and the request queue grows unchecked. The four viable signals that do respond to the real load are: vllm:num_requests_waiting (the queue, the primary metric), vllm:gpu_cache_usage_perc (pressure on the KV cache pool), TTFT P95 via the vllm:time_to_first_token_seconds_bucket histogram (the SLO guarantee) and the batch fill ratio num_requests_running / max_num_seqs (utilisation of the concurrency ceiling). For an HPA to consume Prometheus metrics you need an adapter; as of May 2026 there are two mature options: prometheus-adapter (cluster-wide sigma, static configuration, external.metrics.k8s.io output) and KEDA (ScaledObject with a Prometheus trigger, configurable polling, optional scale to zero, cron integration). KEDA is the dominant option for LLM on a generic cluster because it solves the “warm pool + cron + engine metric” pattern in a single CRD. The dominant operational challenge is not the scaling logic but the cold start: a vLLM pod with Llama 70B BF16 (140 GB) takes between 90 seconds (model pre-cached on a local PV) and 6 minutes (image pull + model download from object store) to serve the first token. The five levers that cut it down are an image pre-pulled via DaemonSet, the model cached on a regional PV or tmpfs, a warm pool with minReplicaCount > 0, predictive scaling via KEDA cron when the traffic pattern is predictable (offices 9–18 h), and parallel model download. The three specific LLM scale-down pitfalls: cutting streaming SSE connections halfway through a response (graceful drain with terminationGracePeriodSeconds ≥ 60 s), scale-out/in oscillation from a badly calibrated stabilization window, and forgetting that the HPA only scales pods; GPU nodes are scaled with cluster-autoscaler over labelled nodepools. This post includes the minimal YAML manifests.

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

An artisan bakery has three wood-fired ovens. Each oven takes 25 minutes to reach temperature from cold. Once hot, it bakes bread continuously with a run of 18 minutes per batch. The manager wants to maximise bread sold per day without burning wood for nothing, and she knows three things: that there is a demand peak at 7:30 every morning, that Mondays sell almost nothing, and that when the bread on the counter runs out the customers go to the supermarket next door.

The cheap strategy, lighting ovens when there is a queue in the shop, does not work. By the time the queue grows and the manager lights the second oven, that oven will not be ready until 25 minutes later; the customers in that window are lost. The “queue at the counter” signal arrives late.

The smart strategy: light the second oven at 6:55, ahead of the predictable 7:30 peak, and leave it running until 10:00 even though the queue drops at 8:15. Keep the third oven off from Monday to Wednesday because demand does not arrive; light it proactively on Thursdays at 12:00 because historically it goes up. Keep a stock of pre-proved raw dough in the chiller so that when the oven is ready the bread goes in within 30 seconds and there is no two-hour proving wait.

Autoscaling an LLM inference cluster works the same way:

Lighting ovens from cold = reactive scale-out when the queue grows (slow, loses customers).
Proactive cron = predictive scaling when the pattern is known (working hours, expected peaks).
Pre-proved dough = warm pool of replicas with the model loaded but under 0 load.
Switching off ovens with no bread inside = scale-down respecting the active streams (you do not shut the oven with bread in it).

The key metric, “how many customers are in the queue”, is called num_requests_waiting. The metric that says “the oven is about to run out of dough for new loaves” is called gpu_cache_usage_perc. And the quality-of-service metric, “how long the first loaf takes to come out when a new customer walks in”, is called TTFT.

Why HPA on CPU is no use

The classic Kubernetes HPA looks at the pod’s resource.cpu. For a conventional HTTP service, Node.js or a REST API, CPU moves linearly with traffic and the HPA scales with reasonable accuracy. For a vLLM or SGLang pod on GPU, the pod’s CPU typically lives between 5 % and 15 % regardless of whether the GPU is at 30 % or 99 % load: the real work is done by the device, not the process. Result: the CPU-based HPA never triggers scale-out even though the GPU is bursting, and clients pile up in the queue until TTFT P95 crosses the SLO. The operator discovers the problem through the TTFT alert, not through the HPA.

memory is no use either: the vLLM process RSS is flat after startup (model plus buffers loaded in one go); it does not reflect the real pressure on the GPU. The only things that rise and fall with the useful inference load are metrics the engine publishes explicitly: request queue, KV cache pool, SLO latencies. Without an adapter that makes them visible to the HPA, autoscaling is blind.

The four viable signals

Signal 1, vllm:num_requests_waiting (queue). It is the most direct metric: how many requests are waiting to enter the batch. It reacts the instant the target concurrency saturates. It is robust against model changes (the number of requests is the same concept whether it is Llama 7B or 70B). It is the primary metric of the LLM HPA. Typical threshold: target = 5 requests waiting on average; if the queue grows above 5 sustained for 2 minutes, scale out.

Signal 2, vllm:gpu_cache_usage_perc (KV pool). It moves before the queue does: the KV pool fills up while the batch slots are still free, until the engine starts rejecting new requests for OOM prevention and the queue forms. It is therefore predictive: it triggers scale-out before the client notices degradation. Typical threshold: target = 0.85 (85 % of pool used).

Signal 3, TTFT P95. The contractual guarantee. If TTFT P95 leaves the SLO, scale out even though the queue and the KV pool look reasonable (there may be a spike of long prompts). It is reactive, it leaves the SLO before your HPA reacts, but it works as a final guardrail.

Signal 4, batch fill ratio. The num_requests_running / max_num_seqs quotient (the latter is engine config, not a metric). Useful for scale-down: if the ratio stays below 0.4 for 10 minutes, there is spare capacity and replicas can be reduced safely.

The recommended policy combines all four: the queue and the KV pool trigger scale-out (whichever arrives first), TTFT confirms it as a guardrail, and the batch fill ratio manages scale-down. Implementing that in a single HPA demands external metrics; KEDA makes this manageable.

The wiring: KEDA as the Prometheus adapter

KEDA introduces two main CRDs: TriggerAuthentication (how to authenticate against the source) and ScaledObject (which deployment to scale with which triggers). For a vLLM deployment with Prometheus as the source:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-llama70b-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama70b
 minReplicaCount: 2 # warm pool
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300 # 5 min before scale-down
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600 # large window to avoid oscillation
 policies:
 - type: Pods
 value: 1
 periodSeconds: 120
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - type: Pods
 value: 2
 periodSeconds: 60
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue_depth
 threshold: "5"
 query: |
 avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv_cache
 threshold: "0.85"
 query: |
 avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_ttft_p95
 threshold: "1.5"
 query: |
 histogram_quantile(0.95,
 sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{deployment="vllm-llama70b"}[5m])))

Three non-obvious operational details:

minReplicaCount: 2. This is the warm pool. Keeping at least two replicas guarantees availability if a node is lost and absorbs spikes without waiting for the cold start of the first scale-up. Dropping it to 0 saves GPU off-peak but introduces 90 s–6 min of latency for the first new client.

stabilizationWindowSeconds: 600 on scale-down. Ten minutes. Models are not nginx: if a replica closes prematurely and two minutes later there is another peak, the cold start of a new pod is what the client waits. Better to keep extra replicas around for twice as long as you would for a normal web service.

scaleUp: stabilizationWindowSeconds: 30. Thirty seconds. Scale-out has to be fast: the new pod’s cold start adds its own delay, and if on top of that the HPA waits several more minutes before triggering, the SLO is already broken.

The big operational problem: cold start

A vLLM pod loading Llama 70B goes through these phases before serving the first token:

Phase	Typical time	Can be accelerated with
Image pull (4–6 GB)	30–90 s	DaemonSet pre-pull
Model download (140 GB BF16)	60–300 s	Cached regional PV, S3 + multi-thread
Loading the model into HBM	30–90 s	tmpfs or local NVMe
CUDA graph capture	20–60 s	`--enforce-eager` (slower at runtime but fast startup)
PagedAttention warmup	5–15 s	—
Health check ready	10–30 s	probe tuning

Total without optimisation: 4–10 minutes. That is how long a new replica takes to absorb traffic. With all the levers combined: 45–90 seconds. The difference between those two numbers is the main platform work for LLM autoscaling.

The five levers

Lever 1, pre-pulled image. A trivial DaemonSet runs ctr image pull (or crictl pull) on the GPU nodes as soon as they join the cluster. The inference engine image stays on disk; new pods skip the 30–90 s of pull. Cost: around 6 GB of disk per node.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: vllm-image-warmer }
spec:
 selector: { matchLabels: { app: vllm-warmer } }
 template:
 metadata: { labels: { app: vllm-warmer } }
 spec:
 nodeSelector: { workload: gpu }
 initContainers:
 - name: pull
 image: vllm/vllm-openai:v0.10.0
 command: ["/bin/true"]
 containers:
 - name: pause
 image: registry.k8s.io/pause:3.10

Lever 2, model on a regional PV. The model download (140 GB BF16 or 35 GB FP8) from central object storage is the dominant component of the cold start. Caching the model on a zone or rack PV, Rook-Ceph RBD or local NVMe provisioned by the operator, cuts 60–300 s down to 5–15 s. The antipattern: downloading the model on every startup from external S3.

volumeMounts:
 - name: model-cache
 mountPath: /models
 readOnly: true
volumes:
 - name: model-cache
 persistentVolumeClaim:
 claimName: llama70b-fp8-pvc  # RWX shared, filled offline

Lever 3, warm pool. minReplicaCount > 0 keeps pre-loaded replicas idle. The cost is idle GPU; the benefit is 0 s of cold start for the first client of a peak. For production clusters with continuous traffic: a warm pool of 2–3 replicas. For nightly batch clusters with 0 traffic: warm pool 0 and accept the cold start, or KEDA with a cron that switches on 10 minutes earlier.

Lever 4, predictive scaling with cron. When the pattern is predictable (offices 9–18 h):

triggers:
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5" # 8:30 Monday–Friday
 end: "0 19 * * 1-5" # 19:00
 desiredReplicas: "6"

Combined with reactive triggers. The HPA scales according to the maximum of the signals: if the cron asks for 6 and the queue asks for 10, the result is 10.

Lever 5, parallel download and an efficient format. For PVs that are not pre-loaded, tools such as nvidia-modelmanager, s5cmd or aria2c parallelise the model download. Going from serial download (~150 MB/s) to parallel with 8 threads (~1.2 GB/s) divides the time by 8. And formats such as safetensors load into HBM faster than the original PyTorch pickle.

When to scale nodes, not just pods

The HPA scales pods. If the cluster has no free GPU nodes, the new pod stays Pending for lack of resources. To scale nodes, you need cluster-autoscaler with a specific GPU nodepool, labelled:

# nodepool config (Karpenter or cluster-autoscaler equivalent)
labels:
 workload: gpu
 gpu-model: h100-sxm-80gb
taints:
 - key: nvidia.com/gpu
 effect: NoSchedule
limits:
 min: 2 nodes
 max: 8 nodes

Without this, the HPA can ask for 10 replicas but the cluster only delivers those that fit on nodes already up. The cold start of a new node (bare metal or cloud provisioning, PXE, OS boot, NVIDIA drivers, cluster join) is much longer than the cold start of a pod: typically 5–15 minutes on preconfigured bare metal, 30–60 minutes on real provisioning. For on-premise clusters, the nodepool must be always sized to the expected maximum, and the “scaling” is only on the pod side. The concept of reactive node scale-out only applies to clouds; on-premise you have to buy for the peak.

Three pitfalls specific to LLM scale-down

Pitfall 1, cutting streaming SSE connections. When a replica enters Terminating, Kubernetes sends SIGTERM to the pod and, by default, kills it 30 seconds later. For vLLM that means cutting streaming SSE connections halfway through the response. The client gets a 502 error with the partial output lost. Solution: terminationGracePeriodSeconds: 120 plus a preStop hook that tells the engine not to accept new requests but to finish those in flight:

spec:
 terminationGracePeriodSeconds: 120
 containers:
 - name: vllm
 lifecycle:
 preStop:
 httpGet:
 path: /shutdown
 port: 8000

This requires the engine to expose a graceful shutdown endpoint; vLLM v1 supports it via --enable-graceful-shutdown. Without it, scale-down breaks the SLO even though the metrics do not capture it (cut requests never enter the TTFT histogram).

Pitfall 2, scale-up/scale-down oscillation. If the scale-down stabilizationWindowSeconds is short (~60 s default), the next drop in the queue triggers scale-down, and two minutes later the next peak triggers scale-up. The system oscillates, pays repeated cold starts, and never reaches a stable regime. Solution: scale-down with a window of 10 minutes minimum and conservative policies (type: Pods, value: 1, periodSeconds: 120, at most one replica fewer every 2 minutes).

Pitfall 3, vllm:num_requests_waiting with avg when there is rebalancing. If two replicas are unbalanced (one with queue 20, the other with queue 0), avg gives 10 and the HPA triggers scale-out when the right move would be to rebalance via the load balancer. To detect it: add an alert on stddev(vllm:num_requests_waiting) per deployment. If the dispersion is high, the problem is not capacity but routing.

Full example manifest

For a vLLM deployment with Llama 70B FP8 on 4×H100 SXM per replica, KEDA with warm pool 2:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama70b
 namespace: inference
spec:
 replicas: 2 # managed by KEDA afterwards
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata:
 labels: { app: vllm-llama70b, deployment: vllm-llama70b }
 spec:
 terminationGracePeriodSeconds: 120
 nodeSelector: { workload: gpu, gpu-model: h100-sxm-80gb }
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args:
 - --model=/models/llama-3.3-70b-fp8
 - --tensor-parallel-size=4
 - --max-num-seqs=64
 - --enable-prefix-caching
 - --enable-graceful-shutdown
 ports:
 - { name: http, containerPort: 8000 }
 - { name: metrics, containerPort: 8000 }
 resources:
 limits:
 nvidia.com/gpu: "4"
 memory: 200Gi
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 60
 periodSeconds: 10
 failureThreshold: 30 # tolerates the warmup
 lifecycle:
 preStop:
 httpGet: { path: /shutdown, port: 8000 }
 volumeMounts:
 - { name: model-cache, mountPath: /models, readOnly: true }
 - { name: dshm, mountPath: /dev/shm }
 volumes:
 - name: model-cache
 persistentVolumeClaim: { claimName: llama70b-fp8-pvc }
 - name: dshm
 emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: vllm-llama70b-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: vllm-llama70b } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: vllm-llama70b-scaler, namespace: inference }
spec:
 scaleTargetRef: { name: vllm-llama70b }
 minReplicaCount: 2
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600
 policies:
 - { type: Pods, value: 1, periodSeconds: 120 }
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - { type: Pods, value: 2, periodSeconds: 60 }
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue
 threshold: "5"
 query: avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv
 threshold: "0.85"
 query: avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5"
 end: "0 19 * * 1-5"
 desiredReplicas: "6"

This set is the minimum viable one for LLM autoscaling on a generic cluster with the NVIDIA GPU Operator. Each team adapts it to its own SLO.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB per node, 4 GPU nodes:

Each node hosts one vLLM replica with TP=4 and Llama 70B FP8 (one model per node, they are not shared).
Warm pool of 2 replicas off-peak; the KEDA cron raises it to 4 during working hours.
Cluster-autoscaler does not apply (4 physical nodes bought; scaling is pods only). The number of concurrent replicas is at most the number of available nodes (if each replica uses all 4 GPUs of the whole node).
If the sizing requires more simultaneous replicas than nodes, there are two routes: (a) lower the TP of each replica so two fit per node, (b) expand the physical nodepool. The decision is dictated by capacity planning, see Capacity planning for on-premise LLM inference.

KEDA event volume: around 5 evaluations/min per ScaledObject. For 10 models served in parallel, 3,000 evaluations/h. Manageable with one KEDA operator per cluster.

What we have not covered (upcoming articles)

Cluster-autoscaler for on-premise GPU nodes: how to orchestrate bare metal provisioning (Tinkerbell, Metal³) as a function of demand.
Multi-cluster autoscaling: scaling across clusters in different DCs for geographic resilience.
Cost-aware autoscaling: prioritising nodes by hourly energy cost (on clusters with indexed tariffs).
Predictive ML-based scaling: instead of a static cron, training a model that predicts demand 30 minutes ahead.
Multi-tenant quotas and fairness: KEDA with namespace quotas so that one tenant does not monopolise the HPA.

References

KEDA project — keda.sh (documentación oficial de triggers Prometheus y cron).
Kubernetes — Horizontal Pod Autoscaler walkthrough (kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale).
NVIDIA — GPU Operator on Kubernetes (Helm chart oficial con DaemonSet de drivers y DCGM).
vLLM project — production_monitoring/ (métricas Prometheus expuestas por el servidor).
Karpenter — NodePool spec (etiquetado y taints para nodepools GPU).
Cluster Autoscaler — Scaling GPU nodes (caveats de descubrimiento de recursos GPU).
Kubernetes — Pod lifecycle and termination (preStop, terminationGracePeriodSeconds).

Seven greenfield deployment phases for an on-premise LLM platform: from hardware in the room to the first production token

Sun, 31 May 2026 08:00:00 +0200

TL;DR

The two previous posts in this architectural trilogy settled the pieces: the seven layers of the LLM inference stack describe the components above the cluster, and the five maturity levels of the platform describe the strata below. This post settles the when: in what order each thing gets deployed when you start from zero, hardware bought, racks installed, physical cabling done, and you want to reach a cluster serving the first production token to a customer. Seven nominal phases F0 to F6 with no calendar commitments, organised by technical dependencies (you do not enter F3 without the F2 gate) and with an identifiable critical path. F0 hardware inventory and electrical/network connectivity. F1 bare metal OS + drivers + container runtime. F2 Kubernetes cluster with CNI and Ceph storage operational. F3 GitOps and infrastructure observability. F4 identity, TLS, secrets and policies. F5 GPU platform with LLM-aware observability. F6 the LLM stack operational and open to production traffic. For each phase: what gets built, what has to be ready first (dependencies between phases), the gate that validates its closure, and the typical trap that delays the critical path. The thesis: an on-premise LLM platform sinks far more often from sequencing badly than from choosing badly. The tools have all been invented; the order is the only thing every team rediscovers.

You are here: the seven phases and their dependencies

The phases do not run in pure series. F2 and F3 can be started at the same time to speed things up (installing Kubernetes and preparing the GitOps repo in parallel). F4 can overlap with the tail end of F3. F5 waits for F4 to close because GPU pods demand NetworkPolicy and RBAC from day one. F6 is a single atomic step: the cluster goes into production or it does not.

The red arrows are the critical path: the sequential bottleneck that no parallelism can shorten. The grey arrows are dependencies that admit partial overlap. Recognising where to overlap and where not is the difference between a three-month deployment and a six-month one for the same perimeter.

The analogy: the expedition to an eight-thousand-metre summit

An expedition to a high alpine summit is not a long trek. It is a series of camps that are set up in order, each with its altitude, its function and its validation gate: if you do not acclimatise properly at base camp, you cannot go up to C1 without risk; if C2 does not have its kitchen and its radio running, you cannot send people higher; if the summit push is attempted without porters in the high camps, there is no safe descent.

Greenfield deployment of an LLM platform works identically. F0 is the arrival of the material at base camp: boxes, sponsors, permits, first review. F1 is setting up an operational base camp: kitchen, tents, generator. F2 is the climb to C1: there is real altitude now (a k8s cluster running) and you breathe differently. F3 is C2: it adds communications, planning and operational acclimatisation. F4 is C3, the last night before the push: team roped up, oxygen ready, every protocol verified. F5 is the day of the summit push, intense effort, thin margins. F6 is the summit and the start of a safe descent: from here on the expedition is in day-to-day operation, no longer in construction.

The analogy holds two useful lessons: you do not skip a camp (going straight from base camp to the summit kills the team), and the gates are technical, not emotional (if the barometer says storm, you do not set out, however much enthusiasm there is). The platform team that follows those two rules reaches the summit. The one that negotiates them does not.

F0 — Hardware in the room: base camp

What gets built in this phase. An inventory of the hardware received (servers, switches, PDUs, BMC), racks assembled, electrical and data cabling terminated, physical labelling of each machine (rack/U/function), connectivity to the corporate network, management IPs assigned, BMC reachable over VPN with MFA, first ping to each node from the bastion.

Dependencies. None technical, this is the phase before the software. There are procurement dependencies (servers bought, switches bought), civil works dependencies (a room with enough cooling, a raised floor) and administrative ones (datacenter access for the technicians).

Validation gate that closes F0.

Every node appears in the inventory with (hostname, MAC, management IP, data IP, rack, U, function, owner).
Each node’s BMC responds to ipmitool power status and to the HTTPS UI from the management VPN.
The top-of-rack switch has its configuration versioned in git (even if there is no cluster GitOps yet, the switch configs are).
A command for h in $(cat hosts); do ping -c1 -W1 $h.mgmt; done returns 100 % success.

Typical trap. Cabling “however it comes out” with no physical labelling and no diagram. When F4 arrives and a NetworkPolicy has to be troubleshot, not knowing which physical interface carries which VLAN doubles the diagnosis time of every incident for ever.

Why F0 does not overlap with F1. Until every server has a management IP and a live BMC, OS bootstrap cannot be automated. Every hour invested in F0 saves hours in every later phase. It is the phase with the best ROI in the project and the only one that admits no shortcuts.

F1 — Bare metal: the operational base camp

What gets built. The operating system image (stable Debian or Ubuntu LTS) provisioned via PXE or cloud-init with the cloud-config versioned in git. Each node has: a coherent hostname, LVM partitions, kernel ≥ 6.6, the containerd container runtime, NVIDIA drivers on the GPU nodes, chrony synchronising against your own servers, the operator’s SSH key as the only way in, and nvidia-smi passing a smoke test on the GPU nodes.

Dependencies. F0 closed. It needs the management network working so PXE responds and so the bastion can reach every node.

Validation gate that closes F1.

ansible -i inventory all -m ping returns 100 % success (or the equivalent with Salt / Pulumi / etc).
Every GPU node passes nvidia-smi showing the expected GPUs with a consistent driver across nodes.
Each node’s clock drifts by less than 50 ms from the reference NTP.
A physical reboot of a node leaves it in exactly the same state after boot (idempotency).

Typical trap. NVIDIA drivers installed by hand with apt install or with NVIDIA’s .run script. It works on day one and breaks the day of the first kernel update. The operational rule already established in the post on the five levels: the drivers end up managed by the GPU Operator in F5; whatever is done now is only so that nvidia-smi passes the smoke test, not for production.

Possible overlap. F1 can start for some nodes while F0 is still being finished on others (real greenfield rarely delivers every server on the same day). The F1 gate is per cluster, not per individual node.

F2 — Kubernetes cluster operational

What gets built. RKE2 installed with three HA control plane nodes, all workers joined (CPU and GPU), Cilium as CNI with kubeProxyReplacement enabled and a BGP control plane pointing at the ToR switches from F0, Rook-Ceph deployed on the storage nodes to cover block (RBD), filesystem (CephFS) and object (S3-compatible RGW), kubectl get nodes returning every node Ready, a first test pod with a PVC mounting and data persisting after the pod restarts.

Dependencies. F1 closed (drivers + container runtime). Switches with BGP configured (closed in F0). NVMe disks partitioned or available raw for Ceph OSDs.

Validation gate that closes F2.

kubectl get nodes -o wide shows every node Ready with the expected Kubernetes version.
A Deployment with replicas=3 and per-node antiAffinity starts and the pods land on different nodes.
An RWO PVC (RBD) creates a volume, the pod writes data, the pod is deleted, another pod mounts it and reads the data.
An RWX PVC (CephFS) does the same with two pods writing simultaneously.
An RGW bucket via s3cmd or mc accepts put and get over TLS.
Hubble (the read side of the CNI) shows flow logs between two pods in different namespaces.
Chaos test: draining a non-GPU worker node; the workloads are rescheduled automatically.

Typical trap. Starting to kubectl apply real workloads in F2 without GitOps. The backlog of things-applied-by-hand grows faster than the capacity to migrate it to git afterwards. The rule: in F2 only the cluster prerequisites are applied (CNI, CSI, default storage class). Any application workload waits for F3.

Possible overlap. F2 ↔ F3. While the cluster is being built, the GitOps repo is prepared in parallel (directory structure, first Helm releases). When F2 closes, Flux is plugged into the repo and everything that was going to be kubectl apply is already a reconciled manifest.

F3 — GitOps and infrastructure observability

What gets built. Forgejo deployed first (it is a prerequisite for everything that follows). A gitops-infra repo with the initial structure (apps/, infrastructure/, tenants/, clusters/). Flux installed and reconciling that repo. The prerequisite workloads applied by hand in F2 are moved to the repo and reconciled (no operational kubectl apply remains). VictoriaMetrics + vmagent scraping metrics. Grafana with initial dashboards (USE/RED + cluster + Ceph + Cilium). Loki receiving logs via vector/fluent-bit. Alertmanager + Keep routing alerts to a chat channel. Barman Cloud backups for Postgres (future CNPG) and scheduled Ceph snapshots.

Dependencies. F2 closed. An RGW bucket to store backups (covered by the Ceph from F2).

Validation gate that closes F3.

A change applied to the repo is reflected in the cluster in under 5 minutes with no manual intervention.
A change applied with kubectl edit directly to the cluster is detected by Flux and reverted (binding drift detection, not merely observational).
Grafana shows dashboards for the cluster, Ceph, Cilium and GPU nodes (DCGM does not arrive until F5, but basic node metrics do).
A test alert sent to Alertmanager reaches the chat channel in under 1 minute.
Restoring a Postgres backup on a temporary cluster returns coherent data (the test defines the real RPO).

Typical trap. Having Helm charts in git but still applying them with helm install from the terminal. That is level 1 dressed up as level 2. F3 only closes when Flux is the only authority that applies changes and humans edit the repo, not the cluster.

Possible overlap. F3 ↔ F4. While F3 is being closed, the Defguard and cert-manager manifests can be prepared in the repo. When they reconcile they have somewhere to land.

F4 — Identity, certificates, secrets, policies

What gets built. Defguard deployed with its dedicated Postgres (CNPG). An initial realm with the platform operators enrolled with MFA and WireGuard. OIDC integrated into kube-apiserver (--oidc-issuer-url, --oidc-client-id, --oidc-username-claim), into Forgejo, into Grafana, into Alertmanager, one single SSO. cert-manager installed with an internal CA issuing internal certs for mTLS and with Let’s Encrypt ACME for edge certs. SOPS configured with a KMS (it can be a physical HSM, an age key in a vault, or an external Vault) and External Secrets Operator syncing secrets into the cluster. Kyverno deployed with initial policies in audit mode for a week, then promoted to enforce. A default-deny NetworkPolicy applied to every existing namespace. Tetragon enabled for runtime security. The kube-apiserver audit log sent to Loki with long retention.

Dependencies. F3 closed (Flux applies the manifests, VM/Loki ingest metrics and logs).

Validation gate that closes F4.

kubectl with a shared admin kubeconfig stops working; every operator uses their own OIDC token with MFA.
A secret in plain data: in a commit is rejected by the pre-commit hook (or by Kyverno admission).
A pod without securityContext.runAsNonRoot=true is rejected by Kyverno at admission.
A deliberately wrong NetworkPolicy (allow-all) in a tenant namespace is rejected.
An audit of the last day returns the complete list of actors and changes (minimum regulatory footprint).
Basic internal pen-test: an attacker with a forged kubeconfig fails MFA and is recorded.

Typical trap. Kyverno in permanent audit mode because “we do not want to break production workloads”. F4 closes when the policies are in enforce. Until then, you are still in F3 wearing an F4 face.

Why F4 does not overlap with F5. F5 introduces GPU pods that move a lot of VRAM and a lot of compute. Without default-deny NetworkPolicy, without OIDC RBAC, without Kyverno blocking insecure configurations, the GPU pods are the juiciest attack surface in the cluster. Any compromise in F5 with F4 not closed is near-total access to the expensive hardware.

F5 — GPU platform with LLM-aware observability

What gets built. NVIDIA GPU Operator via Flux with the driver version decided in F1 (no longer touched by hand). DCGM Exporter exposing GPU metrics to VictoriaMetrics. MIG manager configured for the nodes where it makes sense (for example, on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs and embeddings). Topology Manager with the single-numa-node policy. KEDA with the Prometheus scaler installed and an example ScaledObject pointing at a vLLM metric (vllm:num_requests_running). OpenTelemetry Collector with OTLP receivers, attributes processors (enriching spans with tenant_id, priority_tier), exporters to Langfuse and to Tempo. LeaderWorkerSet API enabled for tensor parallel topologies. OME (Operator Model Engine) or vLLM Production Stack deployed as a controller, still with no models loaded.

Dependencies. F4 closed (the GPU pods inherit default-deny NetworkPolicy, OIDC RBAC and Kyverno policies).

Validation gate that closes F5.

A test pod requesting nvidia.com/gpu: 1 is scheduled on the right node and nvidia-smi from inside the container sees the right GPU (whole or a MIG slice).
DCGM Exporter exposes metrics in Grafana (utilisation, VRAM, temperature, NVLink bandwidth) for every GPU.
A test vLLM Deployment starts with a small model (for example, a 7B FP16) loaded from Ceph RGW.
An OpenTelemetry span generated by that vLLM reaches Langfuse with correct gen_ai.* attributes.
KEDA scales the test Deployment from 1 to N replicas under synthetic load and back to 1 when it stops.
A GPU Operator upgrade to a new version drains and reschedules the GPU pods with no loss of service.

Typical trap. Loading the large model “just to try” before DCGM and OTel are green. When something fails, there will be no metrics to distinguish OOM, thermal throttling, a driver mismatch or a network problem, and you diagnose blind. The rule: small model first, golden path green, then the large model.

Possible overlap. None with F6. F6 is atomic.

F6 — LLM stack in production

What gets built. The seven layers of the inference stack described in the corresponding post, deployed in this order:

Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets). Some components already existed from F3 as data; here they specialise for RAG with their initial collections and schemas.
Embeddings + reranker (Infinity with multilingual-e5-large, TEI with bge-reranker-v2-m3). This is the layer that must be green before any large model, because RAG depends on it.
LLM inference (vLLM Production Stack with the general LLM and the code LLM). It loads models from Ceph RGW. The initial multi-LoRA pool is empty.
Gateway (Envoy AI Gateway) with Defguard OAuth, routing by body.model, rate-limit per tenant. This is the point that opens traffic to the outside.
LLM-aware observability (Langfuse plugged into the OTel from F5).
GitOps control plane and dependency tracking were already active since F3 and F4 respectively; here you simply add the catalogue of the new LLM services to them.

Dependencies. All the previous phases closed.

Validation gate that closes F6.

A curl to the public endpoint with a Defguard bearer token receives a correct chat completion response in technical Spanish, with trace_id propagated.
The trace appears in Langfuse with complete gen_ai.* attributes, a latency breakdown and its own tenant_id.
A 5 % traffic canary to the new model over 24 h degrades neither quality nor latency metrics.
A controlled traffic burst triggers KEDA, the replicas scale, and P95 latency stays within budget.
A deliberate failure of one vLLM pod does not affect endpoint availability (replicas + reschedule).
The internal operator demonstrates the complete path for revoking a tenant’s access in under 5 minutes (Defguard → Kyverno → closing the NetworkPolicy).

Typical trap. Opening real customer traffic before having the incident runbook signed off, the SLO negotiated and the continuity plan tested. F6 is technically closed; operationally, the platform is still an experiment until the first real postmortem proves the team knows how to respond.

The maths that matter: relative weight of effort per phase

Without committing to calendar weeks, we can quantify the relative weight of the engineering effort per phase in a typical greenfield. The curve is not uniform:

$$ \text{effort}_{F_i} \approx \text{base}_i \cdot (1 + \epsilon_i) $$

where $\text{base}_i$ is the nominal effort and $\epsilon_i$ is the surprises factor (wrong cabling, incompatible drivers, badly issued certificates, version conflicts). The table below gives the nominal relative weight and the typical surprise factor observed:

Phase	Nominal weight	Typical surprise factor ε	Mean effective weight
F0 — Hardware	8 %	0.5 (1× to 2×)	12 %
F1 — Bare metal	6 %	0.3	8 %
F2 — k8s cluster	12 %	0.4	17 %
F3 — GitOps + obs	14 %	0.5	21 %
F4 — Identity + policies	18 %	0.7	31 %
F5 — GPU plane	10 %	0.4	14 %
F6 — LLM stack live	8 %	0.3	10 %
Buffer / integration	24 %	—	—

Two operational observations. F4 concentrates more surprises than any other phase (OIDC federation across four or five apps with different configurations, Kyverno policies that knock over legitimate workloads, secrets broken by poorly tested encryption). F0 has a high surprise coefficient relative to its size because any cabling or labelling error is discovered late and paid for dearly. The two practical consequences: plan F4 with generous margin and do not skimp on time in F0, because every hour saved there costs five later.

Critical path and slack. The critical path is linear F0 → F1 → F2 → F3 → F4 → F5 → F6. The only real slack is the overlaps already identified:

F2 ↔ F3 (slack ~30 %): prepare the repo and initial dashboards while the cluster is being built.
F3 ↔ F4 (slack ~20 %): identity manifests ready as F3 closes, applied immediately.
Within F4: policies in audit mode running in parallel with the Defguard setup.

Nothing shortens the critical path by more than about 15 % of the total. Anyone promising a productive greenfield in half a reasonable timescale is selling something else: probably skipping F4, or loading F6 with an F5 that is green-but-not-validated.

Final diagram: the complete deployment schedule

The schedule is not decorative: each row defines what gets built in its phase and the gate that closes it. A phase is not considered finished until its gate is green. A phase with a yellow gate drags every later one; trying to jump to the next with a partially met gate is what produces, several weeks later, the incident that forces you to “go back to F4 with production running”, the most expensive situation in the whole cost matrix of the post on the five levels.

Typical planning mistakes

Patterns that delay or sink the greenfield deployment, regardless of the tools chosen:

1. Buying the LLM before the cluster. Starting the project with “which model are we going to serve” instead of “what platform can sustain any reasonable model”. The model is an interchangeable parameter; the platform is not.

2. Underestimating F0. “The network team does that.” Yes, but the output of F0 is consumed by every later phase. If the network team delivers late, the whole project is late, and nobody had marked it as the critical path.

3. Overlapping F4 with F5 “to save time”. It is the one dependency with no slack. If you try to overlap, F5 ends up operating with policies in permanent audit (you are not in F4) or without OIDC integrated (operators with a shared kubeconfig touching GPUs). Both antipatterns stay in production.

4. Skipping the small model smoke test in F5. “Let’s go straight for the 70B.” When something fails (and something will), there will be no baseline to diagnose against.

5. Treating F6 as “turning on vLLM”. F6 includes gateway, LLM-aware observability, runbook, SLO, continuity plan. Turning on vLLM takes five minutes; closing F6 is weeks of validation and sign-off.

6. Not defining gates in writing. If the gates are not written down, they are negotiable after the fact. “This already counts as F4” is the sentence that precedes the next six months of retrofit.

7. Assigning a phase to a single owner. Every phase needs at least two people who understand it. Staff turnover on long projects destroys knowledge; written gates + cross review preserve it.

8. Forgetting the descent route. The post focuses on going up. Day-to-day operation (the descent, in the analogy) is another story that also deserves planning: runbooks, on-call, upgrade capacity, end-of-life plan. Teams that only plan the ascent reach the summit and stay there with no oxygen.

Applied to typical on-premise hardware: 4×H100 SXM

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM per GPU node, 3 control plane nodes, 3-5 CPU worker nodes, 2 GPU worker nodes), the temporal split of the work is distributed like this:

F0 (hardware)
└─ 8 physical servers racked + switches + BMC + management IPs
 ├─ 3 nodes cp-01..03 — control plane (no GPU)
 ├─ 3 nodes worker-cpu-01..03 — CPU plane (Forgejo, Ceph, observability)
 └─ 2 nodes worker-gpu-01..02 — GPU plane (4×H100 SXM each)

F1 (bare metal)
└─ OS + drivers + containerd on all 8 nodes alike
 (NVIDIA drivers only on the 2 GPU nodes, `nvidia-smi` smoke)

F2 (k8s cluster)
└─ RKE2 control plane on cp-01..03 (HA with embedded etcd)
 workers joining: 3 CPU + 2 GPU
 Ceph OSDs on the 3 CPU worker nodes
 default pools: RBD-replicated-3, CephFS-replicated-3, RGW

F3 (GitOps + obs)
└─ Forgejo + Flux + VM/Grafana/Loki + Keep on the CPU plane
 first `gitops-infra` repo reconciles what came from F2

F4 (identity)
└─ Defguard on the CPU plane (StatefulSet with Postgres CNPG)
 OIDC in kube-apiserver, Forgejo, Grafana, Alertmanager
 Kyverno as a Deployment on the control plane

F5 (GPU plane)
└─ NVIDIA GPU Operator targets the GPU workers
 MIG manager: 1st GPU MIG 7g.80gb (= passthrough), 2nd 2×3g.40gb
 OTel Collector as a DaemonSet on the GPU plane + CPU plane
 first vLLM with a 7B FP16 model green

F6 (LLM stack)
└─ The 7 layers reconcile via Flux from a second repo `gitops-llm`
 first public endpoint with Defguard OAuth
 first production customer enrolled under an SLA

The physical distribution of the cluster exploits the isolation between planes defined in F0: the control plane never touches a GPU, the CPU plane concentrates relevant state (Forgejo, Ceph, Postgres CNPG, Langfuse, Defguard) and the GPU plane specialises as much as possible. That separation, decided in F0 before the first server is installed, conditions the success of the remaining phases. It is another reminder of why F0 matters more than it looks.

What we have not covered (upcoming posts)

This post walks the ascent route to the summit. Some pieces still deserve an article of their own:

The safe descent: day-to-day operation, runbooks per component, on-call, continuous capacity planning, the cluster upgrade cycle with no downtime.
Multi-site (the second summit): how two clusters are federated with Cilium Cluster Mesh and what extra phases that introduces. F3.5 (Cluster Mesh) and F4.5 (cross-site replication) are the missing phases.
The brownfield route: what changes when there is already a cluster with workloads. The phases stay the same, but the gates are applied retroactively and every step requires migration planning.
The real calendar cost: typical ranges in weeks for a platform team of 2-3 people, split by phase, with uncertainty bands.
The handoff to operations: how the platform is handed from the deployment team to the operations team, which documents are signed, what is inherited and what is renegotiated.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook documentation — rook.io/docs
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai

Five maturity levels of the platform under the LLM: from a Linux server to a cluster ready for vLLM

Sun, 31 May 2026 07:00:00 +0200

TL;DR

The post on the seven layers of the LLM inference stack took many pieces for granted: an operational Kubernetes cluster, GitOps reconciling, identities resolved, GPUs visible to the scheduler, observability capable of carrying gen_ai.*. Before vLLM makes any sense, you have to reach that starting point, and you reach it by levels. This post defines five maturity levels of the platform that lives under the LLM, from a bare metal server with Linux installed (level 0) to a cluster ready to run the inference layer (level 4) and the handoff to the previous post (level 5). Each level unlocks a concrete capability, running containers reproducibly, rebuilding the cluster from git, authenticating humans via OIDC, scheduling GPUs with MIG and DCGM metrics, demonstrating compliance with no manual intervention, and each has a validation test that decides whether you are really there or just telling yourself you are. For each level: which OSS pieces cover it in 2026 (Cilium, RKE2, Flux, cert-manager, Defguard, NVIDIA GPU Operator, KEDA, Trivy, Kyverno…), the deployment order within the level, the decisions that are expensive to skip, and the antipatterns that drop you a level when you thought you were higher. The thesis: moving up a level costs little effort if you do it in time, and a lot of refactoring if you try to skip it. LLM inference demands at least level 4; anyone trying to serve LLMs from level 1 or 2 ends up paying in night-time incidents what they saved on the platform.

You are here: the five levels at a glance

Before the detail, the ladder. Each rung adds a capability absent from the one below. The level’s test is the question whose honest answer says whether you are already there.

The levels are not interchangeable. A cluster at level 2 cannot run LLMs in production with any guarantee: technically the vLLM pod loads, but at the first night-time incident you discover there is no TLS, no identities, no alerting, no GPU metrics, and no way to know who changed what. Moving up a level after LLMs are already in production costs orders of magnitude more than moving up while the cluster is still empty.

The analogy: from the street stall to the starred restaurant

Picture the scale of a hospitality business. Level 0 is the street stall: a griddle, a gas bottle, a cook improvising. It can sell food, it works, but anything that deviates from a normal day (a health inspection, an allergic customer, an order for 200 portions) sinks the business. Level 1 is the tapas bar: a properly sized kitchen, a short repeatable menu, several shifts. The cook no longer improvises daily; he works from a written menu, even if the recipes live in the head of the boss. Level 2 is the restaurant with a set lunch menu: written procedures, fixed suppliers, stock control, an incident book. If the head cook falls ill, the second can run the service without carnage. Level 3 is the restaurant with an à la carte menu and formal service: traceability of every ingredient, allergens on the menu, health certification, contracts with suppliers, mandatory staff training. Level 4 is the kitchen specialised in a complex product (sushi, haute cuisine, artisan bakery): specific tools the normal restaurant does not need (wood-fired oven, special knives, proving chamber), calibrated processes, quality metrics. Level 5 is the restaurant with a Michelin star: the whole system works, the dish is the result of the organisation, not of one person’s talent.

The analogy holds to the end, including the most interesting detail: you can operate at any level, but the promises you can keep are different. The street stall cannot promise a consistent experience to 80 diners with a booking. The cluster at level 1 cannot promise productive multi-tenant LLM service with an SLA. In both cases the problem is not the technical capability of the last component (the griddle cooks; the pod starts); it is the organisational capability of the whole system.

Let us go level by level.

Level 0 — Chaotic: the server with Linux and nothing else

The capability it gives. Running containers with docker/podman, running binaries, connecting the server to the network. The operator can SSH in, do things, and see results.

The test for the level. “If I reinstall the server from scratch, can I leave it identical to how it was in one afternoon, using only saved notes?”. If the answer is no (because the steps are in the head of whoever built it, in .bash_history, in an out-of-date wiki), you are at level 0.

Minimum pieces to settle before moving up to level 1.

Piece	Suggested decision in 2026	Why it matters when moving up
Linux distribution	Stable Debian or Ubuntu LTS	Long, predictable support, recent kernel available
Kernel	Recent LTS (≥ 6.6) with BPF and modern schedulers	Cilium/eBPF and recent NVIDIA drivers demand it
NVIDIA drivers	The version that matches the CUDA of the LLM engine you will serve	A driver/CUDA mismatch blocks vLLM before you start
Container runtime	`containerd`	CNCF standard, integrated with RKE2/kubeadm
Root filesystem	XFS or ext4 + LVM thin pools	Snapshots, online expansion
Time synchronisation	`chrony` with your own servers	TLS, correlated logs and short certificates demand it
Management network	Dedicated VLAN, ACLs on the switch	Isolate the control plane from workload traffic
Cluster network	LACP + jumbo frames + BGP (if you are going with Cilium)	Intra-node NVLink does not rescue the service network
BMC / IPMI	Out-of-band access with TLS and MFA	Recovery when the operating system does not boot

Antipatterns that pin you at level 0.

Pet servers (with their own names, configured by hand, not replaceable).
Changes applied with vi directly over /etc/... with no commit to a repo.
Deployment with docker-compose with no healthchecks and no automatic restart.
An inventory that lives in an Excel sheet nobody updates.

Deployment order within the level. System image from PXE/cloud-init with the initial configuration (LVM, hostname, network, SSH key, chrony) → bastion/jump host bootstrap → inventory in Ansible (or an equivalent declarative tool, even if it is later replaced) → NVIDIA drivers + container runtime → smoke test (a CUDA container passes nvidia-smi). At this point, the server is ready for Kubernetes to come in.

Level 1 — Repeatable: an operational Kubernetes cluster

The capability it gives. Scheduling containers with a scheduler, network abstraction between pods, persistent volumes, workload lifecycle, manual horizontal scaling.

The test for the level. “Can I lose a node and have the workloads rescheduled without human intervention?”. If yes, you are at level 1. If not, because the pods are pinned to nodes, because there are no replicas, because the PVCs do not reattach, you are still at 0 with Kubernetes on top.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Main alternative
k8s distribution	RKE2 (CIS-hardened by default, no commercial surcharge)	k3s for very small edge, plain kubeadm for custom cases
CNI	Cilium with kube-proxy replacement, BGP, Gateway API	Calico (without BGP it does not compete with Cilium in 2026)
CSI block + filesystem + object	Rook-Ceph (RBD + CephFS + S3-compatible RGW)	OpenEBS Mayastor + Garage for small deployments
Ingress	Cilium Gateway API (better to unify with the CNI)	NGINX Ingress, Traefik
Basic certs	Self-signed bootstrap	(cert-manager comes in at level 3)
Workload handling	`kubectl apply` + Helm from the terminal	No GitOps yet
Container registry	Any internal registry (or a trusted external one) with TLS	(a managed internal registry comes in at level 2)

Antipatterns that drop you to level 0.

Services deployed with kubectl apply from one person’s terminal and with the YAML saved nowhere.
Persistent volumes with no backup policy.
A “single-node cluster” as permanent production, an architectural single point of failure.
A CNI with no NetworkPolicy available or no BGP when the network requires it.

Deployment order within the level. RKE2 installed on at least three nodes for an HA control plane → Cilium installed in kube-proxy replacement mode + BGP control plane → Rook-Ceph on at least three nodes covering block (RBD) + filesystem (CephFS) + object (S3-compatible RGW) with 3× replication or Erasure Coding depending on the pool → smoke test (a Deployment with a PVC starts, the pods reschedule when a node is cordoned, the data persists).

Level 2 — Defined: the cluster rebuilds from git

The capability it gives. The cluster state lives in a repository. Every change goes through a commit. Anyone can rebuild the cluster (or an equivalent one) from the repo and the backups. Basic observability warns when something breaks.

The test for the level. “If I lose the whole cluster, can I recreate it in X hours from the repo + the backups, with no manual intervention beyond the bootstrap?”. The hours are negotiable; what defines the level is that the repo + the backups are enough, not that the person-who-knows is available.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
Forge	Forgejo (or Gitea, GitLab CE)	Self-hosted OSS, community fork of Gitea, open governance
GitOps reconciler	Flux	CNCF graduated, native multi-tenancy, lightweight
Image registry	Forgejo Container Registry	Next to the code, no extra piece
Metrics TSDB	VictoriaMetrics + vmagent	Higher throughput than plain Prometheus, long retention, PromQL compatible
Visualisation	Grafana	The de facto standard
Logs	Loki or Vector	OSS, integrated with Grafana
Alerting	Alertmanager + Keep (OSS orchestrator)	Keep adds multi-channel routing with no lock-in
DB backups	Barman Cloud (Postgres)	The standard for CNPG
Object / dataset backups	Ceph RGW multisite + CephFS snapshots	Cross-pool and cross-site

Antipatterns that drop you to level 1.

kubectl apply run in production outside the repo (undetected drift).
main branches with write permissions for humans without review.
A monolithic repo with no tenant/infra/apps separation (cross changes that cannot be audited).
Metrics kept no longer than 7 days (no SLO observable over a month).
Alerting that fires for everything (fatigue) or for nothing (silence).

Deployment order within the level. Forgejo deployed first (it is a prerequisite for everything else) → Flux installed and pointing at the manifests repo → an initial repository with the Cilium and Rook-Ceph Helm releases reconciled by Flux (replacing the level 1 kubectl apply) → VictoriaMetrics + Grafana + Loki via Helm/Flux → Postgres backups and Ceph snapshots scheduled → smoke test (tear the cluster down, restore from repo + backup, the services come back).

Level 3 — Managed: identities, certificates, secrets and policies

The capability it gives. Every human operating the cluster does so with their own identity (no shared kubeconfig), with MFA and with limited permissions. Automatic internal TLS. Encrypted versioned secrets. Policies that reject insecure configurations before they reach the cluster. A complete audit of who did what.

The test for the level. “If an attacker gets hold of an administrator’s laptop, what can they do in production?”. At level 3 the answer is “little”: MFA blocks the second factor, Kyverno policies block destructive changes without approval, the NetworkPolicies prevent lateral movement, the secrets are encrypted with an external KMS, and the audit log remains. At level 2, “everything”.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
IdP / OIDC	Defguard	Spanish OSS, WireGuard + OIDC + 2FA, multi-org
Federation with the cluster	OIDC in kube-apiserver, OIDC in Forgejo, OIDC in Grafana	Consistent SSO
Internal PKI	cert-manager + Trust Manager	De facto standard, ACME and internal CA
External ACME	Let’s Encrypt for edge certs	No cost, automated
Secrets in git	SOPS + age or an external KMS	Versionable, encrypted in the repo
Secret syncing	External Secrets Operator	Pull from KMS / Vault into the cluster
Policy as code	Kyverno (or OPA Gatekeeper)	Kyverno has a shallower learning curve
NetworkPolicy	Cilium NetworkPolicy + L7	Default deny per namespace
Runtime security	Tetragon (Cilium)	eBPF, complements NetworkPolicy with detection
Vulnerability scanning	Trivy in the CI pipeline + admission	SBOM per image, blocking of critical CVEs
Audit log	kube-apiserver with `--audit-policy-file` sent to Loki	Regulatory traceability

Minimum Kyverno policies to keep alive.

Deny :latest images or images without a sha digest.
Deny pods without securityContext.runAsNonRoot=true.
Deny pods without resources.limits (CPU + memory).
Deny Services without an owner=<team> label.
Deny changes in critical namespaces (kube-system, flux-system) without an approval label.

Antipatterns that drop you to level 2.

A kubeconfig shared between administrators.
Secrets in plain data: in a manifest committed to the repo.
NetworkPolicy absent from new namespaces by default (implicit allow-all).
kubectl edit or kubectl patch in production without going through the repo.

Deployment order within the level. Defguard deployed and enrolled with WireGuard / OIDC → OIDC integration with kube-apiserver, Forgejo and Grafana → cert-manager installed and issuing internal certificates (your own CA for mTLS, Let’s Encrypt for the edge) → SOPS configured and External Secrets Operator installed → migration of secrets from plain to encrypted → Kyverno with initial policies in audit mode, then enforce → default-deny NetworkPolicy per namespace → Tetragon enabled → smoke test (try to bypass each policy and check that admission rejects it).

Level 4 — Optimised for GPU: the cluster now knows what an H100 is

The capability it gives. The Kubernetes scheduler sees the GPUs, distinguishes them, can partition them (MIG) or multiplex them (time-slicing), expose DCGM metrics, autoscale with KEDA using metrics from the LLM workload itself (vllm:num_requests_running, vllm:gpu_cache_usage_perc), and carry traces with the GenAI semantic conventions. Everything needed for the LLM inference stack to rest on a platform that understands its nature.

The test for the level. “If I deploy a pod asking for nvidia.com/gpu: 1, does it get scheduled on the right GPU, with the right slice, with DCGM metrics exposed, with GenAI observability ready to receive spans?”. If yes, you are at level 4. If the answer requires “it depends which node and who deploys it”, not yet.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
GPU device plugin	NVIDIA GPU Operator	Deploys drivers, container toolkit, DCGM and MIG manager with one operator
HW partitioning	MIG (Multi-Instance GPU) on the H100 where it applies	Real hardware isolation, not time-slicing
GPU metrics	DCGM Exporter	SM utilisation, VRAM, temperature, throttling, NVLink bandwidth
LLM metrics	vLLM Prometheus endpoint + scrape	TTFT, TPOT, KV cache, prefix hit rate
Autoscaling	KEDA with a Prometheus ScaledObject	Scales on LLM metrics, not on CPU
LLM operators	vLLM Production Stack / OME (Operator Model Engine)	Declarative handling of models / adapters
Traces	OpenTelemetry Collector with OTLP receivers + processors + exporters	`gen_ai.*` semantic conventions (post)
LeaderWorkerSet	LeaderWorkerSet API (k8s 1.30+)	Tensor parallel topology coherent with NVLink
Topology Manager	enabled with `single-numa-node`	Pinning GPU pods to the right NUMA node

The key decision: MIG, time-slicing or passthrough.

MIG divides an H100 into 1g.10gb, 2g.20gb, 3g.40gb, 7g.80gb (slices with real HW isolation). Useful for serving several small models or reserving capacity per tenant with a guarantee. Limitation: up to 7 instances per GPU, predefined profiles.
Time-slicing shares a GPU between several pods with no HW isolation. Useful for dev/test, not for multi-tenant production with an SLA.
Passthrough assigns the whole GPU to one pod. Useful for tensor parallel across several GPUs on the same node (a large LLM with TP=4).

For a productive LLM platform, the practical rule: passthrough for the large models with TP, MIG for embeddings and small models that cohabit, never time-slicing in production.

Antipatterns that drop you to level 3.

Installing NVIDIA drivers by hand outside the GPU Operator (silent breakage when Kubernetes is upgraded).
Serving an LLM with requests.gpu: 1 without having decided on MIG / passthrough (you end up with idle GPUs from fragmentation or pods stepping on each other).
KEDA autoscaling on CPU (the classic HorizontalPodAutoscaler) on pods that are almost always at 10 % CPU but 95 % KV cache.
OpenTelemetry deployed but without the gen_ai.* semantic conventions (the traces are not LLM-aware).

Deployment order within the level. NVIDIA GPU Operator installed via Helm/Flux with the driver version that matches the chosen LLM engine → DCGM Exporter enabled and metrics visible in Grafana (NVIDIA dashboards imported) → MIG manager configured for the nodes where it makes sense (a typical mix on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs + embeddings) → OpenTelemetry Collector with attributes processors to enrich spans with your own labels (tenant_id, priority_tier) + exporters to Langfuse and to Tempo → KEDA installed with an example ScaledObject pointing at vllm:num_requests_running → vLLM Production Stack or OME to declare models as a CRD → smoke test (a vLLM Deployment declared via CRD starts, serves a token, exposes metrics, the trace reaches Langfuse, KEDA scales under synthetic load).

Level 5 — Handoff: the cluster is an LLM platform, the seven layers go on top

Once at level 4, the cluster meets the contract that the post on the seven layers assumed as its starting point. Level 5 adds no infrastructure: it adds the LLM stack proper. For completeness, the seven components of level 5 are:

Gateway (Envoy AI Gateway) — it comes first, directing traffic to LLM inference and embeddings.
LLM inference (vLLM Production Stack or OME with vLLM) — on the GPUs already discovered by the level 4 GPU Operator.
Embeddings + reranker (Infinity, TEI) — a pod separate from the LLM, already covered in the previous post.
Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets) — most of it already existed at level 2 as data; now it specialises for RAG.
LLM-aware observability (Langfuse) — it plugs into the level 4 OTel chain.
GitOps control plane — the one from level 2 remains the only legitimate authority.
Dependency tracking (Hubble flows + Otterize) — on the Cilium that already existed at level 1.

The criterion for promoting from level 4 to level 5 is not technical: it is contractual. The cluster already supports LLMs; the decision is when to open real customer traffic. Promotion demands: a green golden eval of the model, a signed incident runbook, negotiated SLOs, a continuity plan, and mapping to ENS / NIS2 / 42001 where it applies.

The maths that matter: what skipping a level costs

To quantify the thesis of the post, an order-of-magnitude estimate of the cost of moving up each level in time versus moving up after production is running. The figures are engineering time with a small platform team (2-3 people), assuming templates and prior experience.

Level	Time to build on an empty cluster	Time to retrofit with production running
0 → 1	1-2 weeks	1-2 weeks (little downstream refactoring)
1 → 2	2-3 weeks	4-8 weeks (migrating everything to git)
2 → 3	2-4 weeks	8-16 weeks (image rebuilds, secret migration, retroactive RBAC)
3 → 4	1-2 weeks	4-8 weeks (reconfiguring GPUs, moving models to MIG, instrumenting `gen_ai.*`)
4 → 5	1-2 weeks	2-4 weeks
Total 0 → 5	~10-15 weeks	~20-40 weeks if done in the wrong order

The typical multiplier observable in practice: 2× to 3× the cost if it is done in the wrong order. And that assumes it gets done at all, since many projects never get past level 2 because “the identity thing” can always wait for another sprint. By the time the incident arrives, it is too late to start.

Beyond time, the operational cost (night-time incidents, security escapes, invisible debt) grows exponentially with the gap between the actual level and the required level. A cluster at level 2 serving productive LLMs to regulated customers is a time bomb: technically it works, organisationally it does not.

Final diagram: the complete ladder with pieces

The ladder is not decorative: each level enables the next. You cannot have LLM-aware observability (level 4) without OTel deployed via Flux (level 2). You cannot have automatic internal TLS (level 3) without a root PKI that lives somewhere (registry and certificates managed from level 2). You cannot have KEDA scaling on vLLM metrics (level 4) without Prometheus / VictoriaMetrics scraping (level 2). The levels are not a conceptual hierarchy: they are a hierarchy of installation dependencies.

Typical design decisions that break progress

Mistakes seen repeatedly that drag the cluster back a level:

1. Jumping from level 1 straight to level 4. “We are in a hurry to serve the LLM, we will do the identity and GitOps stuff later.” Later is always two orders of magnitude more expensive and always arrives after the first incident.

2. Confusing Helm with GitOps. Having Helm charts is not level 2. It is level 1 with templates. Level 2 demands that a reconciler (Flux/ArgoCD) applies the charts from a repo, detects drift and warns.

3. cert-manager with no usage policy. Having auto-renewed certificates but using TLS only at the ingress, with no internal mTLS between services, leaves the TLS promise lame and turns level 3 into a cosplay of level 3.

4. NVIDIA drivers by hand. It works on day one and breaks the day of the first kernel upgrade. The rule: drivers always via the GPU Operator, never operating system packages.

5. Prometheus metrics but 7 days of retention. Without long retention (≥ 90 days) there is no honest SLO. VictoriaMetrics with a year of retention costs little more than Prometheus with 7 days, and it unlocks compliance and serious postmortems.

6. OIDC only for kube-apiserver. If Forgejo, Grafana, Defguard and vLLM each have their own auth system, you do not have SSO, you have islands. An honest level 3 demands federation.

7. Kyverno in permanent audit mode. Policies that do not reject are not policies, they are alerts. At some point you have to move to enforce. Until then, you are still at level 2 wearing a level 3 face.

8. MIG with no conscious decision on the profile. Configuring MIG with the default profile without having measured the size of the models that will cohabit leaves fragmented GPUs with slices nobody uses. The rule: MIG only if you have measured and decided the profiles in advance.

They all share one root: declaring the level without passing the level’s test. Saying “we already did GitOps” when things are still applied with kubectl edit in prod. Saying “we already did identity” when there is a shared admin kubeconfig. Saying “we are ready for LLMs” when there is no DCGM Exporter and no Langfuse plugged in.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), a reasonable setup after passing the five levels distributes the components like this:

control plane (3 nodes without GPU, hostnames cp-01..03)
├── kube-apiserver, etcd, controller-manager, scheduler
├── Flux, Forgejo, cert-manager, External Secrets, Kyverno
└── Tetragon (DaemonSet here too)

worker plane (≥ 3 nodes without GPU, hostnames worker-cpu-01..03)
├── Cilium agent (DaemonSet)
├── Rook-Ceph OSDs + MONs + MDS (CephFS) + RGW (S3)
├── VictoriaMetrics + Grafana + Loki
├── Defguard (StatefulSet)
└── Langfuse + OTel Collector

GPU worker plane (≥ 2 nodes with 4×H100 SXM, hostnames worker-gpu-01..02)
├── NVIDIA GPU Operator (driver + container toolkit)
├── DCGM Exporter (DaemonSet)
├── MIG manager (configuring the chosen profile)
├── vLLM (Deployment) — general LLM TP=4 occupies 4 GPUs (passthrough)
├── vLLM (Deployment) — code LLM TP=2 occupies 2 GPUs
├── Infinity (embeddings) — 2 replicas cohabit on 2 MIG slices
└── KEDA scaler listening to vLLM metrics

The operational rule: the control plane and the CPU plane are separated from the GPU plane. An incident on the GPU plane must not take the control plane with it (the control plane is what recovers the cluster). And the CPU plane concentrates everything that moves relevant state (Forgejo, Rook-Ceph, Postgres CNPG, Langfuse): it is the heart to protect.

The GPU hardware specialises as much as possible: GPU pods run only on GPU nodes, and GPU nodes run nothing CPU-bound beyond the operational overhead (Cilium, GPU Operator, DCGM). This is enforced with nodeSelector + taints/tolerations + a Kyverno policy that rejects pods with no GPU requests being scheduled onto GPU nodes.

What we have not covered (upcoming posts)

This post walks the vertical path upwards. Some horizontal and cross-cutting pieces still deserve an article of their own:

Multi-site active/standby: how two clusters are federated with Cilium Cluster Mesh and what changes at each level when there are two sites instead of one.
Migration between levels with real traffic: how a cluster already in production is retrofitted to the next level with no downtime.
Day-to-day operation: runbooks per level, which dashboards to look at each morning, which SLOs to define per component.
The cost plane: what each level costs in hardware, energy, engineering hours, optional OSS licences (commercial support from Rancher, Cilium Enterprise, etc.) and when each expense is justified.
Operationalised compliance: how levels 3 and 4 map to ENS High, NIS2 and ISO/IEC 42001 controls without turning the cluster into a paperwork exercise.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook-Ceph — rook.io
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
vLLM Production Stack — docs.vllm.ai/en/latest/serving/production_stack.html
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
CIS Kubernetes Benchmark
NIST SP 800-207 — Zero Trust Architecture

Anatomy of an on-premise LLM inference stack: the seven layers that have to hold each other up

Sat, 30 May 2026 16:00:00 +0200

TL;DR

The on-premise LLM inference stack is not a model server: it is a building of seven layers that hold each other up. The inference layer (vLLM / SGLang) serves tokens, but without a dedicated embeddings layer RAG does not work; without the gateway layer the client couples its SDK to the specific engine; without an LLM-aware observability layer (Langfuse + OpenTelemetry GenAI) any quality degradation goes unnoticed; without a GitOps control plane layer (Flux + Forgejo) any manual change leaves invisible debt; and without a dependency tracking layer (Hubble + intent-based policies) decommissioning a Service silently breaks applications nobody remembers were using it. This post is born from a concrete incident: a pipeline that reported status: completed and matched_jobs: 0 for days because it kept calling an Ollama that had already been scaled to zero, while a badly written except labelled the generic ConnectError as “ChromaDB indexing error” and the vector store, innocent, took the blame. Each of the three symptoms was the cry of a layer that was missing or badly designed. The body of the post walks the seven layers with their canonical OSS piece, the design decisions that break them, the sizing maths on a generic 4×H100 SXM cluster (320 GB of VRAM, NVLink) and a final diagram of the connected stack. The thesis: a stack that passes the incident test is not measured by its peak throughput, it is measured by how long it takes to shout when something drifts from the design.

You are here: the seven layers seen from above

Before going into detail, the map. The seven layers are not seven servers: they are seven responsibilities the stack has to cover. A layer can collapse into one pod (gateway) or spread across several components (observability = traces + metrics + flow logs). What it cannot do is be missing.

Layers 1–4 sit on the request path: if they fall, the client notices in seconds. Layers 5–7 sit on the path of design and operations: if they fall, there is no immediately visible error, and that is why they are the ones that produce silent incidents, which is how most serious incidents end up. This post argues that the quality of the stack is measured in layers 5, 6 and 7, because 1–4 are commodities where everyone picks roughly the same pieces.

The analogy: the office building with shared services

Picture a twelve-storey office building with several tenants. It has a concierge desk (gateway: filters who comes in), it has lifts (LLM inference: they move the heavy load), it has stairs and a goods lift (embeddings: light but constant traffic, hardly anyone notices them until they break), it has plumbing and water tanks (vector store: what actually holds the state), it has a distribution board and sensors (observability: what warns you when something is drawing more than planned), it has a building manager (GitOps control plane: the only legitimate authority to move anything), and it has a tenant register (dependency tracking: who is connected to which shared service).

When a new client moves into the third floor and asks to install a server that needs more amperage than planned, the problem is not electrical: it is administrative. If the client can go straight to the board and plug in whatever they like, the building survives for a while and then a breaker trips at three in the morning. If the client has to go through the manager, the manager consults the tenant register (is anyone else hanging off that same circuit?), reviews the electrical planning (are we at the limit?) and either authorises or redirects. The building stays standing not because of its electrical installation but because of the discipline of going through the manager.

The LLM inference stack works identically. The physical layers (1–4) are the visible ones, the ones marketing people put on the slide. The governance layers (5–7) are the ones that distinguish a platform from a pile of pods with good luck.

Now to the incident that motivates the whole post.

The hook: the log that lied for six days

The application is called jobhunter internally. It is a cron pipeline that every six hours sweeps public job-ad sources, filters by EU geography, embeds the new ads, indexes them in a vector store and matches them against search profiles. The last step fires notifications.

For six days the pipeline reported the same thing on every run:

status: completed
total_found: 756
new: 23
matched_jobs: 0 ← zero, run after run

And in the logs:

[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/delete → 200 OK
[INFO] matcher: Purged 12 expired jobs from ChromaDB
[ERROR] pipeline: ChromaDB indexing error: All connection attempts failed
[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/get → 200 OK
[ERROR] pipeline: Matching error: All connection attempts failed

This is a log that invites you to blame ChromaDB. And in fact the first post-mortem written internally pointed at a version incompatibility between the Python client and the v2 server. A reasonable hypothesis, technically plausible, completely false.

The real cause: weeks earlier the general LLM on the platform had been migrated from Ollama to vLLM. The migration was clean for the two applications that depended directly on the large model, their manifests were pointed at the new endpoint. What nobody did was look at who else was calling the Service ollama.ollama.svc:11434. jobhunter called it to generate the embeddings for the ads. When the Ollama deployment was scaled to zero, the Service was left empty, and any outbound connection received a generic ConnectError("All connection attempts failed") from httpcore. The try/except wrapping the whole matching stage caught the exception and labelled it “ChromaDB indexing error”, because that was the lexical wrapper of the block, not because ChromaDB had anything to do with it. ChromaDB was answering 200 to delete and get in those same logs.

Three factors kept the incident alive for six days:

The except named the wrapper rather than the stage that failed. The log said “ChromaDB indexing error” when the error was an embeddings call against a nonexistent Service.
The pipeline returned status: completed even when there were errors. No alerting based on status fired. The metric that would have fired (matched_jobs stuck at zero) was not instrumented.
The pipeline image ran as :latest with no versioning, no SBOM, no reproducibility. When it started failing, it was impossible to know which Ollama client it had been built with.

Each of the three symptoms is the cry of a layer that was missing. The first asks for LLM-aware observability that distinguishes stages (layer 5). The second asks for the pipeline logic and the Prometheus metrics to behave like contracts (layer 5, SLI/SLO dimension). The third asks for GitOps with pinned images and an auditable SBOM (layer 6). And the whole incident, decommissioning a Service without knowing who consumes it, shouts dependency tracking (layer 7).

The rest of the post walks the seven layers through that lens: what each one solves, which OSS piece implements it in 2026, and which typical design decision breaks it.

Layer 1 — Gateway: the client SDK must not couple to the engine

What it has to solve. That the client sends POST /v1/chat/completions with the standard OpenAI SDK and never learns which engine (vLLM, SGLang, TensorRT-LLM), which specific model, which LoRA adapter or even which GPU pool is serving the request. Authentication, rate limit, routing by body.model and by tenant, header injection for tracing.

Canonical OSS piece in 2026. Envoy AI Gateway (Envoy with GenAI extensions: routing by model, token-based rate limit, fallback chains) or Cilium Gateway API with your own filters. For cases where the client wants multi-provider without distinguishing on-prem from SaaS, LiteLLM Proxy is the lightweight equivalent.

What breaks the layer. Exposing the inference engine endpoint directly. If clients call http://vllm-prod.svc:8000, any change of engine, model or pool forces you to touch the code of every app. The rule: the engine changes, the contract does not. The OpenAI SDK is standard; the routing behind the gateway is where design freedom lives.

Where jobhunter failed. There was no gateway. The app called ollama.ollama.svc:11434 directly. When Ollama died, there was no intermediate layer that could answer with a fallback, a retry against another pool, or at least a 503 error with a descriptive body.

Layer 2 — LLM inference: vLLM as the default choice, SGLang when prefix caching rules

What it has to solve. Serving tokens with throughput and latency under control: continuous batching, PagedAttention so the KV cache does not fragment the VRAM, FP8 so a 32B model fits with room to spare, multi-LoRA for per-tenant customisation without replicating the base, structured output for function calling with schema guarantees.

Canonical OSS piece in 2026. vLLM is the default choice: it covers the state of the art (PagedAttention, continuous batching, FlashAttention, FP8 quantisation, multi-LoRA, structured output, speculative decoding). SGLang comes in when the workload has high prefix caching (long chat with large system prompts, agents with repeated instructions), because its RadixAttention composes better than standard prefix caching. For very specialised inference with proprietary NVIDIA kernels, TensorRT-LLM, accepting the hardware lock-in.

What breaks the layer. Assuming a single model “does everything”. A 32B chat LLM does not serve /v1/embeddings; if you try, vLLM answers BadRequestError: "The model does not support Embeddings API". Assuming it did was one of the concrete wounds of the incident: the app expected an endpoint the model did not implement.

The design decision that costs more later. Serving the LLM with aggressive quantisation (INT4) without a quality eval calibrated for your corpus. INT4 with AWQ or GPTQ saves VRAM, but it degrades answers in technical or legal Spanish in a measurable way. The rule: any quantisation change goes through the same golden eval as a model change.

Layer 3 — Embeddings: separate from the LLM, fixed dimension, a life of its own

What it has to solve. Generating dense vectors for the RAG corpus and for retrieval queries. The embeddings model is a different thing from the chat LLM: different architecture (encoder, not decoder), different size (hundreds of millions of parameters, not tens of thousands of millions), different API (an /embeddings endpoint that takes text and returns a fixed-dimension vector).

Canonical OSS piece in 2026. Infinity or Hugging Face Text Embeddings Inference (TEI) to serve models of the bge-*, multilingual-e5-*, nomic-embed-* families with high throughput and multi-model support. OpenVINO Model Server when Intel hardware is available. sentence-transformers as a fallback embedded in the application itself when the corpus is small and the deployment is constrained.

What breaks the layer. Treating it as “the LLM does that too”. It does not if it is chat-only; and even when it does (some models have a dual endpoint), mixing chat and embeddings serving in the same process punishes the throughput of both. The physical separation is the design.

The technical fact people forget. The vector store and the embeddings model form an indivisible unit. Changing the model from multilingual-e5-large (1024 dim) to multilingual-e5-small (384 dim) is not a substitution: it is creating a new collection (my_corpus_v2) and re-embedding the whole corpus. If you upsert into the old collection, you hit a dim mismatch at runtime that takes the pod down. This seems obvious and is violated constantly, because the embeddings model is chosen once and then forgotten.

Layer 4 — Vector store + relational data + storage: what actually holds the state

What it has to solve. Persisting the vectors with efficient filters (tenant_id, created_at, source), persisting the relational metadata (users, configs, versioned prompts, traces), and persisting the model weights, the LoRA adapters, the datasets and the original corpora.

Canonical OSS pieces in 2026.

Qdrant for large collections (>200 k vectors), payload-aware filters, multi-tenancy via collection or via field.
pgvector for small collections with mandatory relational joins (wanting to do WHERE doc.author = ... AND vector <=> $1 in the same SQL).
PostgreSQL operated by CloudNativePG (CNPG) for the relational side: Barman Cloud backups, replication, connection via pooler.
MinIO for S3-compatible objects: bucket per tenant, cross-site replication, weights and adapters versioned by sha256.
Redis for queues, rate-limit counters and a cache of frequent retrievals.

What breaks the layer. Assuming the vector store is stateless ephemeral. It is exactly the opposite: it is the component where losing state costs the most. Without verified backups of the vector store, an index corruption forces you to re-embed the whole corpus, and on a corpus of millions of documents that is hours or days of GPU.

The design decision that pays later. Forgetting to version the collection by dimension and embeddings model. Suggested convention: my_corpus__embed-multie5l__1024d__v3. The name carries metadata; any change in any of the three attributes forces a new collection. It is ugly but it protects against an accidental upsert with the wrong dim.

Layer 5 — Observability: LLM-aware traces + infrastructure metrics + flow logs

What it has to solve. That any inference can be recovered from its trace_id, with all the gen_ai.* attributes (semantic conventions) plus your own attributes (tenant_id, adapter_id, priority_tier), a breakdown of latency (queue → prefill → decode → network), tokens consumed, tools invoked, exact model and adapter. And in parallel: Prometheus metrics from vLLM (vllm:num_requests_running, vllm:gpu_cache_usage_perc, vllm:prefix_cache_hit_rate), from the GPU (DCGM Exporter), from the network (Hubble flow logs with drops and NetworkPolicy enforcement).

Canonical OSS pieces in 2026.

OpenTelemetry Collector as the single transport for traces, metrics and logs, with OTLP receivers and separate exporters per destination.
Langfuse self-hosted for the LLM-aware side: tracing, prompt versioning, evals with LLM-as-judge (post and post).
VictoriaMetrics + Grafana for high-throughput TSDB metrics with long retention.
Hubble (Cilium) for L3/L4/L7 flow logs and NetworkPolicy visualisation.
DCGM Exporter for GPU metrics.

What breaks the layer. Wrapping the whole pipeline stage in a single try/except that labels the exception with the wrapper’s name. The operational rule: one try/except per stage, with the stage label in the message and a Prometheus metric with labels={"stage": "<name>"}. That way “ChromaDB indexing error” would never have been the log for an embeddings failure; it would have been “Embeddings call failed: ConnectError(ollama:11434)”.

Complementary rule. The pipeline returns status: completed if and only if there were no errors. With errors it returns completed_with_errors or failed, and the metric pipeline_errors_total{stage} is incremented. An alert based on increase(pipeline_errors_total[1h]) > 0 fires before the second failed run. Without this discipline, observability exists but does not warn.

Layer 6 — GitOps control plane: the only legitimate authority

What it has to solve. That the cluster state is the state declared in git. That any divergence between git and the cluster is visible and, in critical components, auto-reconciled or auto-alerted. That every deployed image has an immutable tag (sha digest or semver pin), an SBOM (Trivy) and traceability back to the commit that introduced it.

Canonical OSS pieces in 2026.

Flux (or ArgoCD) as the reconciler.
Forgejo (or Gitea, GitLab CE) as the self-hosted OSS forge.
cert-manager + Trust Manager for internal PKI.
External Secrets Operator + SOPS for encrypted versioned secrets.
Kyverno (or OPA Gatekeeper) for binding policies: deny :latest images, deny pods without a NetworkPolicy, deny Services without an owner label.
Trivy for SBOM and vulnerability scanning of images in the CI pipeline.

What breaks the layer. Images with a mutable tag (:latest, :main). Any kubectl edit in production that is not reflected in git. main branches with write permissions for humans without review. The rule: if a human can mutate the cluster without going through a signed commit, you do not have GitOps, you have somebody’s whiteboard.

How it applies to the incident. If the pipeline image had been pinned to a sha digest (registry.interno.local/jobhunter@sha256:9af2...), the team could have audited immediately which Ollama client it carried. With :latest, not even that.

Layer 7 — Dependency tracking: the layer the incident exposed

What it has to solve. Knowing who calls which Service, both declaratively (what the gitops repo says) and observed (what has been seen crossing the network over the last N days). And, on mature platforms, propagating that information as policy: if nobody declares and nobody observes traffic to the Service ollama.ollama.svc, decommissioning it is safe; if somebody declares it or uses it, decommissioning it opens a ticket.

Canonical OSS pieces in 2026.

Hubble (Cilium) for observed flow logs: hubble observe --to-namespace ollama --since 14d gives the list of source namespaces that have talked to Ollama in the last two weeks.
Otterize for intent-based policy: each Deployment declares “I need to talk to ollama-svc”, and the operator generates the corresponding NetworkPolicy and maintains a browsable catalogue of who is trying to talk to what.
manual kubectl-grep as a fallback: kubectl get deployments,cronjobs,statefulsets -A -o yaml | grep -E 'ollama[.-]' produces the declarative list.
NetworkPolicy as code reviewed in CI: every PR that touches a Service requires the associated policy to be kept or explicitly updated.

A pre-decom checklist that the incident suggests codifying as a CI hook:

SVC="ollama.ollama.svc.cluster.local"

# (a) declarative grep in the gitops repo
git -C $GITOPS_REPO grep -l "$SVC" || echo "OK declarative"

# (b) observed grep in Hubble (last 14 days)
hubble observe --to-fqdn "$SVC" --since 336h --output json \
 | jq -r '.source.namespace' | sort -u || echo "OK observed"

# (c) live grep in the cluster
kubectl get all -A -o yaml | grep -E "$SVC" || echo "OK live"

If all three return empty, the decom is safe. If any of them has content, there is downstream debt still open. The jobhunter incident is exactly what happens when this check does not exist: the team that decommissioned Ollama looked at the list of applications it knew depended on it directly; nobody looked at the list of those that depended on it silently.

The maths that matter: sizing the stack on 4×H100 SXM (320 GB)

The generic reference cluster for everything that follows: 4×H100 SXM 80 GB, NVLink between the four, 640 GB of system RAM, 2×NVMe NVMe-oF for local storage of weights and caches, redundant 25/100 GbE to the switch. Aggregate VRAM: 320 GB.

The VRAM budget is not free. A first reasonable split for a stack that serves one large general LLM, a medium code-specialised model, embeddings and a reranker, with room for multi-LoRA and for the KV cache:

Component	Reference model	Quant	Model weights	Reserved KV cache	Total VRAM
General LLM (TP=4)	70B-instruct	FP8 W8A8	70 GB	60 GB	130 GB
Code LLM (TP=2)	32B-coder	FP8 W8A8	32 GB	28 GB	60 GB
Embeddings	multilingual-e5-large	FP16	1.3 GB	n/a	8 GB (×2 replicas)
Reranker	bge-reranker-v2-m3	FP16	0.6 GB	n/a	4 GB
Multi-LoRA pool (over the general LLM)	up to 16 adapters	bf16	16 × 0.4 GB ≈ 6 GB	reuses the LLM’s KV	6 GB
Reserved for fragmentation + overhead					~30 GB
Total committed					~238 GB / 320 GB
Free headroom					~82 GB

The 26 % of free headroom is not waste: it is what lets the vLLM scheduler avoid preempting requests under moderate pressure, lets continuous batching group large batches without aborting, and lets a failover from the other site promote a standby without OOM.

Expected throughput, with the 70B general LLM in FP8 and tensor parallel 4, on an H100 SXM with continuous batching active and prefix caching at 35–55 % (typical in multi-turn chat with shared system prompts):

$$ \text{aggregate tokens/second} \approx 1500 \text{ to } 2500 $$

for concurrency between 32 and 64 requests, with sub-second TTFT P95 on short prompts (<2k tokens) and TPOT P95 around 40–60 ms/token as perceived by the client. These numbers are reasonable orders of magnitude, not guarantees: real throughput depends on the prompt mix, on whether speculative decoding (EAGLE-3) is active and burnt-in, and on the network cost between gateway and inference pods.

Embeddings throughput on two replicas of multilingual-e5-large with dynamic batching:

$$ \text{embeddings/second} \approx 3000 \text{ to } 6000 \quad (\text{optimal batch} \sim 64) $$

Enough to reindex a corpus of 1 million documents in roughly an hour, assuming chunks of 512 tokens and two chunks per document on average. For corpora of tens of millions of documents, re-embedding is done by delta via CDC over the source (covered in RAG corpus curation), not by full sweep.

Retrieval latency on Qdrant with HNSW (M=16, ef_construct=200) over a collection of 5 million 1024-dim vectors filtered by tenant_id:

$$ \text{P95 latency retrieve top-50} \approx 8 \text{ to } 25 \text{ ms} $$

Below the cost of cross-encoder reranking (bge-reranker-v2-m3 over top-50 = roughly 30–60 ms more), and below any call to the LLM. The bottleneck in a well sized RAG pipeline is never the vector store: it is the LLM’s decoding.

Final diagram: the complete connected stack

The solid lines are the request path: client → gateway → inference engine → vector store/embeddings → response. The dashed blue lines are telemetry: each component emits OTel to the collector, which routes traces to Langfuse, metrics to VictoriaMetrics and logs to Loki. The dashed red lines are reconciliation: the GitOps control plane keeps any layer in its declared state and warns of divergence.

The diagram is not decorative: every arrow is a stable contract between two layers. If one layer changes (vLLM → SGLang, multilingual-e5 → bge-m3, Qdrant → pgvector), the arrows hold. That stability of contracts is the architectural property that lets a team migrate components without breaking downstream apps.

Typical design decisions that break the stack

A short list of mistakes seen repeatedly in stacks that looked well designed on paper:

1. Coupling the client SDK to the inference engine. Dropping the gateway because “vLLM already speaks OpenAI-compatible” works on day one and hurts the day you have to add a fallback, a canary or a second model.

2. Sharing the LLM and embeddings endpoint. A qwen2.5-32b-Instruct is chat-only; BadRequestError: "The model does not support Embeddings API" is the cry of a design that confused the two layers.

3. Reusing the vector store collection when changing the embeddings model. Different dimensions do not allow upsert. Versioning the collection by (model, dim, version) is ugly but it saves the day of the change.

4. A try/except wrapping an entire pipeline with the wrapper’s label. The log lies because the label is lexical, not causal. Every stage in its own try/except with its own label and its own metric.

5. status: completed with errors. The pipeline has to distinguish completed, completed_with_errors and failed, and the alerting has to fire on the last two. Without this, observability exists in theory and does not warn in practice.

6. Images with a mutable tag. :latest and :main are not tags, they are aliases. Without a sha digest there is no reproducibility and no auditable SBOM.

7. Decommissioning a Service without a pre-decom check. The three-grep check (declarative + observed + live) takes two minutes and costs six days of incident when it is skipped.

8. Default limits.memory on pods that load models. A sidecar loading sentence-transformers + torch + tokenizer needs 2–4 GB; with limits.memory: 1Gi you hit OOM on the first pod restart, and sometimes with no alert because the liveness probe answers through another route.

All of them are variants of the same principle: the stack does not fail in its most expensive layer (inference, where nobody underestimates the cost), it fails in the cheap, boring layers (gateway, observability, GitOps, dependency tracking) where it is tempting to save.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), the suggested split into pods:

gpu-node-01 (4×H100 SXM, intra-node NVLink)
├── vllm-llm-general (TP=4) ~130 GB VRAM (4 GPUs)
└── (shares GPUs with the multi-LoRA pool on the same deployment)

gpu-node-02 (4×H100 SXM, second node)
├── vllm-llm-code (TP=2) ~60 GB VRAM (2 GPUs)
├── infinity-embeddings (×2) ~16 GB VRAM (shared on 1 GPU with optional MIG)
├── tei-reranker ~4 GB VRAM (cohabitant)
└── failover reserve ~120 GB VRAM free for canary / standby

Not everything fits comfortably on a single node; two nodes with two H100 SXM each would be enough for the conservative setup, and the rest of the cluster (CPU-bound: gateway, vector store, observability, control plane) runs on GPU-free nodes.

The operational rule: inference concentrates, the rest of the stack distributes. Concentrating inference maximises the use of NVLink (cross-GPU tensor parallel without going through PCIe); distributing the rest prevents an event on the GPU node from taking the control plane down with it.

An even more conservative configuration, for SMEs with a single 4×H100 SXM node as a starting point, serves the general LLM (TP=4) and embeddings/reranker cohabiting with MIG (Multi-Instance GPU to partition an H100 into hardware-isolated slices). The code LLM is deferred to a second phase. It is viable and cost-aware; what is not viable is doing without layers 5, 6 and 7.

What we have not covered (upcoming posts)

This post focuses on the static design of the stack. Some pieces still deserve an article of their own:

The multi-site active/standby plane: Cilium Cluster Mesh, cross-cluster Qdrant replication, realistic RTO/RPO, when active-active pays off and when it does not.
The continuous fine-tuning plane: how the LoRA pipeline closes the loop from production feedback to a promoted adapter, in the spirit of the retrain post.
The safety/guardrails plane: where Llama Guard fits, Presidio for PII and XGrammar for guaranteed structured output, connected to layer 1, the gateway.
The cost plane: gen_ai.usage.* instrumentation at tenant and model level, tokens/euro dashboards, GPU elasticity decisions via KEDA.
The compliance plane: how the stack maps to ENS High, NIS2 and ISO/IEC 42001 without turning the deployment into a compliance exercise that paralyses delivery.
Below the engine: the mini-series that opens the stack’s basement — the interconnect (NVLink/NCCL), the host (NUMA, hugepages, CPU isolation) and the RKE2 resource managers that pin each pod to the right NUMA node.

Each one falls into a different series on the blog and will be covered with the same discipline: a concrete piece, a justified decision, the typical mistake seen in practice.

References

The six-stage LLMOps pipeline — the general framework this stack operates in.
The OSS catalogue for LLMOps in six stages — a card per tool in the stack.
Anatomy of an LLM request in production — the same architecture seen from the perspective of an individual request.
OSS vs hyperscalers in LLMOps — a comparison with the AWS/Azure/GCP stacks.
LLM tracing with OpenTelemetry GenAI — the backbone of layer 5.
Multi-LoRA serving — the piece that gives per-tenant customisation without replicating the base.
RAG corpus curation and Reranker and hybrid retrieval — everything that goes into layers 3-4 so that RAG does not degrade.
Quantisation for LLM inference — the reason for the FP8 W8A8 in the sizing.

Relevant official documentation

vLLM Production Stack — docs.vllm.ai
SGLang RadixAttention — github.com/sgl-project/sglang
Envoy AI Gateway — aigateway.envoyproxy.io
Langfuse self-hosted — langfuse.com/docs/self-hosting
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
Hubble flow observability — docs.cilium.io/en/stable/observability/hubble
Otterize intent-based access — docs.otterize.com
Flux GitOps toolkit — fluxcd.io

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

Thu, 21 May 2026 07:15:00 +0200

TL;DR

Having an expensive GPU cluster and many different workloads that want to use it is not an infrastructure problem: it is an internal product problem. What separates “we have a cluster” from “we have an inference platform” are four layers the market consolidated in 2026: a gateway layer that centralises authentication, routing and policies (LiteLLM, Portkey, Kong AI Gateway); a GPU isolation model appropriate to the tenants’ profile (MIG hardware isolation for untrusted multi-tenant, MPS for processes from the same team, time-slicing only for dev); a quota and rate limiting system with budgets per tenant/team/project (LiteLLM does this in its core at team/user/api-key level with descriptive 429s); and a multi-tenant observability plane that enables real cost attribution (showback as an intermediate step, chargeback as the destination), per-tenant tracing and differentiated dashboards. Applied to a typical mid-scale GPU cluster (one node with 4-8 H100 SXM and NVLink, a common place to start in production), this translates into concrete decisions: with ~640 GB of aggregate VRAM across 8 GPUs and two typical production models (a large 70B+ model with tensor parallel and a replicated mid-sized model), the cluster serves between tens and low hundreds of simultaneous sessions depending on the mix; GPU isolation is usually resolved with MIG on smaller workloads and per-model dedication on large ones; and the platform’s success metric is effective utilisation, which in typical production sits at 30-40% and where the reasonable optimisation target is to push it to 60-70% without degrading SLAs.

This is the fifth post in the MLOps for LLMs series. It is the most operationally oriented one and cuts across several pipeline stages (Deploy + Observe plus cross-cutting concerns). The “you are here” marker flags the two active stages because the notion of a multi-tenant platform does not live in just one.

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

When a platform team buys expensive GPU hardware and starts building inference, the first version is almost always single-purpose: one model, one client, one target latency. It works. When the second team turns up asking for the same resource, that single purpose becomes internal politics: how many replicas do we give them? What do we do if the SLAs clash? Who pays for the tokens of team B’s experiment? And when the third one arrives, what was an SRE project becomes an internal product project.

The distinction is not technical, it is one of framing. A cluster is infrastructure. A platform is a service with customers, contracts and success metrics. The change of framing implies:

Identifiable customers (tenants), not anonymous users.
Contracts (latency SLA, guaranteed throughput, available models), not “whatever we manage”.
Success metrics that are not technical but product ones: adoption, satisfaction, cost per query per tenant, time to the first “hello world”.

This post walks through how that change of framing is carried out. It lands it on a mid-scale cluster (4-8 H100 SXM with NVLink in a single node), the usual configuration when starting with serious LLM inference; but the principles generalise to any topology, from a single node with two GPUs to multi-node clusters with InfiniBand.

The four layers of a multi-tenant inference platform

The canonical architecture that settled in 2026 has four layers that any serious multi-tenant platform implements, from the outside in:

Each layer solves a concrete problem. Let us take them one by one.

Layer 1 — AI Gateway: the single front door

The AI Gateway is the component your tenants see. It is an HTTP/gRPC API compatible with OpenAI (typically /v1/chat/completions, /v1/embeddings, /v1/models) that centralises everything that happens before touching the inference backends.

Why centralise

Without a gateway, tenants connect directly to vLLM or whatever the model is. Every change (rotating an endpoint, adding a model, changing credentials, applying a policy) means notifying every tenant. Every tenant has its own retry logic, its own logging, its own auth model. It is unworkable from the third customer onwards.

With a gateway, the change is made in one place. Tenants have a stable URL and a set of credentials; the rest is the gateway’s problem.

The three dominant options in 2026

LiteLLM is the most popular OSS option, Python-first, deployed as a proxy. It supports 100+ providers (OpenAI, Anthropic, Bedrock, self-hosted vLLM, Ollama, and so on) behind a unified OpenAI-compatible API. It has a native multi-tenant hierarchy with Organizations → Teams → Users → API Keys, each level with its own budget. The Apache 2.0 version covers the basics; RBAC, SSO, audit logs and team-level enforcement require the paid Enterprise version. Deployed on K8s with the official Helm chart.

Portkey is the most mature commercial / SaaS option. A single control plane that enforces budgets, quotas, permissions and compliance. Real-time spending tracking with alerting. RBAC, audit, workspaces and SSO included. The trade-off: dependency on an external service and a per-request pricing model.

Kong AI Gateway is the option for organisations that already have Kong as their API gateway. An AI plug-in on top of the existing Kong gateway, integrated with its plugin, consumer and rate-limit model. If your platform team already runs Kong, it is the lowest friction.

When to choose each one

Situation	Gateway
Pure OSS, self-hosted, Python-first team	LiteLLM
You need RBAC, SSO, audit log out of the box, and have budget	Portkey
You already run Kong as a corporate API gateway	Kong AI Gateway
Greenfield enterprise with strict compliance	Portkey (probably)
Mid-sized OSS-first company without regulated compliance	LiteLLM (typically)

The minimum the gateway has to do

Whichever option you pick, this is what any serious deployment must enforce:

Auth and identity: every request carries an API key resolvable to a tenant + user + team.
Routing by model: the tenant asks for model: "gpt-4o"; the gateway decides whether it goes to OpenAI, to Azure OpenAI, or to your vLLM with Qwen3 32B (a cheaper fallback), according to policy.
Rate limiting: RPS per tenant, TPM (tokens per minute), concurrency limits.
Caching of identical responses: 5-30% of RAG queries are repeats; caching saves latency and cost.
OTel emission: every call produces a span with gen_ai.* semantic conventions and tenant_id as an attribute. Covered in the Evals post and MCP observability.
Failover: if vLLM goes down, the gateway redirects to the OpenAI API. If OpenAI rate-limits, the gateway falls back to Anthropic. Configurable policy.

Example of a multi-tenant LiteLLM configuration

# litellm-config.yaml — simplified example
model_list:
 - model_name: llama-3-70b
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-llama3-70b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: qwen3-32b
 litellm_params:
 model: openai/qwen3-32b
 api_base: http://vllm-qwen3-32b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: gpt-4o
 litellm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

router_settings:
 routing_strategy: usage-based-routing-v2
 fallbacks:
 - llama-3-70b: [qwen3-32b, gpt-4o]  # if vLLM goes down, fall back to the external one

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL  # Postgres for budgets/keys

# Hierarchy: Organizations → Teams → Users → API Keys
# Created via the API, not in static YAML

Creating a team with a budget:

curl -X POST http://litellm/team/new \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_alias": "soporte-chat",
 "max_budget": 500, # 500 USD/month
 "budget_duration": "30d",
 "tpm_limit": 100000, # 100K tokens/min
 "rpm_limit": 1000, # 1000 requests/min
 "models": ["llama-3-70b", "qwen3-32b"] # access to these
 }'

And the team’s API key:

curl -X POST http://litellm/key/generate \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_id": "<team-id>",
 "duration": "30d",
 "metadata": {"environment": "production", "app": "support-bot"}
 }'

That API key is what the tenant uses. Every request made with it consumes from the team’s budget. When it runs out, LiteLLM returns HTTP 429 with a description.

Layer 2 — Policy & Quota Plane: what each tenant can do

The gateway is where enforcement happens. Policy is what gets enforced. Five axes of multi-tenant policy:

Technical quotas

TPM (tokens per minute): the hard consumption limit. For a Llama 3 70B at TP=5, ~3000 sustained output tokens/s = 180K aggregate TPM. If you have 10 tenants, assign 18K each as the ceiling.
RPS / RPM: load control, not consumption control. A 4K-token session counts as one request; so does a batch of 100 mini-completions. Useful against abuse.
Concurrency: how many simultaneous active requests per tenant. Important for latency SLAs: 100 RPS with concurrency=50 works out at 2 seconds per request.

Economic budgets

Monthly per tenant: a hard cap in USD.
Daily and hourly: soft caps to avoid a runaway in a single day.
Per project / API key: fine granularity within a single tenant.

LiteLLM has a max_budget field at every level of the hierarchy (organization, team, user, api key). Budgets are inherited and constrained downwards.

Model whitelist and blacklist

Tenants with critical workloads → only stable models (llama-3-70b, gpt-4o). Research tenants → access to experimental models as well.

Priority classes

Not all requests are equal. Three typical classes:

Guaranteed: workloads with an SLA, latency respected even under pressure.
Best-effort: normal workloads without a strict SLA.
Spot: batches that can wait, evictable if a guaranteed one arrives.

The Token Management in Multi-Tenant AI Inference Platforms paper (2026) formalises this with a model of token pools per priority class that has started to be adopted in production. It maintains guaranteed P99 latency for guaranteed workloads even under overload, with selective throttling of spot.

Admission control

Before accepting a request: is there capacity? If not, return 429 immediately instead of queueing and degrading everyone. It is the most underrated operational discipline. A cluster with admission control done properly has predictable latency; without it, catastrophic degradation when the peak arrives.

The typical pattern in 2026

# Conceptual policy for a "soporte-chat" tenant
tenant: soporte-chat
quotas:
 tpm: 50000
 rpm: 500
 max_concurrency: 30
budget:
 monthly_usd: 800
 alert_thresholds: [0.5, 0.8, 0.95] # warn when you get there
models_allowed:
 - llama-3-70b
 - qwen3-32b
priority: guaranteed
fallback_on_overload:
 - qwen3-32b  # if guaranteed fills up, fall back
 - gpt-4o-mini  # last resort, external model

Layer 3 — Isolation Plane: isolating workloads physically

This is the technically densest layer. You have a node with several H100 SXM GPUs interconnected by NVLink. How do you partition them between tenants?

MIG (Multi-Instance GPU) is the strongest isolation. It partitions the GPU into up to 7 instances with physically separate HBM memory and dedicated compute units (SMs). Tenants in different MIG instances cannot touch each other: one workload does not consume memory another needs, one does not degrade another’s throughput. Hardware isolation. Available on A100, H100, B100 and B200.

MPS (Multi-Process Service) is soft. Several processes share the GPU concurrently and NVIDIA distributes SMs according to usage. Good performance if all the processes are yours and you trust them. Worse for multi-tenant setups between customers who do not know each other, because a noisy process can degrade the others.

Time-slicing is the simplest: the GPU is assigned alternately, slot by slot, to different processes. Much worse latency (waits between slots); not recommended for production workloads with an SLA.

The choice for multi-tenant in 2026

According to the enterprise adoption survey: 80% use MIG for untrusted multi-tenant (different customers who do not know each other) and MPS for trusted environments (processes from the same team) where you want to maximise throughput. Time-slicing is only used in dev/staging so that every developer can touch a GPU without the cost of exclusivity.

An important limitation of MIG: it isolates compute and HBM memory, but the PCIe path remains shared. For PCIe-bound workloads (a lot of host↔device traffic), tenants in different MIG instances can still affect each other. For LLM inference the main path is HBM, so this is rarely a problem. But it is worth knowing.

MIG partitions on the H100

An H100 (80GB HBM3) can be partitioned into fixed profiles:

Profile	SM	Memory	Max instances per GPU
1g.10gb	14	10 GB	7
1g.20gb	14	20 GB	4
2g.20gb	28	20 GB	3
3g.40gb	42	40 GB	2
7g.80gb	98	80 GB	1 (the whole GPU)

For a mid-scale cluster with NVLink, MIG has a fundamental problem: when you partition with MIG, NVLink between GPUs is disabled. An H100 in MIG mode does not take part in multi-GPU tensor parallel. If you are going to serve a large model with tensor parallel (Llama 3 70B with TP=4 or TP=8, for instance), those GPUs have to be whole, without MIG.

That defines the architectural decision. There are two main approaches:

Approach A — A shared large model with quotas at the gateway

All the node’s GPUs serve a single large model with tensor parallel spanning the whole node. All tenants share that instance. Isolation happens at the gateway layer (quotas, rate limiting) and the policy layer (priority classes). The cluster’s kernel is one enormous vLLM instance with --max-num-seqs=128 or similar; internally, vLLM shares GPU time between the active requests with continuous batching.

Advantages: you use all the GPUs to the full, NVLink stays active, better KV cache utilisation. Disadvantages: soft isolation. A tenant that saturates does not degrade the others directly (vLLM batches), but it does compete for batch slots. You need serious priority classes.

Approach B — Dedicating GPUs per model / tenant

You split the GPUs into pools dedicated to different models. Examples on an 8-GPU node:

4 GPUs: a large 70B model with TP=4.
2 GPUs: a mid-sized 32B model replicated (2 independent instances) for tenants with a strict SLA.
2 GPUs: miscellaneous workloads (smaller models, experimentation).

Advantages: physical isolation between critical models / tenants. Disadvantages: worse aggregate utilisation; some GPUs idle while others saturate.

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

If you have small workloads (4B, 7B models), you can use MIG on 1-2 GPUs to serve them and dedicate the rest to tensor parallel for the large model. It combines strong isolation for the small workloads with full use of NVLink for the large model.

The operational choice: start with A, move up to C if you need to

In most deployments, Approach A (a shared large model plus quotas) is the right starting point. Utilisation is better, operation is simpler, and the gateway’s soft isolation works for reasonable workloads.

When there is a tenant with a strict SLA that cannot tolerate competing with others, you move to Approach B for that tenant in particular (dedicating GPUs to an instance of the model just for them), keeping the rest of the cluster shared.

Approach C is for when you have 10+ tenants with very heterogeneous profiles.

Isolation at the Kubernetes level

Independently of GPU isolation, pod-level isolation is applied in K8s:

Namespaces per tenant: tenant-soporte, tenant-legal, and so on.
ResourceQuotas and LimitRanges: CPU/memory limits per namespace.
NetworkPolicies: tenant A cannot talk to tenant B’s namespaces.
K8s PriorityClasses: classes with a numeric value defining preemption order if a more critical pod arrives.
PodDisruptionBudgets: how many pods of each deployment can go down simultaneously.

Layer 4 — Observability Plane: seeing what happens per tenant

The fourth layer: observability with a tenant dimension. Without it you cannot do cost attribution, you cannot debug incidents for a single tenant, you cannot show dashboards to stakeholders.

The four mandatory properties

1. tenant_id on every span. The AI gateway resolves the API key and attributes a tenant_id. That ID is propagated via params._meta or OTel headers to every downstream component (vLLM, retrieval, MCP servers, tools). Any span in any system carries that label. It is what lets you reconstruct tenant-specific traces.

2. Metrics labelled per tenant. gen_ai.usage.input_tokens{tenant="soporte-chat"} or equivalents. Prometheus, Grafana, groupable by tenant.

3. Real cost attribution. The sum of tokens × cost/token per tenant gives the cost. For self-hosted vLLM, the cost is per GPU hour plus the proportional share of tokens (you can compute an equivalent cost per 1k tokens).

4. An immutable audit log. Every API key used, every model invoked, every quota change, every budget exceeded. For compliance.

Showback vs chargeback

An important FinOps distinction that gained clarity in 2026:

Showback: visibility without consequence. “Support team, you have consumed 623 USD this month on LLMs”. Information, not an invoice. It lets you spot abuse without penalising anyone before the team understands what is going on.

Chargeback: the cost is charged to the team’s budget. When it runs out, it runs out. It changes behaviour.

The practice that works: 6-18 months on showback while tags are calibrated, misattributions are identified and teams are trained. Then chargeback, once the numbers are credible. Launching chargeback on day 1 when the costs are still dirty creates an immediate political fight; launching showback prepares the ground so that chargeback lands in an orderly way.

Only 14% of organisations have chargeback active according to a recent survey, which indicates this is still mostly showback in real production.

Tools

Kubecost: cost allocation per namespace, deployment and pod on Kubernetes. For the cost of the shared GPU, allocate proportionally to the tokens consumed per tenant.
Finout: a FinOps platform that combines cloud bills and LLM API costs in a unified view with virtual tagging.
Langfuse: already covered. Cost tracking per trace, groupable by user or session metadata.
LiteLLM native tracking: LiteLLM’s master DB keeps running spend per team, user and API key, accessible via the API or the UI.

The minimum multi-tenant dashboard

Any platform should have:

Summary per tenant: monthly spend, current RPS, TPM consumed, % of budget spent, active sessions.
Top users within each tenant (for detecting internal abuse).
p95 latency per tenant: SLA tracking.
429 / 503 errors: how many requests are being rate-limited or rejected because of overload.
Cost trend: the monthly trajectory with a projection.
Drift per tenant (from the post-tracing series): if a tenant starts getting worse results, raise an alert.

Sizing on mid-scale GPU clusters: concrete decisions

Let us get down to hardware. As a reference we take a node with N H100 SXM (between 4 and 8) with NVLink/NVSwitch, 80 GB HBM3 each. That gives between 320 GB and 640 GB of aggregate VRAM. Inter-GPU connectivity is 900 GB/s (NVLink 4) or 600 GB/s (NVLink 3) depending on generation. HBM bandwidth per GPU is 3.35 TB/s.

Default decisions

Start with Approach A: all the node’s GPUs serving a single large 70B model in BF16 with tensor parallel = N. Expected real capacity (computed for a standard 8-GPU HGX node as an example; it scales roughly linearly with N):

Model VRAM (70B BF16): ~140 GB (≈ 17.5 GB/GPU at TP=8).
vLLM overhead + activations VRAM: ~10 GB/GPU.
VRAM free for KV cache: ~52 GB/GPU. On an 8-GPU node that is ~416 GB aggregate; on a 4-GPU one, ~210 GB.
With --kv-cache-dtype=fp8 and a 70B GQA model: ~320 KB/token.
Aggregate cache capacity (8-GPU node): ~1.3M tokens to distribute between simultaneous sessions.

That translates into throughput and concurrency (indicative figures for an 8-GPU node):

Simultaneous sessions	Average context per session	Aggregate throughput (tokens/s)
32	16K	~5000
64	8K	~8000
128	4K	~12000

Typical latencies: TTFT ~150ms at low traffic, TPOT ~15-20 ms/tok. Under high concurrency, TTFT rises to ~500ms if the queue is saturated.

An example tenant scheme

A cluster with 4 tenants and a research pool:

Tenant	TPM cap	RPM cap	Concurrency	Budget	Priority	Models
Support chat	80K	800	50	1500 USD/month	Guaranteed	llama-3-70b, qwen3-32b
Legal RAG	30K	200	15	600 USD/month	Guaranteed	llama-3-70b
Code agent	50K	300	25	1200 USD/month	Best-effort	llama-3-70b, qwen-coder
Data extr. batch	40K	1000	40	400 USD/month	Spot	llama-3-70b, qwen3-32b
Research / notebooks	10K	100	5	200 USD/month	Spot	all

TPM total: 210K. Aggregate cluster capacity: ~180K sustained TPM. It is overcommitted by about 15%, assuming not every tenant hits the ceiling at the same time. That is normal and desirable; if they all do it at once, the priority classes degrade in an orderly way.

When to add hardware

Signals that the node has become too small:

Sustained p95 TTFT > 500 ms during peak hours → the queue is building up.
vllm:num_requests_waiting constantly > 20 → admission control starting to reject.
Sustained GPU utilisation > 80% at critical hours without dropping in quiet hours → there is no headroom.
429 rate on guaranteed tenants > 1% → the platform is breaking its SLA in production.

When several of these hold, the natural next step is to add another HGX node with internal NVLink and stand up a second vLLM instance of the same model. The gateway load-balances between the two instances. Aggregate throughput doubles; latency stays the same.

Common operational traps

A gateway without auth: a backdoor into the cluster

Your vLLM sits on a ClusterIP Service and the main app talks to it. Some tenant discovers the endpoint directly and hits it without going through the gateway. Quotas and costs are silently evaded. Use a strict NetworkPolicy: only the gateway can talk to the vLLM Services; the rest of the cluster cannot.

MIG and NVLink are incompatible

You enable MIG on a GPU thinking you will get isolation plus multi-GPU; you discover that MIG disables NVLink. Any large model with TP becomes unusable. Decide MIG vs NVLink globally per cluster, not per individual GPU.

Quotas pinned to the cluster’s ceiling

You add up the TPM of all the tenants and it comes to exactly the cluster’s capacity. When two tenants peak at the same time, both wait or one gets rejected. An overcommit of 10-20% is healthy (it assumes they do not all peak at once); more than that is dangerous.

No multi-tenant observability from day 1

You launch with quotas and isolation but without tenant_id on spans. Three months later, your CFO asks “how much does the support agent cost versus the legal one?” and you cannot answer. OTel with a mandatory tenant_id from the first version, even if there are no dashboards yet; having the data is worth more than having perfect dashboards with no data.

Showback that never reaches chargeback

You have been on showback for 18 months, the teams know the numbers, nobody changes behaviour. Without the pressure of real chargeback, the incentive dilutes. Set an explicit calendar for the transition to chargeback, with an owner and a deadline.

Non-whitelisted models eating the budget

A team discovers that LiteLLM has gpt-4o configured. They use it without permission. The budget burns on an external API when the idea was to use the cheap self-hosted one. Keep an explicit per-team whitelist of accessible models.

Badly calibrated priority classes

Everyone declares themselves “guaranteed”. At the first peak, there is nothing left to degrade and everything suffers. Priority classes only for critical cases, with justification. The majority should be best-effort.

No failover from the gateway

Your vLLM goes down. The gateway has no fallback configured and returns 503 to every tenant. Configure a fallback to another model, ideally an external one (OpenAI) for guaranteed workloads, even if it costs more per hour. Availability is worth more than cost per hour.

An operational roadmap for getting started

If you start from scratch with an empty GPU node, the minimum order is as follows. Each milestone is a day’s work with slack, not a tight one:

Day 1-2 — Base K8s infrastructure. NVIDIA GPU Operator + nvidia-device-plugin + dcgm-exporter + cluster-default NetworkPolicies. Validation: a basic pod with nvidia.com/gpu: 1 gets scheduled.

Day 3 — vLLM with a large model and tensor parallel across the whole node. The vLLM Production Stack Helm chart (or bare vLLM manifests). Model weights on a shared PVC (CephFS or NFS). Validation: a curl request against the internal Service responds.

Day 4 — AI Gateway: LiteLLM. Helm chart, Postgres for budgets, master key, a first model_list pointing at vLLM. Validation: an OpenAI-compatible request through LiteLLM responds with the same content as vLLM directly.

Day 5 — Basic multi-tenancy. Create teams, API keys, budgets, model whitelists. Test with two teams. Validation: the second team, using the model it does not have whitelisted, gets a 403.

Day 6 — Minimum observability. Prometheus + Grafana scraping vLLM and LiteLLM. A dashboard with TTFT, TPOT, throughput, num_requests_waiting and budget_consumed_per_team. Validation: visible in Grafana with real data.

Day 7-8 — Pilot customer. One real tenant (ideally a controlled internal one) starts using it. Measure real latencies, discover the first operational incidents.

Day 9-10 — Tuning. Adjust --max-num-seqs, --gpu-memory-utilization, priority classes and quotas based on what the pilot taught you.

Day 11-14 — Onboarding the second tenant plus iteration. Repeat. Every new tenant onboarded reveals new cases.

After two weeks you have an operational platform with two real tenants and data to decide whether it is ready for more. The line of progress from here on is horizontal (more tenants) until you saturate; from then on, vertical (more hardware).

What we have not covered (upcoming posts)

Continuous fine-tuning in production (post 6, already decided): LoRA/QLoRA/DPO, dataset curation, eval gates, A/B versioning with real traffic between model versions.
Constitutional AI and alignment at runtime: an option still on the table.
Edge LLMs: when an H100 cluster is too expensive for a specific workload, distilled models running on NPUs or consumer GPUs.
GPU networking deep dive: NCCL, InfiniBand, GPUDirect, RDMA. For multi-node clusters with cross-host tensor parallel.

References

Multi-tenancy and GPU isolation:

Multitenant GPU Infrastructure: 4 Powerful Design Rules — survey of enterprise patterns.
Run Multiple LLMs on One GPU: MIG, Time-Slicing, and MPS Guide (Spheron).
A Practical Guide to GPU Partitioning with MIG (Medium).
GPU Partitioning for AI Workloads: NVIDIA MIG with SUSE Virtualization (KubeCon EU 2026).
Predictable LLM Serving on GPU Clusters (arxiv 2508.20274).
Token Management in Multi-Tenant AI Inference Platforms (arxiv 2603.00356) — paper on priority + admission control.

AI Gateways:

Multi-tenant FinOps:

Cross-references:

Previous posts in series 4: MLOps for LLMs overview, RAG over Kafka, The six-stage pipeline, PostgreSQL + Qdrant.
Relevant posts from the inference series: vLLM on Kubernetes — the multi-GPU HGX node scenario we develop here. LLM operators on K8s — the vLLM Production Stack and OME that the gateway can direct.
Observability: Evals, MCP observability, eBPF + drift.

Hubble: eBPF network observability, state of the art 2026 and the new frontier with AI agents

Tue, 19 May 2026 06:00:00 +0200

TL;DR

Hubble is Cilium’s native network observability, built on the same eBPF programs Cilium uses for enforcement. It does not duplicate the datapath or instrument the kernel its own way: it listens to the hooks Cilium already has and produces structured flow logs with Kubernetes context included, covering pod, namespace, labels, service, policy verdict and L7 payload where applicable. It is what happens when someone decides that tcpdump with grep does not scale to 10,000 pods and builds a distributed system of their own (a Hubble server per node, plus Hubble Relay as the aggregator, plus a CLI and a UI) with practically zero overhead, because the capture was already happening. In 2026 it is at version 1.19.3 (April 2026), with Cilium 1.19 marking the project’s tenth anniversary; IP options tracing has arrived, along with filtering by encryption status, the drop event tagged with the exact NetworkPolicy that caused it (direct attribution), a stabilised field mask API, and the first wave of ML anomaly detection applied to flows for predictive security in IoT/5G clusters. And, most interesting of all for 2026: a new frontier appears where the same eBPF observes AI agents such as Claude Code, Gemini CLI and MCP agents, intercepting SSL/TLS and stdio without instrumenting the code, which turns the Cilium + Hubble + Tetragon + AgentSight stack into a complete set of tools for understanding what an agentic system does inside a cluster.

This article is part 3 of the eBPF series. Part 1: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack. Part 2: Tetragon: Cilium’s security cousin that sees every syscall in the kernel. Here we complete the observability quadrant: network with Hubble, process with Tetragon, AI agent with AgentSight.

The analogy: tcpdump that speaks Kubernetes

If you have administered networks over the last twenty years, tcpdump and Wireshark have been your daily bread. They capture packets on an interface, parse them, and let you filter with tcp.port == 443 and host 10.0.0.5. They work, they have been working since the 90s, and they are the first thing you open when something smells wrong.

Now stick tcpdump on a Kubernetes cluster with 10,000 pods. The problems show up in order:

One tcpdump session per node. You wanted to “see the traffic between the frontend and the API”; you need SSH to every node, tcpdump on every NIC, synchronised timestamps, manual aggregation.
There is no K8s context. You see a packet from 10.244.5.7 to 10.244.8.42. Which pod was that? Which namespace? Which label? You have to correlate with kubectl get pod -A -o wide every single time.
No L7 understanding. You see a POST to HTTPS, and you cannot know the method and path because it is encrypted on the wire. If there is mTLS between pods, worse.
High cost: full packet capture with a copy to userspace slows down the datapath. Under dense traffic, you notice it.

Hubble is tcpdump redesigned for all of that. It reuses the eBPF programs that are already processing every packet (Cilium puts them there for enforcement) and, while they take their allow/deny decision, they emit a flow event with all the context: source and destination pod identity, namespace, labels, protocol, verdict, and, if Cilium has done L7 parsing via Envoy, the HTTP method, path, status code, DNS query, Kafka topic. That event travels through a ringbuffer to userspace, it is received by the Hubble server that lives inside the node’s Cilium agent, and it is exposed over gRPC. A separate service, Hubble Relay, aggregates the streams from all nodes and gives you a single cluster-wide API. On top of that: a CLI (hubble) and a web UI with a real-time service graph.

Zero additional copies. Zero duplicated parsing. And the result is understandable by anyone who knows what a Pod is.

Architecture: four pieces visible from outside

Hubble is made up of four logical components, all optional depending on what you want to do:

1. Hubble Server (embedded in every Cilium agent)

It lives inside the Cilium agent process (it is not a separate binary). Each node exposes a gRPC endpoint locally on the Unix socket /var/run/cilium/hubble.sock. The server listens to the events the eBPF programs emit to the ringbuffer, enriches them with Kubernetes metadata (which the agent already has in memory), and makes them available to consumers.

Enabling it: --set hubble.enabled=true in Cilium’s Helm chart. By default, the server is only accessible locally; if you want to consume it from another node, it has to be exposed (which is what Hubble Relay does).

2. Hubble Relay (aggregator)

It is a separate Deployment (typically 1 replica, scalable) that connects to every Hubble server in the cluster and aggregates their streams into a single API. When your CLI or UI asks for “the last 1000 flows in the cluster”, the Relay collects them in parallel from all nodes and returns the union.

Enabling it: --set hubble.relay.enabled=true. Without the Relay you only see the traffic of the node you are connected to, which is useful for local debugging but not for a cluster-wide view.

3. Hubble CLI (`hubble`)

A Go binary that speaks gRPC with the Relay (or with a local Hubble server). It supports two main modes:

hubble observe: a real-time stream of flows, with very expressive filters (by namespace, pod, port, verdict, protocol, label).
hubble status: the state of the Hubble cluster (how many nodes connected, lag, flow rate).

And the equivalent of tcpdump’s pcap dump: hubble observe --output jsonpb > flows.json to process afterwards with jq or other tools.

4. Hubble UI

A web frontend that connects to Hubble Relay and shows:

A service graph in real time (which Pod talks to which Service, which protocols it uses, which verdict).
A filterable list of flows.
L7 details where there are any (HTTP method/path/status, DNS query/response).

Enabling it: --set hubble.ui.enabled=true. Useful for presentations to non-CLI teams; it does not replace the CLI for serious debugging.

What you see: the Hubble flow log from the inside

A Hubble flow in JSON format looks roughly like this (simplified):

{
 "time": "2026-05-19T03:12:45.182Z",
 "verdict": "FORWARDED",
 "source": {
 "ID": 5482,
 "identity": 24871,
 "namespace": "prod-api",
 "labels": ["app=checkout", "team=payments"],
 "pod_name": "checkout-7c9f-x8j2",
 "workloads": [{"name": "checkout", "kind": "Deployment"}]
 },
 "destination": {
 "ID": 12041,
 "identity": 18356,
 "namespace": "prod-db",
 "labels": ["app=postgres", "tier=primary"],
 "pod_name": "postgres-0"
 },
 "Type": "L3_L4",
 "l4": {
 "TCP": {
 "source_port": 41982,
 "destination_port": 5432,
 "flags": {"SYN": true}
 }
 },
 "node_name": "rke2-worker-03",
 "Summary": "TCP Flags: SYN"
}

When L7 parsing is active (via embedded Envoy or Hubble’s lightweight parser), the same flow adds:

"l7": {
 "type": "REQUEST",
 "http": {
 "code": 200,
 "method": "GET",
 "url": "/api/v1/cart/items",
 "protocol": "HTTP/1.1",
 "headers": [{"key": "user-agent", "value": "checkout/1.4.2"}]
 }
}

The protocols natively supported for L7 parsing:

HTTP/1.1 and HTTP/2 (including gRPC over HTTP/2).
DNS (queries and responses, with domains, types, response codes).
Kafka (topics, API keys).
TLS handshake (SNI, not the encrypted payload by default).
MySQL, Cassandra (with optional modules).

For HTTP and gRPC, Cilium can enable the embedded Envoy proxy for the flows you want to inspect (not all of them; it is selective via CiliumNetworkPolicy with L7 rules). Without Envoy there is lightweight parsing, but less detailed.

Verdict and drop attribution

Every flow has a verdict: FORWARDED, DROPPED, ERROR, AUDIT, REDIRECTED, TRACED, TRANSLATED. For the DROPPED case, Hubble includes a structured reason (drop_reason) and, since Cilium 1.19, the exact NetworkPolicy that caused it.

That last point changes day-to-day operations. Before, when a pod could not talk to another one, the debug flow was:

See the dropped flow in Hubble.
Look at every CiliumNetworkPolicy in the namespace.
Reason manually about which one, with which labels, is blocking it.

With Cilium 1.19’s attribution, the policy_match_info field tells you directly “it was dropped by the frontend-egress policy, rule 3”. You go from “Sherlock Holmes for 20 minutes” to “kubectl get -o yaml of that specific policy”.

Prometheus metrics and Grafana dashboards

Hubble also exposes aggregated metrics in Prometheus format, separate from the gRPC flow stream. Enabling it: --set hubble.metrics.enabled=true (Helm) plus a list of the set you want to export.

The usual metric groups:

flow: total flows by verdict, source/dest, protocol.
http: requests by method, response code, latency (histogram).
dns: queries, response codes, top-N domains.
tcp: handshakes, retransmissions, congestion window.
drop: drops by reason, with NetworkPolicy attribution.
port-distribution: histogram of active ports.
policy: hits by policy and verdict.

These metrics have rich K8s labels (source_workload, destination_workload, namespace, and so on) that make them pivotable in Grafana. There are prebuilt dashboards on Grafana Labs covering the common cases; importing one and having immediate visibility costs five minutes.

Cost: metrics with many K8s labels can blow up cardinality in Prometheus. For large clusters (>1,000 pods), it is worth reviewing which set you export and using drop rules in Prometheus to keep a lid on it.

Deployment: Helm on one screen

The canonical Cilium install with full Hubble:

# values.yaml
hubble:
 enabled: true
 metrics:
 enabled:
 - dns:query;ignoreAAAA
 - drop
 - tcp
 - flow
 - port-distribution
 - icmp
 - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
 serviceMonitor:
 enabled: true # auto-discovered by kube-prometheus-stack
 relay:
 enabled: true
 rollOutPods: true
 ui:
 enabled: true
 rollOutPods: true
 ingress:
 enabled: true
 className: cilium
 hosts:
 - hubble.example.local

And the install itself:

helm upgrade --install cilium cilium/cilium -n kube-system -f values.yaml

After installing, validate with cilium status (the Cilium CLI) that the Hubble section shows OK, and try your first flow with:

cilium hubble observe --namespace prod-api --pod checkout-7c9f-x8j2

State of the art in 2026

Cilium 1.19 was released in February 2026, marking the project’s tenth anniversary. Hubble reached version 1.19.3 on 22 April 2026. The relevant news:

Direct attribution of drops to a NetworkPolicy

Already covered above; it is probably the most valuable operational change in the release. Any dropped flow carries the name, namespace and specific rule of the policy responsible. It is also available via Prometheus metrics, which allows alerts of the form “policy X is dropping >N requests/second”.

Tracing with IP options

Hubble can now trace individual packets with IP options enabled. It is a mechanism similar to traceroute, but at L3: you put a mark on the packet and Cilium reports it every time the packet crosses a node or an eBPF decision. Useful for debugging multi-cluster paths, fabric mesh, or NetworkPolicies applied at different layers.

Filtering by encryption status

A new CLI flag: hubble observe --encryption-status=encrypted (or unencrypted). Useful for validating deployments with WireGuard or IPsec enabled pod-to-pod: you confirm that the traffic that should be encrypted is, and you spot regressions quickly.

Hubble field mask API stabilised

The field_mask lets you ask for only the parts of the flow you care about, hugely reducing bandwidth and processing when all you need is, say, source/dest and verdict. It used to be experimental; now it is stable and it is default-on in the CLI.

AI-driven anomaly detection (predictive security)

This is the most talked-about addition of 2026. Cilium 1.19 adds hooks so that an external consumer, typically an ML system, can process flows in streaming and detect statistical anomalies: pods that suddenly talk to new destinations, latency spikes on an API, odd DNS sequences. The detection part happens outside the Cilium agent (nobody wants heavy ML in the datapath), but Cilium exposes the flows with the pre-computed features the model needs. The published use cases focus on IoT and 5G, where traffic is high in volume and low in variety, ideal conditions for anomaly detection.

Scaling to 10,000+ pods

Cilium 1.19 has done serious work on scalability: Hubble Relay can now aggregate streams from hundreds of nodes without saturating; the default field_mask reduces inter-node bandwidth; and flows can be sampled under high load if your use is statistical analysis rather than forensic debugging.

Cilium 1.20 in development

Cilium 1.20 is in the development branch. The most relevant bits for Hubble:

Unification of preferIpv6: the hubble.preferIpv6 flag was deprecated in favour of the global preferIpv6 that applies to every Cilium component.
tetragon-python SDK: although it belongs to Tetragon, not Hubble, it sets a trend, namely eBPF policies written in Python instead of YAML. Hubble will probably follow a similar path.

The new frontier: eBPF and AI agents

So far, the classic Hubble content. But there is a 2026 twist worth covering, because it closes the loop with the other series on this blog.

When a Kubernetes cluster starts running AI agents, such as Claude Code, Gemini CLI, or LangGraph-based agents calling APIs and MCP servers, the observability problem changes shape. It is no longer enough to know “which pod talked to which pod” (that is Hubble) or “which process ran what” (that is Tetragon). You need to know:

Which external APIs the agent is calling and with which prompts.
Which MCP tools it is invoking, with which arguments.
How many tokens it consumes, which model it picks, how much it costs.
Whether the agent drifts from expected behaviour (out-of-policy queries, jailbreak attempts, secret leakage).

The traditional solutions, instrumenting the agent’s code with OpenTelemetry or parsing structured logs, do not work well when the agent is a third-party binary (Anthropic’s Claude Code, Google’s Gemini CLI) or when the MCP servers live in other languages with stdio as their transport.

AgentSight: zero-instrumentation for LLM agents

AgentSight (a project from the eunomia-bpf group, the same ecosystem behind several high-profile eBPF runtimes) attacks this problem with the same philosophy as Hubble: do not instrument; listen. It puts eBPF hooks at two critical points:

uprobes on SSL/TLS libraries (libssl, boringssl, rustls). It captures the plaintext before encryption on send and after decryption on recv. For an HTTP call to https://api.anthropic.com/v1/messages, AgentSight sees the full JSON of the prompt and the response without decrypting anything in transit, simply because it has reached the syscall level before the TLS layer does its work.
stdiocap BPF: it captures read, write and dup on a process’s stdin/stdout/stderr file descriptors. This is what makes it possible to observe MCP servers that speak stdio with their client, the usual pattern for local MCP servers. You capture the JSON-RPC going back and forth without either the client or the server knowing.

Reported overhead: <3% CPU, comparable to Hubble in its own regime.

How it fits with Hubble and Tetragon

The three complement each other cleanly:

Hubble tells you: “the agent’s pod opened a TCP connection to api.anthropic.com:443 with verdict ALLOW”.
Tetragon tells you: “the claude-code process with PID 1843 did a connect() to that IP” (plus the binary, the arguments, the pod namespace).
AgentSight tells you: “the HTTPS content of that connection was a prompt messages=[{role:'user', content:'analyze this repo and modify the firewall config'}] and the response included a tool call to read_file with argument /etc/passwd”.

It is the difference between flow, process and semantics. For a security team that wants to watch AI agents in production, all three are necessary. For anyone who wants to understand cost, all three are useful (Hubble for network latency, Tetragon for resource usage, AgentSight for tokens and chosen model).

Emerging use cases

The patterns consolidating in 2026:

Agent audit trail: recording every LLM call and every tool call for compliance, especially in regulated sectors.
Jailbreak and prompt injection detection: applying rules over the prompts captured by AgentSight (similar to Tetragon’s TracingPolicy, but over semantic content).
Cost accountability: seeing which team/agent consumes which tokens, without instrumenting.
Replay and debugging: reproducing an agent’s reasoning in production without asking it to run again (which is non-deterministic).

It is a young field, AgentSight is months old rather than years, but the “eBPF as zero-instrumentation observability” pattern is very clearly spreading beyond network and process. The coming year will see consolidation and, probably, native integration with Hubble.

Common Hubble use cases

Back to Hubble proper, the cases in which any organisation deploys it:

1. NetworkPolicy debugging

The classic use: “this pod cannot reach this Service”. Without Hubble, it meant SSH, tcpdump, comparing rules. With Hubble:

hubble observe --from-pod prod-api/checkout --to-pod prod-db/postgres --verdict DROPPED

If there are drops, you see the policy responsible (Cilium 1.19+). If there are no drops, the problem is not policy: it is DNS, routing or the target service.

2. Auditing inter-namespace communication

For compliance: validating that isolated namespaces are not communicating against what was declared.

hubble observe --from-namespace prod-payments --to-namespace 'NOT prod-db' --output json

3. Exfiltration detection

Outbound traffic to suspicious public destinations. Hubble detects them by IP/SNI, not by payload (which is encrypted):

hubble observe --to-fqdn 'NOT *.example.com' --to-fqdn 'NOT *.internal' --protocol tcp

Combined with Prometheus metrics and Grafana alerts, this gives you an exfiltration radar at zero cost.

4. Real-time service SLOs

The hubble:http:response_time_seconds metrics with source_workload, destination_workload, method and status_code labels allow SLO dashboards without needing to instrument the apps. The SRE sees the p95 latency of checkout → catalog directly.

5. Performance debugging

hubble:tcp:retransmissions_total and hubble:tcp:flags_total{flag="RST"} are early signals of network problems. A rise correlated with a latency regression points you at something in the infrastructure (NIC, switch, MTU) before you go down to investigate the app.

6. Post-incident forensics

Configuring Hubble to export flows to persistent storage (via OTLP to Tempo/Loki, or hubble observe --output jsonpb to S3) gives you forensic capability: if at T+30 days you detect that something was wrong at T, you can reconstruct the traffic.

Hubble and the rest of the observability stack

Hubble does not replace Prometheus, Loki, Tempo or Jaeger; it complements them:

Prometheus: receives Hubble’s aggregated metrics. Hubble exports a native Prometheus endpoint.
Loki: receives the structured flow logs if you export them as logs. Hubble has no native Loki exporter, but a Fluent Bit with an OTLP plugin or a custom one bridges it easily.
Tempo / Jaeger: the Cilium Operator has an OTLP exporter of flows in trace format (each HTTP/gRPC flow is a span). It integrates with Tempo or any other OTLP tracing backend.
Grafana: there are already public Hubble dashboards. Combined with Prometheus, Loki and Tempo, they give you a unified panel: metrics, logs, traces, all correlated by K8s labels.

The full-stack pile seen in production in 2026 (described in Building a Production eBPF Observability & Security Stack for Kubernetes in 2026):

Data: Cilium + Hubble (network), Tetragon (process), AgentSight (AI agent).
Pipeline: OTLP Collector as the single router.
Storage: Prometheus (metrics), Loki (logs), Tempo (traces).
UI: Grafana with domain-specific dashboards.
Alerting: AlertManager with rules over the Hubble + Tetragon metrics.

Comparison with the alternatives

System	Layer	Focus	Model
Hubble	L3-L7 network	K8s cluster with Cilium	eBPF, pull metrics, push flows gRPC
GKE Dataplane v2 obs	L3-L7 network	GKE managed	eBPF (Cilium-based, managed)
Tigera Calico Whisker	L3-L7 network	Cluster with Calico	eBPF + pcap, own UI
Tetragon	Process/syscall	K8s cluster	eBPF, push events gRPC
Falco	Process/syscall	K8s cluster	eBPF in userspace or kernel module
AgentSight	LLM agent	Agentic systems	eBPF (SSL uprobes + stdio)
Beyla (Grafana)	Application	App L7 + tracing	eBPF (uprobes on libs)
Pixie	App + system	Broad cluster visibility	eBPF + PXL script
Parca	CPU/mem profiling	Performance	eBPF profile sampling

If your CNI is Cilium, Hubble is the natural entry point and it does not compete with the rest: it complements them. For Calico clusters, Whisker is the equivalent. For profiling, Parca. For AI agents, AgentSight. The era of “one tool for everything” is passing: the modern stack combines several specialised pieces, all eBPF-based, exposed via OTLP.

Operational traps

Cardinality in Prometheus

Hubble metrics with all the K8s labels can blow up Prometheus. Measure cardinality before exporting everything. The most prolific metrics are flow and httpV2; start with drop and port-distribution and add the rest incrementally.

L7 visibility costs CPU

Enabling L7 parsing via embedded Envoy adds load to the agent (not to the base datapath, but yes to the node’s envoy proxy). For heavy HTTP traffic, measure. For flows where you only need L4, leave Envoy disabled.

Hubble Relay without HA

A single Relay replica is a single point of failure for the CLI and the UI (not for the local agent, which keeps working). For production, deploy with replicas: 2+ and topologySpreadConstraints so both do not go down together.

Encryption status reporting depends on the Cilium config

The new --encryption-status filter only gives real data if Cilium has encryption enabled (WireGuard or IPsec). Without that, everything is unencrypted and the filter contributes nothing.

UI exposed without auth

Hubble UI has no native auth. If you expose it through Ingress, there has to be authentication in front of it: OIDC via oauth2-proxy, mTLS, IP allowlist. It is not optional.

Storage that does not scale

If you keep flows for days for forensics, the volume is serious. For a cluster with 100 active pods, easily 1-10 GB/day of flow logs. Plan the lifecycle (compaction, retention, cold storage) before enabling it.

What we have not covered

Mesh / multi-cluster Hubble: aggregating flows from several Cilium clusters into a single Relay. Use case: a cross-cluster view, debugging a distributed service mesh.
hubble export: local persistence on the agent’s disk for forensics with low retention.
Anomaly detection with your own models: how to connect the gRPC stream to a custom ML consumer.
AgentSight in depth: the project deserves its own article. Next instalment.
eBPF for LLM serving profiling: how to measure vLLM’s TTFT, TPOT and throughput without instrumenting, using uprobes on libcudart.

References

Hubble and Cilium:

Hubble GitHub — main repo.
Hubble — Network Observability (Cilium docs) — official reference.
Cilium 1.19 release notes (InfoQ, feb 2026) — tenth anniversary and 1.19 news.
Cilium releases — all releases.
Hubble L7 HTTP Metrics — Grafana dashboard 19423 — ready to import.
End‑to‑end L7 Visibility with Cilium Hubble (cloud-cod.com, mar 2026).
Cilium Hubble Observability Platform Internal Analysis (Young-ju).
CiliumNetworkPolicy Python Hubble: L7 Visibility 2026 — one of the threads on the Python SDK.

State of the art 2026 and the full stack:

eBPF + AI agents:

AgentSight (GitHub eunomia-bpf) — the referenced project.
Harnessing eBPF for High‑Performance LLM Workloads (Klizo Solutions).

Cross-references:

Part 1: eBPF from zero to Cilium.
Part 2: Tetragon: Cilium’s security cousin.
LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators — where the traffic Hubble observes carries the prompts AgentSight inspects.

Tetragon: Cilium's security cousin that sees every syscall in the kernel

Tue, 19 May 2026 05:00:00 +0200

TL;DR

Tetragon is the runtime security and observability engine that the Cilium project published as a companion to the CNI. Its job is not to route packets, Cilium is already there for that, but to observe what happens inside the node’s processes in real time: which binary runs in each pod, which files it opens, which syscalls it invokes, which capabilities it asks for, which network connections it establishes, which kernel modules get loaded. It does this by loading eBPF programs into the kernel’s hook points (kprobes, tracepoints, uprobes, LSM hooks) and filtering the relevant events inside the kernel itself with a declarative language expressed as a CRD (TracingPolicy and TracingPolicyNamespaced). The result is a stream of events enriched with Kubernetes metadata (pod, namespace, labels) that costs less than 1% of CPU and, the thing that sets Tetragon apart from the competition, can block actions inside the kernel, killing the process with SIGKILL or overwriting a syscall’s return value, before they finish executing, with no race conditions. Against Falco (which parses syscalls in userspace, 5-10% overhead, detection-only), Tetragon is “cheaper and with enforcement”; against the bare kernel, it is “a declarative layer your operations colleague can read”. This article is the extensive introduction you need to take it on seriously: architecture, all the hooks and selectors, the operating modes, a guide to use cases (exec auditing, sensitive file access, container escape, cryptomining, rootkit detection, network observability) and the traps you see in production.

This article is part 2 of the eBPF series. Part 1, eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack, covered basic eBPF, the networking hooks (XDP, TC, sock_ops), how Cilium implements the datapath and the BGP Control Plane v2 CRDs. Here we take those same eBPF hooks and use them for something different: observing and, if needed, stopping what the cluster’s processes do.

The analogy: auditd on steroids in eBPF

Anyone who has spent a few years administering Linux has used auditd. It is the classic kernel subsystem for auditing syscalls: you configure a rule with auditctl (for instance, “monitor any open on /etc/shadow”) and the kernel sends events to a userspace daemon that persists them. It works, but it has two limitations that weigh heavily on modern Kubernetes clusters:

No Kubernetes context. auditd reports processes by PID and UID. Knowing which pod, which namespace, which image, which labels, the information that actually matters when responding to an incident, requires correlating afterwards with data from cri-o or containerd. It is operationally miserable.
No granular enforcement. auditd can generate events, but it cannot take the decision to kill the offending process before the syscall finishes. You leave that to a higher layer that reads the events, processes them and kills the process… if it gets there in time. A race by design.

Tetragon is auditd on steroids: the same conceptual ideas, hooks on syscalls and events to userspace, but implemented with modern eBPF, with filtering inside the kernel so you do not pay the cost of waking the daemon for every irrelevant syscall, with Kubernetes metadata injected by an agent that knows the cluster, and with actions executed inside the kernel itself without waiting for userspace to decide. If the rule says “kill any process that opens /etc/shadow from the prod namespace”, the decision is taken in the kernel kprobe and SIGKILL is delivered before the open completes. There is no race; there is no window between detection and action.

What Tetragon is, architecturally

Tetragon is an agent deployed as a DaemonSet (one pod per node) and a set of CRDs that define the policies to apply. The agent has four responsibilities:

Load eBPF programs into the hook points the active TracingPolicies demand.
Maintain a cache of Kubernetes metadata (pods, namespaces, labels) by reading the API server, so it can enrich every event with the right context.
Collect the events the eBPF programs emit (via ring buffers) and serialise them.
Export the events to configurable destinations: stdout JSON (typical in sidecars or log-collection agents), gRPC streaming (to consume them from Hubble or another consumer), a file, or Fluentd/Loki/SIEM.

The eBPF programs are not written by the user. Tetragon generates the bytecode from the TracingPolicies: it reads the declarative policy, decides which hooks to attack, which arguments to read from the kernel, which filters to apply inline and which actions to execute. The user only writes YAML.

This separation of declarative policy → generated eBPF bytecode is what makes Tetragon usable. Writing eBPF programs by hand is a specialist’s job; writing a TracingPolicy is the job of an SRE with a good example in front of them.

The two CRDs: TracingPolicy and TracingPolicyNamespaced

Tetragon exposes exactly two main CRDs:

TracingPolicy (cluster-scoped, cilium.io/v1alpha1): applies to the whole cluster, every node, every pod. Suitable for platform policies (the whole cluster must be audited the same way): for example, “log every execve in every pod” or “kill any process that tries to load a kernel module”.
TracingPolicyNamespaced (namespaced, same group and version): defined inside a namespace and applied only to the pods of that namespace. Suitable for policies with per-tenant autonomy: for example, “in the prod-payments namespace, kill any outbound connect to an IP outside the corporate range”.

Both CRDs have exactly the same internal structure. The difference is one of scope. The distinction was introduced precisely to allow multi-tenancy: the central security team defines cluster-wide TracingPolicy objects and each tenant can add its own with TracingPolicyNamespaced without needing cluster-admin permissions.

Anatomy of a TracingPolicy

A policy is made up of:

Hook points: which kernel events to observe.
Arguments: which data to read when the hook fires.
Selectors: filters evaluated inside the kernel to discard irrelevant events and, optionally, execute actions when they match.

Supported hook points

The official documentation lists five families of hook points:

kprobes: hook a kernel function. Syscalls are a particular case (when syscall: true) because their ABI differs from that of internal functions. Typical examples: sys_open, sys_openat, sys_connect, tcp_connect, do_mount, commit_creds. It is the most versatile hook and the one used 80% of the time.
tracepoints: hook static tracepoints compiled into the kernel. More stable across kernel versions than kprobes (they do not depend on function names that can change). Examples: syscalls/sys_enter_openat, sched/sched_process_exec.
uprobes: hook functions in userspace libraries or binaries. They serve to observe runtime primitives such as libssl functions, libc, the Go runtime, the JVM.
USDT tracepoints (User Statically Defined Tracepoints): static tracepoints defined in userspace binaries (like those MySQL, PostgreSQL and OpenJDK expose). Useful for application observability.
lsmHooks (LSM, Linux Security Module): hooks of the LSM subsystem, where SELinux/AppArmor plug in. They allow security policies very similar to traditional MAC but programmable with eBPF. Example: file_open, inode_unlink, socket_bind.

Arguments

Every hook can read the arguments of the function it is attached to. The supported types cover the primitives (int, uint64, bool, string, char_buf) and higher abstractions (file, path, sock, linux_binprm, capability, bpf_attr, cred). The high-level types are pointers to kernel structures that Tetragon knows how to parse; instead of having to read an offset, you write type: file and Tetragon gives you the full path of the descriptor’s file.

There is an important capability detail: on kernels ≥ 5.4, Tetragon can read up to 327,360 bytes of an argument if the large buffers flag is enabled. That is the difference between being able to audit execve with all of its long argv complete versus truncating them at 256 bytes and losing context.

Selectors: filtering in the kernel

Selectors are what make Tetragon cheap. Without them, every syscall on the node would fire an event that would travel kernel → ring buffer → agent → processed → filtered → discarded. With selectors, the filtering happens inside the eBPF program itself, in the kernel, and only the events that matter reach userspace.

The available selectors include:

matchArgs: filters by an argument’s value. Operators: Equal, NotEqual, Prefix, Postfix, GreaterThan, LessThan, Mask, SPort (source port), DPort (dest port), Family (AF_INET vs AF_INET6), State (socket state).
matchPIDs: filters by PID; useful for targeted observation.
matchBinaries: filters by the binary executing the syscall (absolute path), with Operator: In, NotIn, Prefix. Essential for avoiding noise from legitimate system processes.
matchNamespaces: filters by Linux namespace (Pid, Mnt, Net, Ipc, Cgroup, User). It allows policies specific to processes in containers versus the host.
matchCapabilities: filters by the process’s effective capabilities. Block actions requiring CAP_SYS_ADMIN that run in pods that should not have them.
matchNamespaceChanges: detects namespace changes (typical of container escape).
matchCapabilityChanges: detects capability changes (privilege escalation).
matchActions: the actions executed when all the preceding matchers hit.

Actions: from a simple Post to Sigkill

When a selector matches, an action is executed. Tetragon defines several:

Post: emits an event to userspace (the observability case). It supports rateLimit to avoid flooding the agent if the condition fires a thousand times per second. The syntax accepts 5 for 5 seconds, 5m for 5 minutes, 1h for 1 hour.
Sigkill: sends SIGKILL to the offending process from inside the kernel, before the syscall completes. This is the only thing that guarantees enforcement without a race.
Override: overwrites the syscall’s return value. Useful for making the process believe the syscall failed (Override -EPERM) without killing it. A better experience for apps that can handle errors; worse for apps that assume success.
Signal: sends any arbitrary signal (not just SIGKILL).
NoPost: does not emit an event, useful when combined with another selector that does emit and you only want the action without duplicated telemetry.
FollowFD and UnfollowFD: mark a file descriptor to follow its lifecycle and enrich subsequent events with the original path. Useful for auditing “which process read this file after opening it”.
TrackSock and UntrackSock: the same for sockets.
GetUrl and DnsLookup: make HTTP requests or DNS resolutions from the kernel. Designed for integrations with external systems (security webhooks, IP reputation lookups).
NotifyEnforcer and CleanupEnforcerNotification: communication with Tetragon’s enforcement subsystem for complex actions.

Modes: detection vs enforcement

A policy can be declared in one of two explicit modes:

enforce: enforcement actions (Sigkill, Override, Signal) are active. This is production.
monitoring: enforcement actions are ignored; only Post events are emitted. This is the “let us see what would happen if this were switched on” mode, critical for testing policies without breaking applications.

Control is done with the spec.options[].name: policy-mode field and value: monitoring or enforce. It is the best practice: start in monitoring, collect events for days, tune the selectors until no false positives come out, and only then switch to enforce.

Full example: blocking writes to `/etc/passwd` in the prod namespace

A realistic policy, commented line by line:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-passwd-write
 namespace: prod
spec:
 kprobes:
 - call: "fd_install"
 syscall: false # kernel function, not a syscall
 args:
 - index: 0
 type: int  # file descriptor
 - index: 1
 type: "file" # struct file*
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values:
 - "/etc/passwd"
 matchActions:
 - action: Sigkill  # kills the process
 rateLimit: "1m" # max once per minute
 options:
 - name: policy-mode
 value: enforce  # enforcement mode active

fd_install runs every time a process obtains a new file descriptor; the second argument is the file’s file struct. Tetragon knows how to resolve it to its absolute path. The matchArgs compares that path with /etc/passwd. If it matches, Sigkill kills the process before the descriptor even becomes usable. rateLimit: 1m stops the agent from saturating if a malicious application tries it in a loop.

Common use cases

Now to real use. These are the six cases that show up in any serious Tetragon deployment in 2026.

1. Execution auditing (`execve`)

The most basic use case and, even so, the most valuable. Which binaries are running in each pod? In a container that is supposed to run only nginx, suddenly seeing an sh or a wget is almost always a red flag.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-execve
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm  # struct linux_binprm*
 selectors:
 - matchActions:
 - action: Post  # events only, no enforcement

With no filters: every execve in the cluster generates an event. With K8s metadata, the event includes pod, namespace, container, image, labels. You turn it into a stream of events towards your SIEM and set up rules: “alert if I see sh, bash, nc, curl, wget or python running in any pod of the prod-api namespace”.

An enforcement variant: instead of Post, use matchBinaries with Operator: NotIn and a whitelist, plus Sigkill if the binary is not on the list. A very rigid box, but effective in pods that are “single-binary” (like a Go microservice).

2. Sensitive file access

Detecting (or blocking) reads and writes on critical files: /etc/shadow, /etc/kubernetes/, Secret mounts, /var/run/docker.sock, /proc/*/cmdline.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: sensitive-file-access
spec:
 kprobes:
 - call: "security_file_open" # LSM-ish via kprobe
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/etc/shadow"
 - "/var/run/secrets/"
 - "/var/run/docker.sock"
 matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/kubelet" # legitimate kubelet access
 matchActions:
 - action: Post

The matchBinaries: NotIn is important: kubelet and other legitimate node agents access these paths constantly and would generate noise. We filter those out in the kernel.

In enforcement: swap Post for Override with argError: -1 (EPERM), so that the open fails but the offending process stays alive and produces the error so that tracing tools pick it up.

3. Unauthorised outbound network connections

Detecting outbound connections to destinations outside the corporate range. Useful for spotting data exfiltration or malware command-and-control.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-external-egress
 namespace: prod
spec:
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr" # destination NOT in these CIDRs
 values:
 - "10.0.0.0/8"
 - "192.168.0.0/16"
 - "172.16.0.0/12"
 matchActions:
 - action: Sigkill
 options:
 - name: policy-mode
 value: enforce

This kills any attempt at a TCP connection to an IP that is not in the corporate CIDRs, in the prod namespace. Cilium already does this with NetworkPolicy, but Tetragon has two complementary advantages:

It gives you the process that attempted the connection, not just “pod X tried to connect to Y”.
It also works for exotic protocols where NetworkPolicy is less expressive.

4. Container escape detection

Container escape is the operational nightmare: a process inside a container manages to break the isolation (via a kernel exploit, a badly set capability, a misconfigured hostPath mount) and gain access to the host. Three typical signals:

A namespace change on the process (it leaves the container’s pid namespace).
setns or unshare in non-init processes.
Access to /proc/1/root or /dev/ from a container.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-container-escape
spec:
 kprobes:
 - call: "__x64_sys_setns"
 syscall: true
 args:
 - index: 0
 type: int
 - index: 1
 type: int
 selectors:
 - matchNamespaces:
 - namespace: Pid
 operator: NotIn
 values: ["host_ns"] # only processes NOT in the host pid namespace
 matchActions:
 - action: Sigkill
 - call: "__x64_sys_unshare"
 syscall: true
 args:
 - index: 0
 type: int
 selectors:
 - matchNamespaceChanges:
 - unshare: true
 matchActions:
 - action: Post

__x64_sys_setns targeting the host namespace from a process in a container is practically always malicious (legitimate containers do not need this at runtime).

5. Cryptomining

Mining processes have fairly recognisable profiles:

Processes with names like xmrig, minerd, cgminer, or legitimate processes such as python running CPU-intensive scripts.
Outbound connections to known mining pools (a public list of IPs and domains).
Anomalous use of /dev/cpu_dma_latency to avoid throttling.

A combined policy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-cryptomining
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm
 selectors:
 - matchArgs:
 - index: 4
 operator: "Postfix"
 values:
 - "/xmrig"
 - "/minerd"
 - "/cgminer"
 matchActions:
 - action: Sigkill
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "DPort"
 values: ["3333", "5555", "7777", "14444"] # common pool ports
 matchActions:
 - action: Post  # log only

The double policy: kill binaries with the classic names (belt) and log connections to pool ports (braces), so you also get an alert when someone renames xmrig to nginx-helper or uses exotic ports.

6. Detecting rootkits and suspicious kernel modules

Modern rootkits load kernel modules to patch functions (hide processes, hide network connections, hide files). Detecting them:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: kernel-module-load
spec:
 kprobes:
 - call: "do_init_module"
 syscall: false
 args:
 - index: 0
 type: "string"
 selectors:
 - matchActions:
 - action: Post
 - call: "security_kernel_read_file"
 syscall: false
 args:
 - index: 0
 type: "file"
 - index: 1
 type: int
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values: ["READING_MODULE"]
 matchActions:
 - action: Post

In a “well configured” Kubernetes cluster no new kernel modules get loaded at runtime; any event here is highly suspicious. Combine with enforcement on machines where modules should be fixed: Sigkill for whoever tries to load one.

Bonus: detecting third-party modification of eBPF maps

As a 2025-2026 trend: loading malicious eBPF programs to hide presence. Tetragon can observe the bpf syscall:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-bpf-syscalls
spec:
 kprobes:
 - call: "__x64_sys_bpf"
 syscall: true
 args:
 - index: 0
 type: int  # bpf cmd
 - index: 1
 type: bpf_attr
 selectors:
 - matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/cilium-agent"
 - "/usr/bin/tetragon"
 - "/usr/bin/bpftool"
 matchActions:
 - action: Post

Any process that is not one of the legitimate agents loading eBPF programs: you want to know about it.

Comparison with Falco

Falco is the closest competitor: it is also runtime security for Kubernetes, also originally based on eBPF (and before that on kernel modules), also with declarative policies. Three years ago they were functionally similar. In 2026 the divergence is clear:

Dimension	Tetragon	Falco
Philosophy	Cilium-native, integrated	Standalone, generic
Filtering	In the kernel (eBPF)	Parsing in userspace
Typical overhead	<1% CPU	5-10% CPU
Enforcement	Yes, in-kernel (Sigkill, Override)	Not native (depends on plugins)
Race conditions	No (action atomic with the syscall)	Yes, in enforcement via plugins
False positive rate	Low (K8s context in the kernel)	Higher (parsing afterwards)
Detection latency	5-26 ms	~10 ms (more constant)
Ecosystem maturity	Young, growing	Mature, plenty of material
Community	Cilium / CNCF Incubating	CNCF Graduated
Integrations	Hubble native	Falcosidekick, many
CRDs per policy	TracingPolicy / Namespaced	No CRDs; rules in YAML

When to choose each one:

Tetragon if you already use Cilium, if you need enforcement in the kernel (not detection-only), if overhead matters to you (workloads with many syscalls), and if you value the Hubble integration. Container escape and cryptomining detection is where its advantage over Falco shows up most.
Falco if you want a tool independent of the CNI, if you need the catalogue of ready-made rules and the wide community, if your cluster is not Cilium, or if the integration with SIEMs and ready-made notifiers (Falcosidekick) saves you work.
Both if the organisation is large: Falco for breadth of detection, Tetragon for surgical enforcement on critical workloads. That is what you see most in companies that have spent years with Falco and add Tetragon for specific cases.

Hubble + Tetragon: unified observability

Hubble is Cilium’s traffic observability component: it shows L3-L7 flow logs with zero impact on latency. Tetragon exposes its events over gRPC with the same format and vocabulary as Hubble, which allows you to:

See them in the same UI (Hubble UI shows Tetragon events as one more “layer”).
Correlate network events (Hubble) with process events (Tetragon) on the same timeline.
Export them together to Loki/Tempo/SIEM as a single stream.

The key synergy: Hubble tells you “this pod made a TCP connection to 1.2.3.4:80”. Tetragon tells you “this pod ran curl 1.2.3.4 from a bash binary launched by pid 1234”. Together they give you the full story.

Deployment and operation

Helm

The canonical install with Helm:

helm repo add cilium https://helm.cilium.io
helm install tetragon cilium/tetragon \
 --namespace kube-system \
 --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
 --set tetragon.exportFileMaxSizeMB=50 \
 --set tetragon.exportFileRotationInterval=24h

Tetragon deploys its DaemonSet, its CRDs and a service for Hubble. By default, it exposes the events on the agent pod’s stdout (any cluster log aggregator picks them up).

The `tetra` CLI

Tetragon ships a CLI called tetra for interactive investigation:

# real-time stream of the node's events
tetra getevents -o compact --pods <pod-name>

# structured JSON events to process with jq
tetra getevents -o json --since 5m --namespace prod

# view loaded policies
tetra tracingpolicy list

It is the best tool for debugging policies in monitoring before moving them to enforce.

Exporting to a SIEM

Three usual routes:

stdout + log aggregator: the agent writes JSON to stdout, Fluent Bit/Vector picks it up and sends it to Splunk/Datadog/Elastic. Simple, works with any logging infrastructure.
gRPC streaming: for low-latency integrations. A gRPC consumer of your own or Hubble Relay.
File + rotation: for air-gapped environments or regulatory audits that require persistent logs with controlled rotation.

Performance

Published benchmarks consistently place Tetragon at <1% of the node’s CPU under real workloads, compared with 5-10% for Falco under the same workloads. The reason is the architectural separation: Tetragon filters in the kernel and only carries the events that actually matter to userspace; Falco carries every syscall to userspace and filters there. On clusters with thousands of pods making hundreds of thousands of syscalls per second, the difference shows up on the bill.

Common operational traps

Permanent `monitoring`

The biggest trap is never reaching enforce: starting well with policies in monitoring, collecting events, tuning selectors, and then never switching. The result: you have detection without prevention, exactly what Falco gave you without paying Tetragon’s complexity. If you are going to use Tetragon, plan the road to enforce for the critical policies.

Selectors that are too lax

A policy with a single matchActions: Post and no specific selectors generates events for every syscall of the chosen hook. On a serious node that means tens of thousands per second, which fill logs, saturate exporters and hide the signal in the noise. Always start with strict filters (matchBinaries, matchNamespaces, matchPIDs) and open up once you know what you are looking for.

A kernel that is too old

Tetragon needs modern eBPF features. Kernels < 5.4 do not have the large buffer support (needed for execve with complete argv). Kernels < 5.10 do not have many of the LSM hooks. Kernel 5.15+ is the recommended minimum for production and 6.1+ to have every feature.

Hooks on renamed kernel functions

kprobes are tied to kernel function names that can change between versions. A policy that uses __x64_sys_setns can fail silently on a kernel where the function is called __do_sys_setns. Solutions: use static tracepoints where they are available (more stable), or keep alternative policies with several call entries for compatibility.

`Sigkill` in critical namespaces

Applying Sigkill to processes in kube-system or cilium-system can break the cluster. Enforcement policies must explicitly exclude the platform namespaces with matchNamespaces Operator: NotIn, or limit the scope with TracingPolicyNamespaced to make sure they do not act on systems they should not.

Missing `rateLimit`

A policy without a rateLimit on Post can suffer a catastrophic fan-out if the condition is met millions of times in an instant (typical in attack loops or application bugs). The agent saturates, events are lost, logs overflow. Always put a sensible rateLimit on detection policies, especially on high-frequency hooks such as tcp_connect or execve.

What we have not covered (upcoming articles)

eBPF LSM hooks in depth: how they relate to SELinux/AppArmor and when Tetragon is the right tool versus classic MAC.
Hubble UI with a Tetragon overlay: configuring the UI to show process observability and network observability on the same timeline.
Integration with OPA/Kyverno: how Tetragon complements admission policy engines (Kyverno validates at admission; Tetragon validates at runtime).
Forensics with eBPF: combining Tetragon with tools such as Beyla or OpenTelemetry to trace the full chain of an incident from the initial connection to the final syscall.

References

Official documentation (May 2026):

Tetragon — official site — entry point.
Tetragon docs — Tracing Policy — conceptual reference.
Tetragon docs — Hook points — kprobes, tracepoints, uprobes, LSM, USDT.
Tetragon docs — Selectors — full filter reference.
Tetragon docs — Enforcement Mode.
Tetragon docs — Kubernetes Identity Aware Policies — TracingPolicyNamespaced.
Tetragon GitHub.

Comparisons and analysis:

Comparative Analysis of eBPF-Based Runtime Security Monitoring (SciTePress paper 2025) — a benchmark with independent numbers.
Best eBPF Security Solutions for Kubernetes (ARMO, 2026) — Falco vs Tetragon vs KubeArmor comparison.
Falco vs. Tetragon (Asim Mirza, Medium) — analysis with use cases.
Deep Dive into Tetragon (A Security Engineer) — a walk through the inside of the agent.
Tetragon Series, Part 2: Enforcing Sensitive File Access (Medium).

Ecosystem:

Cilium Hubble — network observability.
Falco — official site — the other big name in the field.
KubeArmor — the third option, with AppArmor + eBPF.

Cross-references:

Part 1 of the series: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack — the eBPF fundamentals we take as read here.
Kubernetes with Cilium BGP: services reachable without Ingress — the starting point of the Cilium ecosystem on this blog.

eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack

Tue, 19 May 2026 04:30:00 +0200

TL;DR

eBPF is a sandboxed virtual machine inside the Linux kernel that runs verified code at well-defined hooks: kprobes, tracepoints, socket events, network drivers. Before eBPF, changing kernel behaviour meant recompiling it or loading an arbitrary module; with eBPF, you load a small program that passes a formal verifier and runs at native speed with memory safety. In networking, this translates into the fact that the packet does not have to travel through the traditional TCP/IP stack: an eBPF program in the NIC driver (XDP) can drop, forward or rewrite the packet before the kernel has done its first alloc; a program on cgroup hooks (sock_ops) can redirect connections to another socket without the packet ever leaving the machine. Cilium is the CNI that has taken this to its logical conclusion: it replaces kube-proxy with pure eBPF (O(1) instead of the O(N) of iptables), routes pod-to-pod without VXLAN where it can, evaluates Network Policies with BPF maps, and since 1.16 it has remade its BGP control plane with a new set of CRDs, CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, CiliumBGPNodeConfigOverride, which replace the monolithic CiliumBGPPeeringPolicy that is already deprecated. This post goes down the three layers (basic eBPF → eBPF networking → Cilium) and ends with the operational CRDs.

The analogy: signed plugins for the kernel

Think of the browser. Twenty years ago, extending a browser meant compiling a native binary and loading it: any extension could crash it, corrupt memory, read your bank cookies. Today, extensions are JavaScript in a sandbox with a manifest that declares permissions, a runtime that enforces the isolation and a store that signs the code. The extension does not touch the browser binary; it lives in a controlled world and can only talk to the browser through defined APIs. Result: massive extensibility with a bounded attack surface.

eBPF is exactly that for the Linux kernel. Loading a classic .ko module means loading native code with full access to kernel memory: one bug and the system is gone. eBPF is a bytecode VM with a static verifier, a controlled allocator, JIT to native hardware after passing the verifier, and a set of kernel “helpers” it can call. The eBPF program can read the kernel memory the verifier allows it to read, and only that. It cannot enter infinite loops (the verifier demands that it terminate). It cannot jump to arbitrary addresses. It cannot dereference pointers without having validated them first. And, most importantly: it is user code, loaded at runtime, executing inside the kernel at native speed.

The consequences are visible from miles away. Before, observing traffic in production meant patching the kernel or loading a risky module. Today, bpftrace -e 'tracepoint:net:net_dev_xmit { @[args->dev->name] = count(); }' gives you a histogram of packets per interface in three lines and zero downtime. Before, replacing iptables with something faster meant rewriting the netfilter subsystem. Today, Cilium loads 60 KB of eBPF bytecode into XDP and unseats iptables with a hash map.

Basic eBPF: what it is and what it is not

The origin and the scope

The name comes from Berkeley Packet Filter, a 1992 idea (McCanne and Jacobson) for filtering packets with a mini-bytecode that tcpdump used internally. In 2014, Alexei Starovoitov renamed it eBPF and extended it enormously: 11 64-bit registers instead of 2 32-bit ones, a 512-byte stack, maps as structures shared with userspace, JIT to native hardware, and a far more sophisticated formal verifier. From being a packet filter, it became a generic kernel extensibility mechanism.

Today eBPF is used for four things:

Networking: XDP, TC, cgroup hooks, socket ops, lightweight tunnels.
Observability: kprobes, uprobes, tracepoints, USDT. The basis of projects like bpftrace, bcc, Pixie, Parca.
Security: BPF LSM (Linux Security Module in eBPF), syscall blocking with seccomp-bpf. Falco, Tetragon, Tracee.
Scheduling: sched_ext (kernel 6.12+), process schedulers written entirely in eBPF. Still at a very early stage.

The VM

An eBPF program is compiled from C (or Rust, or Go with cilium/ebpf) to eBPF bytecode, not to x86/arm64 directly. The kernel loader (via the bpf() syscall) passes that bytecode through the verifier:

It reconstructs the control flow graph.
It performs static analysis of every possible path: every instruction has to be reachable, every memory access has to be within known bounds, every pointer has to have been validated.
It rejects loops without a known upper bound. Recent kernels admit bounded loops (the bpf_loop helper), but the counter is always finite.
It rejects calls to helpers or kfuncs that the hook’s program type does not allow.

If the verifier accepts the program, the JIT translates it to the host’s native code (x86, arm64, etc.) and it stays attached to its hook. From then on it runs every time the hook’s event occurs, with no context switch to userspace, with no syscall cost. Latencies on the order of hundreds of nanoseconds per invocation.

Maps: the bridge to userspace

An isolated eBPF program is not much use. What makes it useful are maps: data structures shared between the kernel program and userspace. There are several types:

BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_LRU_HASH: hash tables with or without LRU eviction.
BPF_MAP_TYPE_ARRAY, BPF_MAP_TYPE_PERCPU_ARRAY: arrays, optionally per-CPU to avoid contention.
BPF_MAP_TYPE_RINGBUF, BPF_MAP_TYPE_PERF_EVENT_ARRAY: channels for streaming events to userspace.
BPF_MAP_TYPE_PROG_ARRAY: arrays of eBPF programs for tail calls (chaining programs without returning to the base kernel).

Userspace reads and writes these maps via bpf() syscalls; the kernel program reads and writes them directly. It is the basis of any eBPF system: the kernel program collects data into a map, the userspace daemon reads it. Cilium does exactly this: the userland agent (Go) manages the policy and translates it into map entries; the eBPF programs living in XDP/TC read the maps and apply the decisions.

CO-RE: compile once, run on any kernel

A classic nightmare of kernel modules: they are tied to the exact kernel version they were compiled against. Distributing a precompiled module for a fleet of machines with different distros was impossible.

eBPF solves this with CO-RE (Compile Once, Run Everywhere): the bytecode includes relocations that the loader resolves on each specific kernel by consulting BTF (BPF Type Format), a representation of the kernel’s struct layouts that the kernel itself publishes. Result: a single eBPF binary works on kernels 5.10, 5.15, 6.1 and 6.8 without recompiling, because the loader adjusts the struct access offsets at runtime.

This is what has allowed productive eBPF distributions to exist. Without CO-RE, every kernel would be a porting project.

eBPF in networking: the hooks that matter

Inside the Linux network subsystem, eBPF has several hooks. The ones relevant to CNIs:

XDP — eXpress Data Path

XDP is the earliest hook: it runs in the NIC driver, before the packet enters the kernel proper. There is no sk_buff (the struct the rest of the kernel uses to represent packets); there is only a pointer to a RAM buffer with the received bytes.

The actions an XDP program can return:

XDP_DROP: discard the packet immediately. The driver drops it and frees the buffer. Cost: nanoseconds. Use case: DDoS mitigation. Cloudflare processed more than 8 million packets/second per CPU with XDP for dropping SYN floods.
XDP_PASS: let the packet continue to the normal kernel. It becomes an sk_buff and enters the traditional stack.
XDP_TX: send it back out on the same interface after possible modifications. Useful for L4 load balancers that rewrite the destination and return it.
XDP_REDIRECT: send the packet to another interface or to a map (to forward to userspace via AF_XDP, or to another NIC, or to a pod’s veth).
XDP_ABORTED: error (increments a counter, drops).

Real use cases:

Cloudflare L3 DDoS protection: XDP rules that drop millions of packets/s.
Facebook Katran: an L4 load balancer that rewrites the destination IP and returns it on the same interface. Handles 10× more connections per server than classic IPVS.
Cilium XDP acceleration: Service load balancing at the lowest layer possible.

TC (Traffic Control) — clsact with BPF

XDP is very fast but limited: the packet does not have an sk_buff yet and many decisions (conntrack, NAT, encapsulation with metadata) are easier when it does. The TC clsact with BPF hook runs after building the sk_buff but before the routing and netfilter decisions. Actions:

TC_ACT_OK: the packet continues through the stack.
TC_ACT_SHOT: drop.
TC_ACT_REDIRECT: redirect to another interface.
TC_ACT_PIPE, TC_ACT_STOLEN: pipeline control for combining with other qdiscs.

Use cases:

Stateful network policy: Cilium evaluates L3-L7 policies in TC with the full sk_buff and conntrack available.
Marking and QoS: traffic marking so the scheduler applies priorities.
Overlay encapsulation: adding VXLAN/Geneve headers when the mode is tunnel.

XDP and TC combine: XDP for the cheap and early stuff (DDoS, simple LB), TC for what needs an skb and state.

Cgroup hooks: sock_ops and CGROUP_SOCK_ADDR

The most radical conceptual step: hooks that are not in the network layer but in the socket layer. Relevant types:

BPF_PROG_TYPE_CGROUP_SOCK_ADDR: invoked when a process in a cgroup calls connect(), bind(), sendto(). The eBPF program can rewrite the destination address before the connection goes out. This is what lets Cilium do Service load balancing without the packet entering the network stack: if the client tries to connect to 10.96.0.1:443 (a ClusterIP), an eBPF program on this hook rewrites the destination to the real IP of the backend pod before the syscall continues.
BPF_PROG_TYPE_SOCK_OPS: invoked on TCP events (creation, established, retransmission). It allows tuning socket parameters at runtime and, most importantly, pairing local sockets via bpf_sk_assign to shortcut without the packet travelling over the network.

This is the “third layer” of the bypass: it is not just faster, it is conceptually different. The packet is not built, not serialised, does not traverse the IP layer or the TCP layer. It is the difference between speeding up a road and discovering that for some journeys you do not need to take the car at all.

The long road: what the traditional TCP/IP stack looks like

To appreciate what eBPF saves, it is worth tracing a packet’s full journey through the Linux stack. Take the case “packet arrives on a NIC, goes to a local process”:

NIC (DMA into the driver's ring buffer)
↓
driver: napi_schedule, poll, allocates sk_buff
↓
[XDP hook] ← if there is an XDP program, it is decided here
↓
netif_receive_skb
↓
__netif_receive_skb_core
↓
[TC ingress clsact + BPF] ← if there is a TC ingress program
↓
packet_type handlers (IP, ARP...)
↓
ip_rcv → ip_rcv_core
↓
[netfilter NF_INET_PRE_ROUTING] ← iptables PREROUTING
↓
routing decision (FIB lookup)
↓
[netfilter NF_INET_LOCAL_IN] or [NF_INET_FORWARD]
↓
tcp_v4_rcv → tcp_v4_do_rcv
↓
tcp_rcv_established
↓
sk_data_ready
↓
process reads with recv()/read()

Every arrow is a function call with a measurable cost. Every netfilter hook walks all the registered iptables/nftables rules. With kube-proxy in iptables mode and 5,000 Services × 10 endpoints each, there are on the order of 150,000 rules evaluated sequentially at NF_INET_PRE_ROUTING. Published benchmarks show latencies of tens of microseconds per packet in large Kubernetes clusters in the netfilter step alone, before the application receives anything.

And that is the normal path. On the way out the same thing happens in reverse: tcp_sendmsg → ip_output → NF_INET_LOCAL_OUT → routing → NF_INET_POSTROUTING → dev_queue_xmit → driver → NIC.

How Cilium skips this stack

Cilium does not eliminate the TCP/IP stack; it is still there for the cases that need it. What it does is shortcuts at the points where it hurts.

Shortcut 1 — XDP for the Service datapath

For a cluster with 5,000 Services, kube-proxy iptables has an O(N) cost in evaluating rules (even with iptables-restore --noflush and tricks, it is still linear in the number of chains the packet traverses).

Cilium replaces it like this:

Every Service and its endpoints live in an eBPF hash map.
When a packet comes in destined for a ClusterIP, Cilium’s XDP program does an O(1) lookup in that map and obtains the backend endpoint.
It rewrites the destination and does XDP_TX (returns it on the same interface towards the backend) or XDP_REDIRECT (sends it to the corresponding local pod’s veth).

This means the cost does not grow with the number of Services. 100 Services or 100,000, constant lookup in the map. Published benchmarks show latency reductions of 30-50% in clusters with many Services compared with kube-proxy iptables, and of an order of magnitude compared with IPVS in some cases.

Shortcut 2 — socket-LB: the packet is never built

Cilium 1.6+ introduced socket-level load balancing, based on cgroup hooks. It works like this:

When a pod calls connect(10.96.0.1:443) (a Service’s ClusterIP), the syscall enters the kernel.
Before the kernel builds anything network-related, an eBPF program on CGROUP_SOCK_ADDR/connect4 intercepts it and rewrites the destination address to the real IP of the backend pod.
The kernel carries on with the connect as if the client had written 10.0.0.42:8080 directly.

Why does it matter? Because when the backend pod is on the same node, this shortcut turns a call that would have involved:

syscall connect → kernel stack → veth → bridge → veth → kernel stack → syscall accept

into:

syscall connect (with the destination rewritten) → direct loopback

The TCP/IP stack is literally avoided. There is no encapsulated packet, no journey through veth pairs, no netfilter. L7 pod-to-pod latencies on the same node drop to local communication levels (~5-15 µs instead of ~30-50 µs for services with kube-proxy iptables and traditional veth).

Shortcut 3 — pod-to-pod direct routing

The traditional overlay mode (Flannel, Calico VXLAN) encapsulates every pod-to-pod packet in VXLAN/Geneve. Every packet carries an extra 50-byte header, requires encap/decap, and consumes MTU.

Cilium supports direct routing: the pod CIDRs are advertised to the underlying fabric (with BGP, which is where the control plane we will look at comes in) and the physical routers route the pod-to-pod packets without encapsulating. The packet leaves a pod with its original IP as source and the destination pod’s IP as dest, the node’s NIC hands it to the network, the network routes it, it arrives at the destination node and is delivered to the pod. Zero encap, full MTU, minimal latency.

Cilium does this via eBPF programs in TC that rewrite the necessary headers and decide whether the packet goes via encap or direct according to the policy configured per node.

Shortcut 4 — Network Policy in TC with maps

Network Policies in classic CNIs are usually translated into iptables rules, another factor that explodes linearly. Cilium evaluates them in eBPF programs that read identity maps: every workload has a numeric identifier computed from its labels, and the policy is a map (src_identity, dst_identity, port, proto) → allow|deny. One hash map lookup per packet.

This also enables Cilium’s L7 policies (HTTP, gRPC, Kafka filtering): the eBPF program recognises the L7 handshake, selectively redirects to the embedded Envoy proxy (which lives as a sidecar of the datapath, not as a pod sidecar) and only on that subset does it pay the cost of the L7 proxy. All the L3/L4 traffic stays on the eBPF fast path.

Cilium: the architecture

Cilium combines two planes:

Agent (Go): lives as a DaemonSet on every node. It is the “slow” part: it translates the intent expressed in CRDs (CiliumNetworkPolicy, CiliumBGPClusterConfig, etc.) into entries in eBPF maps. It talks to the Kubernetes API server to discover endpoints, services, pods. It embeds a GoBGP for the BGP control plane. It embeds an Envoy for L7 policies.
Datapath (eBPF): the programs loaded into XDP, TC, cgroup hooks. They are the “fast” part: they see every packet, read the maps the agent maintains, and decide in nanoseconds.

This separation is what makes Cilium operationally comfortable: the intent is expressed in YAML, the agent materialises it into maps, the maps are read by the datapath. If the agent goes down temporarily, the datapath keeps working with the last loaded configuration. As in any well-built control/data plane system.

BGP Control Plane v2: the CRDs you have to know

Cilium has had BGP support for several years. The first version used a single monolithic CRD, CiliumBGPPeeringPolicy, which mixed node configuration, peers, timers and advertisements into a single object. Since Cilium 1.16 there is BGP Control Plane v2, which breaks that configuration into separate CRDs with clear responsibilities. CiliumBGPPeeringPolicy (the cilium.io/v2alpha1 API) is deprecated and migration warnings appear in the operator logs if you still use it.

The new CRDs (the cilium.io/v2 API):

1. `CiliumBGPClusterConfig`

Defines BGP instances and the peers they connect to, from the cluster’s perspective. Which nodes apply this configuration is selected with a nodeSelector.

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
 name: cilium-bgp-cluster
spec:
 nodeSelector:
 matchLabels:
 bgp-policy: rack-1  # only nodes with this label
 bgpInstances:
 - name: instance-65000
 localASN: 65000
 peers:
 - name: top-of-rack-1a
 peerASN: 64512
 peerAddress: 10.0.1.1
 peerConfigRef:
 name: tor-shared-config  # → reference to CiliumBGPPeerConfig
 - name: top-of-rack-1b
 peerASN: 64512
 peerAddress: 10.0.1.2
 peerConfigRef:
 name: tor-shared-config

A BGP instance is the abstraction “this node takes part in BGP with this local ASN and these peers”. Several can coexist on the same node (multi-instance for multi-VRF).

2. `CiliumBGPPeerConfig`

Defines the shared parameters of the peering: timers, address families, transport, MD5 password, graceful restart, etc. It is referenced from CiliumBGPClusterConfig via peerConfigRef. This avoids repeating the same configuration for every peer when there are dozens of them.

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
 name: tor-shared-config
spec:
 timers:
 holdTimeSeconds: 30
 keepAliveTimeSeconds: 10
 connectRetryTimeSeconds: 5
 gracefulRestart:
 enabled: true
 restartTimeSeconds: 120
 families:
 - afi: ipv4
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp  # → binds to CiliumBGPAdvertisement
 - afi: ipv6
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp
 authentication:
 password:
 name: bgp-md5-secret  # Secret with the MD5 password
 key: password

A single CiliumBGPPeerConfig can be referenced by many different peers. You change timers or families in one place.

3. `CiliumBGPAdvertisement`

Declares which prefixes are advertised: the node’s pod CIDRs, the ClusterIPs and ExternalIPs of Services, the IPs assigned by CiliumLoadBalancerIPPool for type=LoadBalancer Services. They are bound to CiliumBGPPeerConfig via labels.

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
 name: services-and-pods
 labels:
 advertise: bgp  # ← the label the PeerConfig matches
spec:
 advertisements:
 - advertisementType: PodCIDR  # advertises the node's pod CIDR
 attributes:
 communities:
 standard:
 - "65000:100"
 - advertisementType: Service  # advertises ClusterIPs / LoadBalancer IPs
 service:
 addresses:
 - LoadBalancerIP
 - ClusterIP
 - ExternalIP
 selector:
 matchLabels:
 bgp-advertise: "true" # only Services with this label
 attributes:
 communities:
 standard:
 - "65000:200"
 localPreference: 200

The granularity is very fine: you can advertise different types of prefixes with different BGP communities, different local-preference, different path attributes, and filter Services with label selectors. This was literally impossible with CiliumBGPPeeringPolicy v1.

4. `CiliumBGPNodeConfig` (auto-generated)

This CRD is not configured by hand. The Cilium operator generates one per node from the CiliumBGPClusterConfig that applies to that node. It is the materialised per-node state that each node’s agent reads to bring up its peerings. If you want to see what BGP configuration is actually running on a node, kubectl get ciliumbgpnodeconfig <nodename> -o yaml shows you.

5. `CiliumBGPNodeConfigOverride`

Optional. It allows overriding the generated configuration for a specific node when you need something non-standard. Use cases:

Pinning the BGP router-id to a specific IP (useful when the node has several interfaces).
Specifying the peer’s local address when there are several outgoing interfaces.
Changing timers only for one problematic node.

apiVersion: cilium.io/v2
kind: CiliumBGPNodeConfigOverride
metadata:
 name: node-rack1-master01  # the name must match the node's
spec:
 bgpInstances:
 - name: instance-65000
 routerID: 10.0.1.10 # router-id override
 peers:
 - name: top-of-rack-1a
 localAddress: 10.0.1.10 # specific local interface

Relationship diagram

CiliumLoadBalancerIPPool: it complements, it is not BGP

Although it is not strictly a BGP CRD, it is worth mentioning: CiliumLoadBalancerIPPool is the CRD that supplies IPs to type=LoadBalancer Services. It defines a range (10.20.0.0/24, for example) that Cilium assigns automatically to LoadBalancer Services. Combined with a CiliumBGPAdvertisement that advertises LoadBalancerIP, it gives the complete cycle: new Service → IP assigned from the pool → BGP advertisement to the routers → IP routable from the corporate network, with no external balancer.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
 name: lb-pool-rack1
spec:
 blocks:
 - start: "10.20.0.10"
 stop: "10.20.0.250"
 serviceSelector:
 matchLabels:
 lb-pool: rack1

Full manifest: pod CIDRs + LoadBalancer Services advertised to a redundant ToR pair

A realistic example of a cluster with two top-of-rack switches as BGP peers, both in the same AS (64512), Cilium in AS 65000:

# 1. CiliumBGPPeerConfig — shared config for both ToRs
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
 name: tor-peers
spec:
 timers:
 holdTimeSeconds: 30
 keepAliveTimeSeconds: 10
 gracefulRestart:
 enabled: true
 restartTimeSeconds: 120
 families:
 - afi: ipv4
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp

# 2. CiliumBGPAdvertisement — what gets advertised
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
 name: pods-and-lb
 labels:
 advertise: bgp
spec:
 advertisements:
 - advertisementType: PodCIDR
 - advertisementType: Service
 service:
 addresses: [LoadBalancerIP]
 selector:
 matchExpressions:
 - { key: io.kubernetes.service.namespace, operator: NotIn, values: [kube-system] }

# 3. CiliumBGPClusterConfig — which nodes talk to which peers
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
 name: cluster-bgp
spec:
 nodeSelector:
 matchLabels:
 bgp: enabled
 bgpInstances:
 - name: instance-65000
 localASN: 65000
 peers:
 - name: tor-a
 peerASN: 64512
 peerAddress: 10.0.1.1
 peerConfigRef: { name: tor-peers }
 - name: tor-b
 peerASN: 64512
 peerAddress: 10.0.1.2
 peerConfigRef: { name: tor-peers }

# 4. CiliumLoadBalancerIPPool — range of LB IPs
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
 name: lb-corporate
spec:
 blocks:
 - cidr: "10.20.0.0/24"

Four objects. Before, in v1, it was a single CiliumBGPPeeringPolicy that mixed everything together and turned out to be hard to maintain in clusters of more than 5 nodes with heterogeneous configuration. The new separation is longer but clearly factorable: one PeerConfig per peer type, one Advertisement per advertisement policy, one ClusterConfig connecting nodes with peers.

Common operational traps

`routingMode: tunnel` mode with BGP

BGP only makes sense with direct routing (routingMode: native). If you have tunnel mode (VXLAN/Geneve) and configure BGP, you will advertise pod CIDRs but the packets will still go out encapsulated, producing confusing behaviour (sometimes via tunnel, sometimes direct depending on routes). Configure routingMode: native and disable the tunnel.

eBPF host routing vs `kube-proxy replacement`

They are two different things. kubeProxyReplacement: true enables the replacement of kube-proxy (the Services). bpf.hostRouting: true enables the host’s iptables bypass (the node’s routing decisions are made with eBPF instead of the traditional FIB). The second needs kernel 5.10+ with all the bpf features enabled; if you do not have that kernel, it falls back to legacy mode and the performance is only “almost as good”.

Aggressive BGP timers over flapping NICs

With holdTimeSeconds: 9 / keepAliveSeconds: 3, a NIC that blinks for 5 seconds breaks the BGP session and all the advertised routes disappear from the fabric. That node’s pods become unreachable until the session is re-established. For clusters on hardware with suspect NICs, use the conservative values (holdTime: 30, keepAlive: 10) and consider graceful restart explicitly (it is already in the example above).

Advertising ClusterIP to the corporate network

Advertising ClusterIP to external routers is rarely what you want: these are internal Service IPs, not designed to be reached from outside the cluster. For external exposure, use LoadBalancerIP from a CiliumLoadBalancerIPPool. Advertising ClusterIP only makes sense in very specific topologies (multi-cluster mesh with shared service discovery).

Mixing v2alpha1 (`CiliumBGPPeeringPolicy`) and v2 (`CiliumBGPClusterConfig`)

It does not work well. The operator emits warnings in the logs about the use of the deprecated API, and conflicts between what the peering policy defines and what the cluster config defines can produce strange states. Migrate from one to the other in a single pass; do not run both.

MD5 password and MTU

If you configure an MD5 password in CiliumBGPPeerConfig.authentication, the TCP header is larger. On links with a tight MTU (1500 - 50 for the upstream fabric’s VXLAN, for example), the BGP handshake can fragment and die silently. Either use MTU 9000 between nodes and ToR, or make sure the MSS values are negotiated correctly.

What we have not covered

Cilium Cluster Mesh: federation of several Cilium clusters so their Services can see each other. It fits with BGP when you want native routing between clusters; it has its own CRDs.
L7 Policies and the embedded Envoy: HTTP/gRPC/Kafka policy. Another layer of eBPF + proxy that deserves its own post.
Hubble: eBPF-based traffic observability that Cilium exposes. Flow log dashboards with zero impact on latency.
Transparent WireGuard: pod-to-pod encryption without sidecars, controlled by Cilium via eBPF redirect to a kernel WireGuard dataplane.
Gateway API in Cilium: the successor to Ingress, with first-class support since Cilium 1.16+.
eBPF for LLM serving: the natural connection with the previous inference series. There is recent work using eBPF for multi-tenant fairness on GPUs and for token tracking; paper territory, not production yet.

In other words, there is material left for another three posts in today’s series. Let us take them in order.

References

Conceptual and project:

eBPF.io — canonical documentation of the eBPF ecosystem.
The BPF Compiler Collection (bcc) and bpftrace — tools to get started with.
eBPF en 2026: How Extended Berkeley Packet Filter Became the Engine of Linux Observability and Networking — state of the art.

XDP, TC and firewalling:

Getting started with XDP and eBPF (Red Hat docs).
Full Guide to BPF Firewalls: XDP, tc, and eBPF Integration (Medium, 2025).
Cloudflare blog: XDP for DDoS mitigation.
Facebook Katran (GitHub) — L4 LB with XDP, code and paper.

Cilium:

Cilium documentation — always the first port of call.
Kubernetes Without kube-proxy — the official guide to the replacement.
Cilium BGP Control Plane Resources (docs) — reference for the v2 CRDs.
Configuring Cilium BGP Control Plane (OneUptime blog, mar 2026) — walkthrough.
A Guide to BGP Control Plane and Cluster Mesh in Cilium Networking (Sigrid Jin, Medium) — a deeper post with use cases.

Cross-references:

Previous post on this blog: Kubernetes with Cilium BGP: services reachable without Ingress — the first step, with version v1 (which now needs migrating).
Series on LLM inference: KV cache, vLLM on Kubernetes, PagedAttention deep dive, LLM K8s Operators — where the fast network (which we will see in the following posts of this series) determines real performance.

LLM inference operators on Kubernetes: OME, vLLM Production Stack, NVIDIA Dynamo and llm-d

Mon, 18 May 2026 17:00:00 +0200

TL;DR

Serving an LLM in production is not running a binary: it is coordinating a model (tens of gigabytes that take minutes to load), a runtime (vLLM, SGLang, TensorRT-LLM with a hundred flags), heterogeneous GPUs (NVLink, MIG, PCIe), prefill and decode that live better apart, a KV cache that wants offloading to colder tiers, intelligent routing that exploits prefix caching, and autoscaling that reacts to metrics other than CPU%. A flat Kubernetes Deployment only covers the first 20% of that. The other 80% is covered by LLM inference operators, of which four matter in 2026: OME (LMSYS, July 2025, multi-engine with a focus on SGLang), vLLM Production Stack (a curated Helm chart from vLLM itself with LMCache for tiered KV), NVIDIA Dynamo (the official successor to Triton, multi-engine, with its own Grove scheduler) and llm-d (a CNCF donation of March 2026 by Red Hat + Google + IBM + CoreWeave + NVIDIA, built on vLLM, focused on distributed scale). Behind all four sits KServe, the CNCF parent operator that normalised the InferenceService concept and that several of them lean on. This article walks through the full hierarchy, gives a decision map and teaches you not to get lost when someone drops seven acronyms in the first meeting.

This article closes the LLM inference series. The previous ones were KV cache: the working memory that holds up LLM inference, vLLM on Kubernetes: the LLM inference piece that actually scales and PagedAttention from the inside and the state of the art of the KV cache in 2026. There we explained what happens inside one inference process. Here we explain how many inference processes are coordinated through Kubernetes.

The analogy: from `init.d` to systemd to operators

Anyone with 20 years of sysadmin behind them will recognise the pattern. Decades ago, starting a service on Linux was a shell script in /etc/init.d/: start, stop, status, reloaded by hand. When services got more complex, with dependencies between them, monitoring, restart on failure, per-user slots, it became obvious that a script was not enough. Along came systemd, which turned “a service” into a declarative unit with dependencies, resources, restart policy, sockets and timers. The script did not disappear; it moved up one level of abstraction.

Kubernetes made the same move for distributed services. A Deployment declares “I want N replicas of this container”; a Service declares “these replicas are exposed like this”; an Ingress declares “this HTTP traffic comes in here”. The controller translates the declaration into real state and keeps the system converged.

Serving LLMs in 2024 was the equivalent of /etc/init.d/: every team wrote its own Deployment/Service/HPA with custom scripts for model loading, session draining and GPU handling. We covered it in the vLLM on Kubernetes article: it can be done, and it does work, but it is repetitive, fragile and nobody is extracting the right abstractions. Serving LLMs in 2026 has gone through the same transition as services: the equivalent of systemd has appeared, the inference operators, which normalise the abstractions and let the engineer declare what matters: “this model, with this runtime, this scalable, with this routing policy”.

There are four relevant operators in 2026 and a fifth common ancestor. Let us take them in order.

Why an operator, and not just a Deployment

Listing what an inference operator adds over a flat Deployment is the best way to understand what problem it solves:

The model as a first-class citizen. In a Deployment, the model is “the thing you download in an initContainer and mount as a volume”. In an operator, the model is a CustomResource with metadata (origin, fingerprint, licence, GPU requirements). It can be shared between InferenceServices, versioned, replicated to multiple nodes. It is the difference between “a file” and “a managed artifact”.

The runtime as a first-class citizen. The same goes for the runtime (vLLM/SGLang/TRT-LLM): it is not “a Docker image with flags”; it is a ServingRuntime that declares which args it accepts, which metrics it exports, which deployment types it supports (single-node, multi-node TP, PD-disag). Changing runtime is changing a reference, not rewriting every manifest.

Declarative composition. An InferenceService (the core CRD of KServe and its descendants) references a model and a runtime, declares the scaling policy, wires up observability and configures routing. The controller composes all the pieces: Deployment(s), Service, HPA, possibly a LeaderWorkerSet, a KEDA ScaledObject, a Gateway API HTTPRoute. You declare intent; the operator emits the 8 derived resources.

Operational prefill–decode disaggregation. As we saw in the PagedAttention article, separating prefill and decode into distinct pools can give 7× goodput. Modelling that with flat Deployments is feasible, but it requires coordinating two sets of pods, a transport to move the KV cache and conditional routing. An operator models it as a single InferenceService with two sub-pools.

Autoscaling on LLM metrics. The standard HPA does not understand vllm:num_requests_waiting. An operator integrates KEDA or Prometheus Adapter automatically and exposes the right metrics as knobs on the CRD.

Multi-tenancy. Multiple models on the same cluster, with quotas, priorities and fairness. One Deployment per model scaling independently is fine up to the fifth model; beyond that, coordinating GPUs between tenants becomes operationally hostile.

Model lifecycle. Weights on a shared PVC, warm-up of the first pod, rolling updates with maxUnavailable: 0, draining of active sessions, integrated observability. Things that with a flat Deployment have to be reinvented by every team.

If your workload is one model, one node, up to three replicas, a flat Deployment is enough and an operator is overkill. If your workload is two or more models, serious scaling, disaggregation or multi-tenancy, an operator stops being optional.

KServe: the common ancestor

Before the four new ones, we have to mention KServe, the parent operator from which all the rest conceptually descend. It was born as KFServing inside the Kubeflow project in 2019, was renamed KServe when it became independent in 2021, and in 2025 was accepted into the CNCF as a project incubating towards graduation.

KServe’s conceptual contribution is the InferenceService CRD, which has become the field’s common vocabulary: a declarative K8s object that joins a model (origin + metadata) with a predictor (runtime + resources) and produces a ready HTTP service. Under the hood, the controller emits Deployments, Services, HorizontalPodAutoscalers, Knative Services if you go serverless, Istio VirtualServices if you do traffic splitting.

KServe was designed in a pre-LLM era: its first use cases were traditional scikit-learn, TensorFlow and PyTorch models served as simple REST APIs. That gives it strengths (it is mature, with 6 years in production at Bloomberg, JPMorgan and others) and weaknesses (it was not designed to manage multi-node tensor parallel, prefill–decode disaggregation, or the patterns specific to LLMs).

The way the ecosystem has reacted is elegant: the new LLM operators inherit from or take inspiration from InferenceService but extend the API with LLM-specific primitives. OME is the clearest example: it uses the name InferenceService and the idea of “model + runtime → service”, but adds BaseModel, a ServingRuntime with LLM-aware flags, and deployment modes (PD-disag, multi-node) that KServe does not cover natively.

OME (Open Model Engine)

OME was published by the LMSYS team in July 2025 (announced on their blog). It is an operator that understands SGLang deeply (that is its first-class runtime) but also supports vLLM, TensorRT-LLM and Triton.

The CRD hierarchy

OME models the domain with four main CRDs:

BaseModel and ClusterBaseModel: the model itself. It defines origin (Hugging Face, S3, URL), fingerprint and metadata. The Cluster* version is global; BaseModel is namespaced. It lets multiple InferenceService objects reference the same model without duplicating the download.
FineTunedWeight: LoRA adapters or finetuned weights served on top of a BaseModel. Critical for multi-tenant setups where each customer has their own finetune.
ServingRuntime and ClusterServingRuntime: the runtime (vLLM, SGLang, and so on) with its configuration. It declares which args it accepts, which metrics it exports, which deployment modes it supports.
InferenceService: the central declarative piece, joining BaseModel + ServingRuntime + infrastructure.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
 name: llama3-70b-prod
 namespace: inference
spec:
 model:
 name: meta-llama-3-70b-instruct  # reference to a BaseModel
 runtime:
 name: sglang-h100  # reference to a ServingRuntime
 deploymentMode: PrefillDecodeDisaggregated  # standard | PD | MultiNode | Serverless
 prefill:
 minReplicas: 2
 maxReplicas: 8
 resources:
 requests:
 nvidia.com/gpu: 4
 decode:
 minReplicas: 4
 maxReplicas: 16
 resources:
 requests:
 nvidia.com/gpu: 1
 router:
 type: cache-aware  # SGLang router with cache awareness
 autoscaling:
 metricSource: keda
 metrics:
 - type: prometheus
 metricName: vllm_requests_waiting
 threshold: "10"

That is what the operator takes as input. The output is roughly 8 derived resources that would be horrible to declare by hand: two LeaderWorkerSets (one per prefill/decode pool), two Services, a Deployment for the router, KEDA ScaledObjects for each pool, a Gateway API HTTPRoute, and a PriorityClass that hooks into Kueue for gang scheduling.

The four deployment modes

OME materialises the InferenceService differently depending on deploymentMode:

Standard: one Deployment with N replicas; the classic. For small or single-GPU models.
PrefillDecodeDisaggregated: two coordinated pools; the SGLang router routes between them.
MultiNode: tensor parallel across multiple nodes via LeaderWorkerSet, with NCCL/InfiniBand. For models above 70B where a single node is not enough.
Serverless: Knative-style scale-to-zero. For sporadic workloads where the cost of keeping GPUs powered up does not pay off. The trade-off: the first request pays the model’s cold start cost (minutes).

Integration with the K8s ecosystem

OME does not invent primitives where they already exist. It leans on:

Kueue for gang scheduling: all the pods of a tensor parallel group have to start at once or not at all; Kueue guarantees it.
LeaderWorkerSet (LWS) for multi-node: workers join the leader’s Ray cluster, with an atomic lifecycle (one going down restarts the group).
KEDA for autoscaling on LLM-specific Prometheus metrics (queue depth, GPU cache usage, TTFT p95).
Gateway API and its Inference Extension for advanced routing (model-aware, prefix-aware, weighted canary).

The consequence: OME feels “idiomatically Kubernetes”. It does not introduce new concepts where they are not needed; it uses standard primitives and concentrates on what is specific to the LLM domain.

When to choose it

OME is the natural choice if SGLang is your main runtime and/or if you come from the KServe ecosystem and want an idiomatic evolution of it. It is mature but relatively young (a year old at the time of writing); expect rough edges on advanced features.

vLLM Production Stack

vLLM Production Stack is the official project from vLLM itself for production on Kubernetes. Its philosophy is the opposite of OME’s: instead of an operator with new CRDs, it is a curated Helm chart that deploys a coherent set of pieces.

The three pieces

The stack has three components:

Serving engines: vLLM pods, configured with the flags we have been seeing throughout the series (--enable-prefix-caching, --kv-cache-dtype fp8, and so on). The Helm chart lets you declare them as a list; it deploys the underlying Deployments and Services.
Request router: a proxy in front of the engines that decides which one each request goes to. It supports several policies:
- Round-robin: trivial, for a baseline.
- Session-based: pins each session to one replica to keep its KV cache.
- Prefix-aware: detects shared prefixes between requests and routes them to the replica that already has them cached.
- KV-aware: sees each replica’s gpu_cache_usage_perc and avoids the saturated ones.
- Disaggregated-prefill with native LMCache: separates prefill and decode, with LMCache as the transport for the KV cache between the two.
Observability stack: Prometheus + Grafana with ready-made dashboards. It measures TTFT, TBT (Time-Between-Tokens), throughput, queue depth and GPU memory.

LMCache and tiered KV

One of the most interesting pieces the stack brings is LMCache, which adds a KV cache with multiple tiers: GPU HBM as L1, CPU RAM as L2, local disk as L3, and optionally remote storage as L4. When a KV cache block does not fit in HBM, instead of evicting and recomputing it, LMCache drops it to a lower tier. For workloads with shared prefixes and multi-turn conversations, the saving is huge.

LMCache integrates as a sidecar to the engines and as part of the transport in disaggregated-prefill. Production Stack ships it enabled by default in its Helm chart.

A typical manifest (values.yaml)

servingEngineSpec:
 modelSpec:
 - name: llama3-8b
 repository: vllm/vllm-openai
 tag: v0.6.3
 modelURL: meta-llama/Meta-Llama-3-8B-Instruct
 replicaCount: 3
 requestCPU: 4
 requestMemory: 16Gi
 requestGPU: 1
 vllmConfig:
 enablePrefixCaching: true
 kvCacheDtype: fp8
 maxModelLen: 32768
 enableChunkedPrefill: true

routerSpec:
 routingLogic: prefix-aware  # round-robin | session | prefix-aware | kv-aware
 sessionKey: x-user-id  # when routingLogic=session

cacheserverSpec:
 enabled: true # LMCache for tiered KV
 storageBackends:
 - cpu
 - disk  # offload to local disk

observabilitySpec:
 prometheus:
 enabled: true
 grafana:
 enabled: true
 dashboards:
 - vllm-engine-metrics
 - lmcache-metrics

This is declarative but these are not CRDs: they are Helm chart values. The difference with OME is not semantic (both start from declaration) but operational: with Helm, changes go through helm upgrade; with CRDs, they go through kubectl apply. For teams already living in GitOps with Argo CD or Flux, both approaches integrate cleanly, but the flows are different.

When to choose it

If vLLM is your only runtime and you want the closest thing to “the happy path the project recommends”, this one. It is the productionised version maintained by the same people who write the engine. The downsides: it ties you to vLLM (it is not generic) and it does not solve some advanced cases such as multi-tenancy with strict quotas or gang scheduling, where OME or full-fledged operators are better.

NVIDIA Dynamo

NVIDIA Dynamo is the official successor to Triton Inference Server, announced at GTC 2025 and merged into the brand as Dynamo-Triton in March of that year. Triton had spent years as the most-used inference engine in “serious” NVIDIA infrastructures; Dynamo is what NVIDIA believes the new generation needs.

What it is exactly

Dynamo is a distributed inference framework, not exactly a Kubernetes operator. It has its own runtime (it can run engines), a scheduler (Grove), intelligent routing, multi-tier KV cache management and disaggregation. It supports SGLang, TensorRT-LLM and vLLM as engines, but the engines are executed by Dynamo, not the other way round: the model is “Dynamo manages, the engine executes”.

On Kubernetes, Dynamo is deployed via its own operator plus CRDs, normalised with the K8s integration NVIDIA formalised at the end of 2025 (covered by this InfoQ piece). The CRDs are product-specific: they define a DynamoCluster, a topology of prefill/decode workers and a routing policy.

The four contributions

Dynamo is sold on four pillars, with figures reported by NVIDIA:

Disaggregated serving built in, with its own scheduler.
Smart routing based on cache state: if a worker already has most of a prompt cached, the request goes there.
Multi-tier KV cache: analogous to LMCache, with HBM/RAM/SSD/NVMe.
Autoscaling integrated with Dynamo’s scheduler.

The marketing figure: up to 30× more throughput than legacy Triton on the same hardware. With all the caution a vendor benchmark deserves.

Grove: its own scheduler

A controversial decision in Dynamo is not to rely 100% on the Kubernetes scheduler and instead bring its own scheduler, called Grove, that understands GPU topologies. Grove decides which worker runs on which physical GPU, which interconnects (NVLink/InfiniBand) are relevant, and how to distribute tensor parallel across nodes. That gives it more control than the standard kube-scheduler.

Operationally: if your cluster is “pure Kubernetes” with kube-scheduler and heterogeneous workloads (not only LLMs), Grove adds one more component to operate. If your cluster is dedicated to LLM inference and there is already a team dedicated to running it, Grove gives you more levers.

When to choose it

Dynamo makes sense if:

Your infrastructure is NVIDIA-heavy (Hopper, Blackwell, GB200) and you want to exploit the latest in TensorRT-LLM with the familiar Triton integration, modernised.
You were already a Triton user for legacy inference (vision, recommendation) and want to keep the ecosystem.
You have a dedicated SRE team for inference and Grove’s extra operational complexity is not a problem.

It is the vendor-specific option of the quartet. In exchange it gives you NVIDIA’s support and first-rate integration with their hardware. If your organisation is already fighting NVIDIA for GPUs, they may even call you offering help with Dynamo.

llm-d

llm-d is the youngest and the most “political” of the four. In March 2026, at KubeCon Europe Amsterdam, Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA announced the joint donation of the project to the CNCF as a Sandbox project, with backing from AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley and the University of Chicago. An explicit coalition for vendor neutrality.

Philosophy

llm-d positions itself as the vendor-neutral “Kubernetes blueprint” for distributed inference. It is not a runtime; it is a system that sits on top of vLLM (the default engine) and orchestrates the control plane.

The primitives the project puts on the table:

Intelligent routing with prefix-cache awareness and load-aware balancing.
Tiered KV cache with offload to CPU and disk for multi-turn.
Prefill/decode disaggregation over fast interconnects.
Wide expert parallelism for serving very large Mixture-of-Experts (MoE) models, a critical pattern popularised by DeepSeek-V3 and Mixtral, where the experts live on different GPUs and tokens have to be routed to the right expert.

Numbers

The v0.5 release validates ~3.1k tok/s per decode GPU on B200, and up to 50k output tok/s on a 16×16 B200 prefill/decode topology. The most interesting benchmark: an order-of-magnitude reduction in TTFT against a round-robin baseline. In other words, intelligent routing is worth what they say it is.

CNCF and the future

Donating to the CNCF as a Sandbox project means neutral governance: no vendor is in charge. For an organisation wary of being tied to a single supplier, llm-d is probably the safest medium-term bet. The price: like any Sandbox project, it is not yet “boring” in the way vLLM is. There is API churn, features that move around, documentation that lags the code.

When to choose it

llm-d makes sense if:

You want multi-vendor portability without ties to NVIDIA, Red Hat or Google.
Your workload includes large MoE models (DeepSeek-V3, Mixtral 8x22B, Llama 4 Behemoth if the size is confirmed), where wide expert parallelism is decisive.
Your organisation is already comfortable with CNCF Sandbox (projects under active evolution, not yet a stable 1.0).
You want to bet on the project that will probably be the de facto standard in 2-3 years.

The common ancestor is still there: KServe

It is worth reconnecting before the comparison: KServe is still alive and heavily used in organisations that serve both LLMs and traditional models (scikit-learn, XGBoost, PyTorch CV). Its InferenceService is generic enough to serve any model, including vLLM or SGLang as a ServingRuntime. What it does not do well is the LLM-specific part: disaggregation, multi-node tensor parallel, routing with KV cache awareness. If your organisation already has KServe in production for other models, adding an LLM-specific operator alongside it (OME, vLLM Stack or llm-d) is reasonable. Fighting it all out from pure KServe is not.

Decision map

Dimension	OME	vLLM Prod Stack	NVIDIA Dynamo	llm-d
Philosophy	Classic K8s-idiomatic operator	Curated Helm chart	Framework with its own scheduler	Vendor-neutral CNCF blueprint
Own CRDs	Yes (BaseModel, ServingRuntime, InferenceService…)	No (Helm values)	Yes (DynamoCluster)	Yes (KServe-derived + extensions)
Primary runtime	SGLang (first class), also vLLM/TRT-LLM/Triton	vLLM only	TensorRT-LLM (first class), also SGLang/vLLM	vLLM (first class)
PD-disaggregation	Yes, declarative	Yes, with LMCache	Yes, own scheduler	Yes, native
Multi-node TP	Yes, via LWS	Limited	Yes, via Grove	Yes, via LWS and MoE EP
Multi-model in a cluster	Yes, mature multi-tenant	Yes (list of models in values)	Yes	Yes
Multi-LoRA	Yes, first class (FineTunedWeight CRD)	Limited	Yes	On the roadmap
Tiered KV cache	Via LMCache (external integration)	Native LMCache	Own multi-tier	Yes, native
Intelligent routing	Cache-aware via the SGLang router	Prefix-aware / KV-aware / session-based	Own smart routing	Prefix-cache + load-aware
GPU scheduler	kube-scheduler + Kueue	kube-scheduler	Grove (its own)	kube-scheduler + Kueue
Hardware	NVIDIA, AMD ROCm, Intel	NVIDIA, AMD ROCm	NVIDIA only (with emphasis)	NVIDIA, AMD, Intel — neutral
Maturity (mid-2026)	Young, evolving	Stable	Stable, vendor-driven	CNCF Sandbox, evolving fast
Governance	LMSYS (academic-industrial)	vLLM project (academic)	NVIDIA (vendor)	CNCF (neutral)
Learning curve	Medium (4 new CRDs)	Low (familiar Helm values)	Medium-high (Grove + own CRDs)	Medium (similar to extended KServe)

When to choose each one

Choose OME if:

SGLang is your main engine.
You need multi-LoRA serving in production.
The hierarchical abstraction (BaseModel → ServingRuntime → InferenceService) fits you and you come from, or live alongside, KServe.
You have an appetite for a young and very active project.

Choose vLLM Production Stack if:

vLLM is your only engine and you want to align with what the project recommends.
Your team already lives in Helm and does not want to learn new CRDs.
LMCache + advanced routing inside a single Helm chart is exactly what you need.
Your scale is medium (tens of replicas), not extreme.

Choose NVIDIA Dynamo if:

Your infrastructure is NVIDIA-heavy and you want the most optimised path for Hopper/Blackwell.
You already ran Triton for legacy inference and the transition is natural.
You accept vendor lock-in in exchange for direct NVIDIA support.
Your organisation has a dedicated SRE team for inference.

Choose llm-d if:

You want to bet on the future CNCF standard, neutral between vendors.
Your workload includes large MoE models with wide expert parallelism.
You operate multi-cloud or multi-hardware and portability is valuable.
You accept the immaturity of a Sandbox project in exchange for the bet on the future.

Choose pure KServe if:

You already serve non-LLM models and want to unify; LLMs are a minority of your workload.
You need the most conservative and mature use case.
You accept that advanced LLM features (disaggregation, MoE EP, smart routing) are yours to add with external pieces.

Concrete scenarios

Scenario A — Small startup, 1-2 models, 1-3 GPU nodes. You probably do not need an operator. Deployment + Service + HPA with KEDA metrics, as in the vLLM on Kubernetes article. When you grow to 5+ models, reassess.

Scenario B — Mid-sized company, 5-15 models, internal multi-tenant. vLLM Production Stack or OME are the reasonable options. Production Stack if vLLM is all you are going to use; OME if you want runtime flexibility and idiomatic CRDs.

Scenario C — Internal corporate platform or an external service for end customers. llm-d or Dynamo. llm-d if you value vendor neutrality; Dynamo if you live on NVIDIA infrastructure and want the path they recommend.

Scenario D — Mixed cluster of LLMs plus traditional models. KServe as the base, with an LLM operator alongside (OME is the most natural given its conceptual kinship).

Common traps

“I’ll start with pure KServe because it’s mature”. For mid-sized LLMs and up, pure KServe leaves a lot of optimisation on the table. The sensible route is KServe as the base if you live alongside other models, but with an LLM-specific operator next to it.

“I’ll wire everything by hand to understand it”. Reasonable in a PoC, suicidal in production. There are 8 derived resources per model. Multiply by 10 models. You are writing 80 YAMLs and maintaining them. Use an operator.

“I’ll pick the one I like best and pivot if I’m wrong”. Pivoting between operators is not free: although the InferenceService abstraction is converging, the details (how LoRA is modelled, how routing is configured, how metrics are exposed) vary. Migrating from OME to Dynamo is a project of weeks, not days.

“I’ll go with Dynamo because it’s NVIDIA’s and therefore better”. Only if your organisation is already aligned with its operational philosophy (its own scheduler, acceptable vendor lock-in). For many cases, vLLM Production Stack or llm-d give 95% of the value with less friction.

“Helm chart vs operator is a technical decision”. It is a cultural and operational decision. If your team ships via Argo CD with Helm values in Git, Production Stack fits without friction. If your team lives in direct kubectl apply -f and the idea of operators feels natural, OME or llm-d.

What we have not covered

Mooncake: the KV cache system shared between instances that Kimi/Moonshot runs in production across hundreds of millions of queries. It is a primitive (not a complete operator), but it integrates as a cache tier with several of the above.
Ray Serve LLM: Anyscale’s offering, on Kubernetes through KubeRay. More tied to the Ray ecosystem than to native K8s CRDs. Useful if Ray is already part of your infrastructure.
Fireworks AI, Modular MAX: commercial platforms with similar primitives, but hosted. They are not K8s operators; they are competitors at another layer.
Gateway API Inference Extension: the sigwg proposal to extend Gateway API with LLM primitives (model-aware routing, sticky sessions, fairness). In 2026 it is in alpha; the operators above are already starting to support it. When it matures, routing will stop being each operator’s problem and become part of the Kubernetes standard.
A generic inference observability stack: Prometheus + Grafana is standardising around the vllm:* metrics we covered in the vLLM article. There is an OpenTelemetry effort for LLMs (gen-ai semantic conventions) that is probably the next link in the chain.

Closing the series

This series of four articles has walked through LLM inference in production from the bottom up:

KV cache: the working memory that holds up LLM inference — why each token consumes VRAM, and how much.
vLLM on Kubernetes: the LLM inference piece that actually scales — how a model is served in production with a serious Deployment.
PagedAttention from the inside and the state of the art of the KV cache in 2026 — what happens inside the engine at block level, and what has arrived since.
This one — how many models are orchestrated in a cluster.

If you have got this far, you have the vocabulary and the map to sit in a meeting where five people throw acronyms around and place each one correctly. And, more importantly, to start making reasoned decisions about where to begin.

References

Operators and projects covered:

OME — Open Model Engine (GitHub) — LMSYS operator for LLM serving with SGLang/vLLM/TRT-LLM/Triton.
Introducing OME (LMSYS Blog, jul 2025) — announcement and architecture.
vLLM Production Stack (GitHub) — official vLLM Helm chart for K8s.
vLLM Production Stack docs — installation and configuration.
LMCache (GitHub) — KV cache with tiers.
NVIDIA Dynamo — successor to Triton.
NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges (InfoQ, dic 2025) — K8s integration.
llm-d (GitHub) — CNCF Sandbox project.
IBM, Red Hat, and Google donated llm-d to CNCF (The New Stack) — KubeCon EU 2026 announcement.
Red Hat bets big on Kubernetes inference with llm-d (SiliconANGLE, mar 2026) — coverage of the announcement.

Ancestors and primitives:

KServe (site) and KServe joins CNCF (The New Stack).
Kueue — gang scheduling.
LeaderWorkerSet — coordinated workloads such as multi-pod tensor parallel.
KEDA — autoscaling on external metrics.
Gateway API — the successor to Ingress.

Analysis and perspectives:

Building Efficient LLM Inference with the Cloud Native Quartet: KServe, vLLM, llm-d, and WG Serving (Jimmy Song) — an integrating view.
Complete Guide to llm-d CNCF Sandbox (DEV Community) — an operational walkthrough.
Previous articles on this blog: KV cache, vLLM on Kubernetes, PagedAttention deep dive.

vLLM on Kubernetes: the LLM inference piece that actually scales

Mon, 18 May 2026 13:00:00 +0200

TL;DR

vLLM is the inference engine that turns a general-purpose GPU into a productive LLM server. Its value is not in running a model, since any transformers.pipeline does that in three lines of Python, but in squeezing the GPU down to the last gigabyte and the last cycle: PagedAttention for the KV cache, continuous batching to mix requests, its own scheduler to share GPU time between sessions. Kubernetes is its natural habitat because vLLM behaves like a modern UNIX process, with a health endpoint, Prometheus metrics, orderly draining and declarable resources, and K8s already knows how to manage those. But there are traps: the standard HPA does not scale vLLM well, the model takes minutes to load, and naive rolling updates cut sessions halfway through decoding. This article takes the engine apart and then fits it, with real manifests, into a cluster that can actually serve it.

This article is the natural continuation of KV cache: the working memory that holds up LLM inference. There we explained why every token consumes VRAM. Here we look at what you do with that VRAM when you want to offer it as a service.

The analogy: a multiprocess kernel for your GPU

Imagine you have a single processor and need to serve a hundred concurrent processes without any of them blocking the rest. Nobody in their right mind would write a while-true loop dispatching processes one by one: they would install an operating system. The kernel takes care of scheduling, memory paging, isolation, priorities and cleanup on exit. The “process” becomes a convenient abstraction and the kernel does the dirty work.

vLLM is, for your GPU, what the kernel is for your CPU. As far as the GPU is concerned, a conversation with an LLM is a process that lives across many decoding steps, occupies a slice of VRAM (its KV cache) and demands compute time every time a token has to be generated. You have a hundred of those processes at once. You need to:

Share GPU time between them without pausing any of them entirely (it would be disastrous if one long conversation monopolised the GPU).
Manage memory with paging because, just as in RAM, contiguous allocation is inefficient.
Queue new requests when the GPU is saturated and serve them in a reasonable order.
Reclaim resources when a session ends.

PagedAttention is the virtual memory of the KV cache. Continuous batching is the time-slicing scheduler that shares the GPU token by token. The OpenAI-compatible server is the uniform syscall interface. Calling it a “kernel” for the GPU is marketing, but it is marketing that captures the idea well.

What vLLM does inside

Continuous batching: stop waiting for the slowest

The naive inference engine does static batching: it groups N requests, processes them until all of them are finished, returns and starts another round. The problem is obvious: if one request asks for 8 tokens and another asks for 800, the other seven wait for the slow one. GPU utilisation collapses.

Continuous batching (Yu et al., 2022, popularised by vLLM) changes the model. At every decode step, which produces one token for each active session, the engine composes the batch out of the active tokens of ALL sessions alive at that instant. When a session finishes its generation it releases its slot immediately and another request from the queue takes it. The batch never waits for the slowest session because nobody is blocked: everyone advances at one token per step.

The original paper measured 5–23× more throughput than the equivalent static batching. The exact number depends on how variable the response lengths are, but the order of magnitude holds up in practice.

The consequence for the operator is counterintuitive: a single vLLM replica performs like three naive replicas. There is no point adding pods without justifying it with real metrics.

PagedAttention: virtual memory for the KV cache

We already flagged it in the KV cache article: the naive engine reserves one contiguous block per session, sized for the worst case (max_context_len), and wastes 60–80% of the VRAM because real sessions come nowhere near their ceiling.

PagedAttention borrows the solution operating systems have been using for half a century: split the VRAM into small blocks (16 tokens in the default implementation) and keep a logical → physical page table per session. A session with 273 tokens of context occupies 18 blocks (not necessarily contiguous), and grows block by block as it generates. The paper measured <4% waste, an order of magnitude better than contiguous allocation, and that translates into 2–4× more aggregate throughput on the same hardware, because more sessions fit at once.

There is a cost: every attention operation has to be indirected through the page table. But vLLM’s CUDA kernels are written so that this indirection is cheap, and the net result is massively positive.

Prefill vs decode: two phases with opposite profiles

An LLM request has two phases with radically different GPU profiles:

Prefill: processes the entire prompt in one go. It is compute-bound: it uses the tensor cores intensively, the GPU sits above 90%, and it lasts between hundreds of ms and a few seconds depending on prompt size.
Decode: generates token by token. It is memory-bound: the compute is modest but the whole KV cache has to be read for every token, and it lasts from tens of ms per token up to minutes for long responses.

A naive server treats each request as a unit and serves the two phases in series. vLLM decouples them: it mixes requests in prefill with requests in decode in the same step (a technique called chunked prefill when it also splits long prefills into chunks). The result: the GPU is always busy doing something, the tensor cores with prefills, the HBM bandwidth with decodes, instead of oscillating between phases.

Operational implication: the “% GPU utilisation” metric from nvidia-smi is misleading. A GPU at 100% doing prefills may have its HBM bandwidth idle. A GPU at 40% doing decodes may have its HBM saturated. For LLM serving, the useful metric is effective HBM bandwidth, not the compute percentage.

Tensor parallel: when the model does not fit on one GPU

Llama 3 70B in BF16 is ~140 GB. There is no single GPU on the market that can hold it. The solution is tensor parallel: split each model layer by columns and run the partitions on N GPUs in parallel, synchronising with an all-reduce after each layer.

For N=5 GPUs and a 70B model, each GPU sees roughly 28 GB of weights. That sounds fine until you remember that the all-reduce on every layer means reading and writing large tensors between GPUs. If the GPUs share NVLink/NVSwitch (300–900 GB/s), the all-reduce is cheap. If they only share PCIe (~32 GB/s gen4 x16), the all-reduce eats half the time and throughput collapses.

The implication for K8s, which comes next: the scheduler has to guarantee that the N GPUs are physically close. That translates into NodeAffinity to the right product (NVIDIA-H100-80GB-HBM3), a single pod with nvidia.com/gpu: N (not N pods sharing) and, if multi-node is unavoidable, InfiniBand with NCCL as transport.

The OpenAI-compatible server

On top of all of the above, vLLM exposes an HTTP server with endpoints identical to OpenAI’s: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models. It supports Server-Sent Events streaming. It supports tool calling. It supports logprobs.

The value of this is enormous and underrated: any client using the OpenAI SDK works without changes. Your application points at https://vllm.your-cluster.local/v1 instead of https://api.openai.com/v1, and everything else, the LangChain, LlamaIndex, OpenAI Python and OpenAI JS SDKs, just works. It is the main reason vLLM has gained traction over technically comparable alternatives: it is the boring option that works.

Why Kubernetes is the natural habitat

vLLM is a well-behaved process: it starts, exposes metrics, serves a health endpoint, takes SIGTERM with dignity and declares the resources it needs. Kubernetes has spent ten years perfecting the management of processes like that. The only thing K8s took a while to absorb properly was the GPU, and that is now solved.

The GPU as a primitive resource

The plumbing goes like this:

The node has the NVIDIA driver installed (or the GPU Operator installs it).
A DaemonSet, nvidia-device-plugin, registers the physical GPUs as nvidia.com/gpu resources with kubelet.
The Kubernetes scheduler sees those resources the way it sees CPU and memory, puts them in its accounting and assigns them to Pods that ask for them.
The nvidia-container-toolkit makes sure containerd injects the right devices into the container at startup.

For the pod, asking for a GPU looks like this:

resources:
 requests:
 nvidia.com/gpu: 1
 limits:
 nvidia.com/gpu: 1

Without MIG, MPS or time-slicing configured, a GPU is not shared between pods: you ask for it whole or you do not ask for it. For vLLM, which wants the entire GPU to itself, that is exactly what you want.

The vLLM Pod lifecycle

Differences from a typical webapp Pod:

Long startup. Loading 16 GB of weights into VRAM over the network takes 30 seconds in the best case and 5 minutes in the worst. A readinessProbe with initialDelaySeconds: 30 and failureThreshold: 3 kills the pod before it starts. Solution: a startupProbe with a high threshold before the livenessProbe starts evaluating.
Warm-up matters. The first prefill compiles CUDA kernels specific to the input shape. The first 2–3 requests are noticeably slower. If latency matters from second one, it is worth firing a warm-up POST after ready.
Draining is not instantaneous. SIGTERM must not kill in-flight sessions. vLLM, configured with --disable-graceful-shutdown false (the default), finishes active requests before closing. That can take 30–180 seconds. terminationGracePeriodSeconds has to accommodate it.
Hostile rollouts. A naive rolling update (maxUnavailable: 1) can leave you with no replicas serving if the new one is slow to load. Set maxSurge: 1, maxUnavailable: 0 so the new pod is Ready before the old one is drained.

Anatomy of a serious deployment

First of all: GPU Operator

Without the GPU Operator (or an equivalent manual installation), a Pod with nvidia.com/gpu: 1 stays Pending forever. What the operator installs as DaemonSets on every GPU node:

nvidia-driver-daemonset — the kernel-mode driver (if you do not have it installed at host level).
nvidia-device-plugin-daemonset — registers the GPUs as a kubelet resource.
nvidia-container-toolkit-daemonset — the containerd integration.
nvidia-dcgm-exporter — Prometheus metrics for the GPU (utilisation, temperature, ECC errors, memory).
gpu-feature-discovery — node labels: nvidia.com/gpu.product, nvidia.com/gpu.memory, and so on, essential for NodeAffinity.

The recommended installation is the official Helm chart. The delicate part is aligning the driver with the host kernel version: if the nodes run kernel 6.x, the operator needs a compatible driver branch.

A complete, annotated vLLM Deployment

The following deploys Llama 3 8B with an FP8-quantised KV cache, up to 32K of context, on an RTX 4090. It is the reference manifest; the comments explain the non-obvious decisions.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 replicas: 1
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 0 # never run out of replicas during the rollout
 selector:
 matchLabels:
 app: vllm-llama3-8b
 template:
 metadata:
 labels:
 app: vllm-llama3-8b
 annotations:
 prometheus.io/scrape: "true"
 prometheus.io/port: "8000"
 prometheus.io/path: "/metrics"
 spec:
 # Only nodes with the GPU we expect
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-GeForce-RTX-4090
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 # Pre-download the weights if they are not on the shared PVC
 initContainers:
 - name: model-download
 image: ghcr.io/huggingface/huggingface-cli:latest
 command: ["sh", "-c"]
 args:
 - |
 if [ ! -f /models/llama-3-8b/config.json ]; then
 huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct \
 --local-dir /models/llama-3-8b --local-dir-use-symlinks False
 fi
 env:
 - name: HF_TOKEN
 valueFrom:
 secretKeyRef:
 name: huggingface
 key: token
 volumeMounts:
 - name: models
 mountPath: /models
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-8b
 - --served-model-name=llama-3-8b
 - --tensor-parallel-size=1
 - --max-model-len=32768
 - --kv-cache-dtype=fp8
 - --enable-chunked-prefill
 - --enable-prefix-caching
 - --gpu-memory-utilization=0.92
 - --port=8000
 ports:
 - name: http
 containerPort: 8000
 - name: metrics
 containerPort: 8000 # same port as http; /metrics
 resources:
 requests:
 cpu: "4"
 memory: 8Gi
 nvidia.com/gpu: 1
 limits:
 cpu: "8"
 memory: 16Gi
 nvidia.com/gpu: 1
 startupProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 10
 failureThreshold: 60 # 10 min of grace to load the model
 readinessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 5
 livenessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 20
 failureThreshold: 3
 volumeMounts:
 - name: models
 mountPath: /models
 readOnly: true # no process should write here at runtime
 - name: shm
 mountPath: /dev/shm  # vLLM uses shared memory for IPC between workers
 volumes:
 - name: models
 persistentVolumeClaim:
 claimName: model-cache
 - name: shm
 emptyDir:
 medium: Memory
 sizeLimit: 4Gi
 terminationGracePeriodSeconds: 120 # accommodates draining of active sessions
---
apiVersion: v1
kind: Service
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 selector:
 app: vllm-llama3-8b
 ports:
 - name: http
 port: 80
 targetPort: 8000

Five things that are not visible on a first read:

/dev/shm in memory, 4 GB. vLLM launches worker processes (one per GPU in tensor parallel, plus the driver) that communicate over shared memory. Docker’s default (64 MB) blows up as soon as the model is mid-sized. Without this, the pod starts but fails the moment it serves the first complex request.
--enable-prefix-caching. If the prompts in your workload share structure (a common system prompt, few-shot examples), vLLM reuses the KV cache of the shared part. A free 30–60% gain in TTFT.
--gpu-memory-utilization=0.92. vLLM reserves the given percentage of the VRAM for itself. The remaining 8% leaves room for activations, CUDA kernels and the overhead that does not get counted. Lowering it buys safety; raising it above 0.95 invites OOM.
A ReadOnlyMany PVC is ideal. The model does not change at runtime. Several pods can mount the same PVC without contention.
No livenessProbe shorter than the terminationGracePeriodSeconds. If a drain takes 90s and liveness kills at 60s, rollouts lose sessions.

Multi-pod tensor parallel: LeaderWorkerSet

When the model needs more GPUs than a single node has, the pattern is a group of coordinated pods, one per GPU, that behave as a single replica. For years this was modelled with a StatefulSet plus init scripts; since Kubernetes 1.32, the idiomatic primitive is LeaderWorkerSet (LWS):

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
 name: vllm-llama3-70b
 namespace: inference
spec:
 replicas: 1
 leaderWorkerTemplate:
 size: 5 # 1 leader + 4 workers = 5 pods, 5 GPUs
 restartPolicy: RecreateGroupOnPodRestart
 leaderTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-leader
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-70b
 - --tensor-parallel-size=5
 - --distributed-executor-backend=ray
 # ...
 workerTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-worker
 image: vllm/vllm-openai:v0.6.3
 # the workers join the leader's Ray cluster

LWS guarantees startup order (workers first, leader afterwards) and an atomic lifecycle (if a worker goes down, the whole group restarts, not a single pod). Without it, the coordination is manually fragile.

A simpler alternative, if all the tensor parallel GPUs fit on a single node (the case with HGX H100 boxes with 8 GPUs and an internal NVSwitch): one single Pod with nvidia.com/gpu: 5, --tensor-parallel-size=5, and vLLM handles everything internally. No Ray, no LWS, far simpler. It is the recommended path whenever it is available.

Autoscaling: the standard HPA is no use

An HPA on CPU% is useless for vLLM. The GPU does the work; the pod’s CPU sits at 5–10% even under maximum load. Nor is the GPU utilisation percentage from dcgm-exporter any use: a pod at 100% GPU% with gpu_cache_usage_perc=15% is serving one long session without being saturated, while a pod at 60% GPU% with gpu_cache_usage_perc=95% is on the edge of evicting sessions.

The right metrics are exported by vLLM itself at /metrics (Prometheus format):

Metric	What it says	When to scale
`vllm:num_requests_waiting`	Requests queued and not yet in the batch.	If it stays above 5–10.
`vllm:num_requests_running`	Active requests in the batch.	For capacity planning, not for scaling.
`vllm:gpu_cache_usage_perc`	% of the KV cache occupied.	If sustained above 80%, there is a risk of preemption.
`vllm:time_to_first_token_seconds`	Prefill latency (histogram).	If p95 exceeds your SLA.
`vllm:e2e_request_latency_seconds`	Total latency per request.	An output metric.

For the HPA to consume them there are two routes: Prometheus Adapter (exposes custom metrics to the K8s API) or KEDA (scales on Prometheus queries directly, far more convenient). With KEDA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama3-8b
 minReplicaCount: 1
 maxReplicaCount: 8
 pollingInterval: 10
 cooldownPeriod: 120 # 2 min before scale-down (long sessions)
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.monitoring:9090
 threshold: '5'
 query: |
 sum(vllm:num_requests_waiting{app="vllm-llama3-8b"})

The long cooldownPeriod matters: if you drop replicas while there are sessions decoding, you kill them. Better to leave 2 minutes of slack.

Observability: the four metrics that matter

Of everything /metrics exports, a minimal dashboard needs these four:

TTFT p50/p95 (time to first token) — what the user perceives when they hit send.
TPOT p50/p95 (time per output token) — the “speed” of the streaming.
Aggregate throughput (tokens generated per second across the cluster) — for capacity planning.
Queue depth (vllm:num_requests_waiting) — the leading indicator: if it grows, everything is about to degrade.

To that, add HBM utilisation and free memory per GPU (from dcgm-exporter) to spot bandwidth saturation and fragmentation problems. A decent Grafana dashboard with those 6 charts anticipates 90% of incidents.

Two concrete scenarios

We reuse the same hardware as the previous article for continuity. The same cache arithmetic, now with the engine on top.

Scenario A — 1×RTX 4090 (workstation or small K8s node)

Topology: 1 Pod, --tensor-parallel-size=1, 1 GPU, 1 node.
Model: up to 8B BF16 (Llama 3 8B, Qwen3 8B, Mistral 7B) or up to 14B in FP8/AWQ.
PVC: the node’s local SSD. The 4090 reads 1 TB/s from HBM; an NVMe SSD at 5 GB/s takes 5 seconds to feed 25 GB of weights into VRAM, negligible next to initialisation.
HPA: irrelevant inside the 4090 (always 1 vLLM replica per GPU), but useful across nodes: 3 replicas on 3 nodes with a 4090 each, and the K8s Service spreads round-robin.
Useful concurrency: 4–8 simultaneous sessions with 8K of context, 1–2 with 32K.
Natural use case: PoC, small teams, departmental environments, edge.

The manifest above is sized for this scenario. Changing only the model and the args, the same Deployment serves Qwen, Mistral or whatever is next.

Scenario B — 5×H100 SXM (cluster with NVLink/NVSwitch)

Topology: 1 Pod with nvidia.com/gpu: 5 on an HGX node, --tensor-parallel-size=5. If the platform does not allow grouping 5 GPUs into a single Pod, a LeaderWorkerSet with 5 pods coordinated by Ray.
Model: up to 70B BF16 (Llama 3 70B) or up to 200B+ in FP8 with cache quantisation.
PVC: NVMe attached directly to the node, or fast network storage (Ceph with a 25/100 GbE network, Lustre, GPFS). Loading 140 GB of weights over a slow network takes 5 minutes per start.
HPA: irrelevant inside the 5-GPU cluster (the 5 are an indivisible unit), but useful for adding whole HGX nodes when the load passes a given threshold. This combines with Cluster Autoscaler if the underlying infrastructure allows it.
Useful concurrency: 32–128 simultaneous sessions with mid-sized contexts, 4–16 with huge contexts.
Natural use case: internal corporate service, public exposure with an SLA, multi-tenant.

A and B, side by side

Aspect	A (1×4090)	B (5×H100 SXM)
Pod topology	1 pod, 1 GPU	1 pod with 5 GPUs (or an LWS of 5)
Maximum BF16 model	8 B	70 B
TTFT @ 8K context, idle	~250 ms	~80 ms
TPOT, idle	~30 ms/tok	~15 ms/tok
Throughput @ concurrency 16	~50 tok/s/session	~200 tok/s/session
Session drain	30–60 s	60–180 s
Useful autoscaling	Replicas on peer nodes	Whole nodes via Cluster Autoscaler
Reasonable multi-tenancy	Limited: 4–8 sessions	Comfortable: 32–128 sessions
Indicative cost (hardware)	~2 K €	~250 K € (≈ 125×)

The asymmetry is the same one as in the previous article: 125× more expensive, only ~4× more throughput per session and ~10× more concurrency. What the cluster buys is not proportional; it buys access to models an order of magnitude larger and latencies low enough for interactive use at scale. If your workload is batch or asynchronous agents where latency is not critical, several 4090s come surprisingly close.

vLLM against TensorRT-LLM and SGLang

Honestly, all three are good engines. The choice depends on practical criteria, not technical ones. A decision map, not a benchmark:

Criterion	vLLM	TensorRT-LLM	SGLang
Supported hardware	NVIDIA, AMD ROCm, Intel Gaudi	NVIDIA only	NVIDIA, AMD ROCm
Pure latency (TTFT)	Good	Best: kernels compiled for the exact hardware	Good
Aggregate throughput	Excellent	Excellent	Excellent (RadixAttention)
Deployment	Trivial: Docker image + args	Complex: build an engine per model and per GPU	Moderate
OpenAI-compatible API	Native, complete	Yes, through Triton Inference Server	Yes
Support for new models	Days after release	Weeks (recompile the engine)	Days
Quantization	AWQ, GPTQ, FP8 cache	INT4/INT8/FP8, very mature	AWQ, FP8
Multi-modal	Yes (Llava, Pixtral, Qwen-VL)	Yes	Excellent, a priority
Function calling / tool use	Good	Limited	First class
Community / release cadence	Very active, weekly	Active, NVIDIA-driven	Very active, academic
Licence	Apache 2.0	Apache 2.0	Apache 2.0

When to choose each one:

vLLM: the “boring choice” that works. The path of least friction to production. If your team has no dedicated inference serving specialist, this one. It supports varied hardware, up-to-date models, a stable API and a huge community.
TensorRT-LLM: when per-request latency is the only metric that matters and your model is stable (trained in-house, not swapped every fortnight). The price of the performance is that every model plus every GPU plus every TRT version requires an engine rebuild, and that blocks fast iteration.
SGLang: for agent-dominated workloads (heavy tool calling) or complex multi-modal. Its RadixAttention, structural caching of prompts with shared prefixes, shines in ReAct-style patterns where the same system prompt is repeated thousands of times.

For most teams starting out with on-prem LLM serving, vLLM is the right answer until you have production data pushing you elsewhere.

Common operational traps

A list of gotchas that come up again and again:

The model is downloaded on every rolling update

Symptom: every deploy takes 5+ minutes to become available. Cause: there is no shared PVC. Every new pod downloads the model from Hugging Face from scratch. Fix: a ReadOnlyMany PVC on fast storage, or a local mirror of the registry (a Pod with huggingface-cli serving a directory over HTTP). In CI/CD, hydrating the PVC before the rollout is 1 line of bash.

Short readiness timeouts that kill pods while they load

Symptom: new pods go into CrashLoopBackOff during the first model load. Cause: a readinessProbe with too low a timeout fires before vLLM finishes loading; the livenessProbe finishes the job. Fix: a startupProbe with failureThreshold: 60 or more (10 minutes of grace) before liveness starts evaluating.

An unquantised KV cache, then OOM

Symptom: the pod starts fine, serves for five minutes, then gets OOMKilled when session number five arrives with a long context. Cause: a BF16 KV cache (the default) consumes twice what FP8 does. Fix: --kv-cache-dtype=fp8. Negligible quality loss in the vast majority of cases, double the capacity.

Confusing replicas with concurrency

Symptom: the HPA scales to 8 replicas under light real load and the cloud bill goes up. Latency does not improve. Cause: someone configured targetAverageUtilization: 50% on CPU, thinking that is “load”. The reality: a single vLLM replica serves dozens of simultaneous sessions. Fix: an HPA on vllm:num_requests_waiting. If the queue is empty, one replica is enough even if the GPU is at 90%.

Tensor parallel on GPUs without NVLink

Symptom: throughput 3× worse than expected, GPUs at 30%, a lot of PCIe traffic. Cause: tensor_parallel=4 on 4 GPUs connected only by PCIe; the all-reduce saturates the bus on every layer. Fix: either the GPUs share NVLink/NVSwitch (SXM/HGX models), or you use pipeline parallel (worse latency but less all-reduce), or you reduce TP and accept that the whole model does not fit.

Sessions cut off during a rolling update

Symptom: users see truncated responses during the deploy. Cause: terminationGracePeriodSeconds: 30 (the default) is not enough to drain long generations. Fix: terminationGracePeriodSeconds: 120–180. Combined with maxUnavailable: 0, rollouts are invisible to active users.

What we have not covered (upcoming articles)

vLLM with hot LoRA adapters: serving a base model plus N tenant-specific adapters without reloading weights.
Disaggregated serving: separating prefill and decode into specialised pods, each optimised for its GPU profile.
Quantization deep-dive: AWQ vs GPTQ vs dynamic FP8 vs FP4, the real trade-offs, and when to use each.
Gateway API + AI Inference Extensions: the sigwg proposal to make LLMs first-class citizens in K8s (routing by model, sticky sessions per conversation, multi-tenant fairness).
Multi-modal serving: the same runtime, a different kind of request — images, audio, embeddings.

References

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — the original vLLM paper.
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — the paper that popularised continuous batching.
Official vLLM documentation — operational and well maintained.
NVIDIA GPU Operator — installation and troubleshooting of the GPU layer on Kubernetes.
LeaderWorkerSet — the primitive for coordinated workloads such as multi-pod tensor parallel.
KEDA — event-driven autoscaling, ideal for scaling on queue metrics.
TensorRT-LLM and SGLang — the two most serious comparables.
LMSYS Chatbot Arena — periodic benchmarks comparing the three engines.
Previous article on this blog: KV cache: the working memory that holds up LLM inference.

Kubernetes with Cilium BGP: services reachable without Ingress

Sun, 08 Mar 2026 00:00:00 +0000

One of the advantages of using Cilium as the CNI in Kubernetes is its native BGP support. It lets you advertise ClusterIPs and LoadBalancer IPs directly to the LAN router, making services reachable without needing Ingress or NodePort.

The problem

In a standard Kubernetes cluster, pods and services live on internal networks that are not reachable from outside the cluster. Getting to them requires NodePort, Ingress or an external LoadBalancer.

With Cilium BGP, the pod CIDRs and service CIDRs are advertised over BGP to the upstream router, making the whole cluster network routable from the LAN.

Work in progress — full configurations for RKE2 + CRS327 coming soon.

Kubernetes on lo0 — Tech Blog

deepagents on a cluster of your own: the SDK is free, the server is not, and what you have to build in between

TL;DR

You are here: the harness arrives after the model

The analogy: the scaffolding and the crane

What deepagents actually is

What is no longer there

Where state lives

The backends

What grows in the database

The store, which does have cleanup

The isolation that does not exist

The model, and the trap you need to know about

The tools, and the MCP that is not where it seems

The subagents

Code execution

Observability, and the package it goes through

The server: here is where the free part ends

What you have to build

How it fits with activity logging

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

TL;DR

You are here: the cross-cutting layer

The analogy: the civil registry and building permits

Part 1. What Keycloak is today

Project status

The internal architecture in five pieces

The minimum conceptual model

Part 2. Who can authenticate against it

Part 3. Humans and machines

People

Workloads with no person behind them

Keycloak against SPIFFE

Part 4. MCP, and the two missing standards

What the specification requires

What Keycloak supports

Part 5. The confused deputy

Part 6. Two hours of task against five minutes of token

Part 7. Sizing and operating

On Kubernetes

The official sizing figures

The benchmark

Security

Part 8. The mapping to the ENS

Part 9. The configuration to leave written down

Part 10. What Keycloak does not solve

Antipatterns

Startup checklist

Closing

See also

Sources

The gateway does not live alone: the JWT that does not validate the audience, the user who never reaches the trace, and the seams you have to write by hand

TL;DR

You are here: between the four pieces

The analogy: the building site with five trades

Seam 1. What brings things down and what degrades them

Postgres

Redis

The callbacks

Seam 2. Identity: the three doors and what each one validates

The first thing to know: it is paid for

The litellm_jwtauth fields that decide security

The validation that does not happen

From claim to permission

The vulnerability worth knowing about

Seam 3. Identity does not reach the trace

Seam 4. Langfuse as a neighbour

What can be automated and what cannot

The declarative bootstrap, which does exist

Langfuse credentials per team

The ingestion route

When to put a collector in

Seam 5. Postgres, Redis and the memory policy

Ports, for the NetworkPolicy

The glue you have to write

The `litellm_jwtauth` fields that decide security

A reference `config.yaml`

`prompt_caching`: why it is no use for this

`session_affinity`: the one to use