On-Premise on lo0 — Tech Blog

Real ClickHouse capacity in Langfuse v4: 2,782 bytes per observation, and why almost half of that is not data

Tue, 15 Sep 2026 09:30:00 +0200

Sixth article in the series on operating self-hosted Langfuse v4. The previous five cover what goes into a trace, putting LangGraph in front, two well-known agents instrumented, migrating with no window and the worker queues. This one measures what all of that takes up. Benchmark run on 15 September 2026 against the event schema of version 4.36.0, identical to that of 4.35.0.

TL;DR

Langfuse publishes no capacity figure at all. Not bytes per row, not events per second, not expected size. I searched their code with every reasonable combination and there is nothing. The only quantified thing is the commercial quotas of their managed service, which say nothing about disk.

One observation takes 2,782 bytes of disk. Split across the two event tables: 2,347 in the full table and 435 in the listings one. Measured over two hundred thousand observations shaped the way a real agent produces them.

41.8 % of the full table’s disk is not data, it is indexes. Four full-text indexes add up to 187 MiB against 260 MiB of data. The one covering the input costs 523 bytes per observation, almost three quarters of what the column it indexes costs.

The materialized view trims the text eight times over and saves eighteen per cent of disk. Truncating input and output to two hundred characters leaves the text at 64 MiB of the original 540, and even so the second table adds 18.5 % on top of the first. What survives the trimming is the seventy-eight-column skeleton and a metadata field that is truncated element by element, not as a whole.

In the listings table, metadata costs more than input and output together. 132.6 bytes per observation against 101.8. And its full-text index adds another 111.9, almost as much as the column.

A compression ratio quoted without saying what was compressed is worth nothing. My first pass, with repeated text, gave 38 times. With text of an entropy close to that of natural language it gives 3.68. Same schema, same indexes, same version.

Generations are 11 % of the observations and 65 % of the bytes. Their mean input is 15,655 bytes and the 95th percentile is 55,205, because each one reserialises the full history. It is the quadratic law from the second article in the series, landed on disk.

On deletion, the disk does not go down. A delete stops showing the rows and leaves the space occupied. A second step is needed, applying the deleted mask, and the cron that does it ships disabled, just like the retention one. On top of that the cleaner works table by table, so the second table keeps the rows the first one no longer shows.

And automatic retention is a paid feature when self-hosted. The data-retention entitlement is not in the open edition. Without it, the event tables grow without limit: they carry no expiry clause.

You are here: the sizing you cannot do by reading

The previous five articles describe how version 4 works and how to put it to work. The moment arrives to ask for disk, and there the series runs out of sources.

The official documentation gives a container sizing: two CPUs and eight gibibytes for ClickHouse, in Kubernetes eight of request and sixteen of limit, a hundred gibibytes of disk and three minimum replicas. It is a reasonable starting point and it does not answer the only question that matters, which is how long that disk lasts.

I went looking for it in the code, which is what this series does. I searched for events per second, rows per observation, bytes per row, capacity, benchmark, sizing, across the whole repository and its documentation. There is no figure at all. The closest thing is the ingestion quotas of their managed service, which are commercial API limits, and some working notes on a synthetic load generator that refer to the tables of the previous version and bring no results, only parameters.

So the only honest way out was to measure it. This article is that measurement, with the whole method laid out so that anyone can repeat it and contrast it with their own load.

The analogy: the archive and the index card

A municipal archive keeps case files. Each file has its folder with everything inside, and on top of that the archivist fills in an index card with the header data and the first lines of the subject, so that the file can be searched without pulling out the folder.

Intuition says the cards take up nothing compared with the folders. And that is true if you look only at the paper. It stops being true when the archivist, in order to be able to search by any word, also builds a term file: a box with one card per distinct word appearing in any case file, pointing to where it occurs. That box grows with the vocabulary, not with the number of case files, and ends up taking almost as much room as the case files themselves.

In this setup the folders are the full table, the cards are the listings table, and the term file is the full-text indexes. The article measures the three things separately, because whoever sizes by looking only at the folders falls short by almost half.

The benchmark

I want this to be reproducible, so the method goes in whole.

The schema is the real one. I extracted the ClickHouse migration DDL from the repository, in its single-instance variant. There are not three migrations but six: the three that create the two tables and the materialized view, plus one that adds three ingestion attribution columns, another that adds an n-gram index over the metadata, and another that adds three evaluation columns and two indexes. The result is 78 columns per table, eleven skip indexes in the full table and fourteen in the listings one.

I verified that this DDL is identical between the version I have pinned across the series and the current head of the repository: the diff of the migrations directory is empty.

The engine is ClickHouse 26.9.1. The minimum version 4 demands is 25.12, because of the text indexes the schema needs. I am using a newer version, and that is a difference from an installation that sticks to the minimum.

The data is synthetic and correctly shaped. The benchmark does not invent the rows: the columns and their typical values come from the worker function that builds the record. I write only into the full table, which is what the worker does, and let the materialized view populate the second one. That reproduces the real write amplification.

The topology of each trace reproduces the law we measured in the second article of the series: four fixed observations plus five for each tool call. The number of calls per turn comes from a distribution with a low median and a long tail, capped at fifteen. Generations carry the full message history, which grows with the turn, because that is exactly what the instrumentation does and it is the origin of the quadratic growth in bytes.

Result: 200,012 observations across 22,563 traces, that is 8.86 observations per trace.

And the entropy of the text is calibrated, which is the part that almost ruined the benchmark. My first pass used a repeated paragraph as filler. With that text, the full table compressed 38.1 times and the figures came out beautiful and false. A repeated lorem ipsum is the best possible input for a compressor. I redid the generator with a wide vocabulary sampled from a Zipf distribution and fifteen per cent high-entropy identifiers, calibrated to give the ratio that natural language mixed with identifiers gives.

With that text, the same table compresses 3.68 times. Ten times less. Any ClickHouse capacity figure that does not say what was compressed is noise, and mine would be too without this paragraph.

What one observation takes up

The two tables, after forcing the final merge:

	events_full	events_core
Rows	200,012	200,012
Uncompressed	957.87 MiB	328.73 MiB
Compressed	260.29 MiB	60.34 MiB
Ratio	3.68	5.45
On disk	447.68 MiB	82.92 MiB
Raw per observation	5,022 B	1,723 B
Compressed per observation	1,364.6 B	316.3 B
On disk per observation	2,347 B	435 B

The figure to take away is the last row added up: 2,782 bytes of disk per observation, counting both tables, with one replica and no retention.

And it is worth looking at the distance between the last two rows. The compressed data of the full table is 1,364.6 bytes per observation, but on disk it takes 2,347. The difference, 982 bytes per observation, is the next section.

Where the bytes go

Broken down by column, with what each one costs per observation:

Column	events_full	events_core
`input`	717.5 B	61.5 B
`metadata_values`	389.0 B	132.6 B
`output`	174.7 B	40.3 B
`tool_definitions`	28.0 B	28.1 B
`blob_storage_file_path`	12.6 B	12.5 B
`span_id`	8.1 B	8.0 B
The other 67 columns together	17.3 B	13.3 B

Three things jump out.

The first is that the skeleton is cheap. Sixty-seven columns, most of them empty in a normal observation, cost seventeen bytes between them all. ClickHouse compresses a column of empty strings down to almost nothing. Anyone who feared that the thirteen experiment columns or the telemetry ones would weigh can stop fearing it.

The second is that in the full table the input is half of everything. 717.5 of 1,364.6 compressed bytes. It is the direct consequence of each generation reserialising the history.

The third is the one I was not expecting. In the listings table, metadata costs more than input and output together: 132.6 against 101.8. The reason lies in how the materialized view truncates. It trims input and output to two hundred characters each, but it trims the metadata element by element: an array with forty values can survive with eight thousand characters. The table that exists in order to be light fills up with the one thing its truncation does not control.

The split by observation type

Type	Observations	Mean input	Input p95	Mean output
GENERATION	21,952	15,655 B	55,205 B	1,077 B
AGENT	22,563	2,585 B	6,013 B	1,300 B
TOOL	21,952	319 B	319 B	2,008 B
CHAIN	67,689	413 B	413 B	20 B
SPAN	65,856	272 B	272 B	206 B

Generations are 11 % of the observations and contribute 65 % of the input and output bytes. The 95th percentile of their input is 55 kilobytes, in a benchmark where no conversation goes beyond fifteen tool calls.

This has a sizing consequence worth saying out loud: the size of the database is not determined by the number of requests, it is determined by the depth of the turns. Doubling the users doubles the disk. Doubling the tool calls per turn multiplies it by rather more than two.

The indexes, which are almost half

Here is the finding that changes a sizing exercise the most.

	Data	Indexes	% of disk in indexes
events_full	260.29 MiB	187.31 MiB	41.8 %
events_core	60.34 MiB	22.49 MiB	27.1 %

The marks and the primary key are irrelevant: thirty-two kilobytes and three hundred bytes respectively, with the granule size of 64 mebibytes the full table has declared.

Broken down, and ordered by what they cost:

Index	Table	Bytes per observation
`idx_fts_input_low`	events_full	523.1
`idx_fts_metadata_values`	events_full	319.4
`idx_fts_output_low`	events_full	133.5
`idx_fts_metadata_values`	events_core	111.9
`idx_fts_metadata_names`	both	4.3
`idx_span_id`	both	1.2
The remaining Bloom filters	both	0.1 each

The full-text index over the input costs 523 bytes per observation when the column it indexes costs 717. That is, indexing the input costs almost three quarters of what storing it costs. And the metadata one in the listings table costs 111.9 when its column costs 132.6: eighty-five per cent.

The Bloom filters, on the other hand, are free as far as sizing goes. The four identifier ones and those for model name, experiment and evaluator add up to less than two bytes per observation between them all.

This is not a criticism of the design. Those indexes are what make text search in the interface respond, and without them the experience would be another thing. It is sizing information that is published nowhere: when asking for disk for ClickHouse you have to multiply what the columns say by 1.7.

The materialized view as an amplifier

The second table exists so that listings do not have to touch the large fields. It truncates input and output to two hundred characters and does without the text indexes over them.

The trimming works: the benchmark’s 540.55 MiB of input and output come down to 64.68 MiB, eight and a half times less.

And even so the second table adds 18.5 % of disk on top of the first. For the reasons already seen: the skeleton survives, the metadata truncated element by element survives, and its own text index over that metadata costs another 111.9 bytes.

It is worth keeping in mind because it is counter-intuitive. You would think a table storing two hundred characters of text is an appendix. It is four hundred and thirty-five bytes per observation, almost a fifth of the total, and it is paid always, because the materialized view cannot be switched off without leaving the interface with no listings.

What happens on deletion

The retention cron runs a lightweight delete. I reproduced it: I deleted the observations older than a date, which removed 67,395 rows of the 200,012.

Moment	Visible rows	Rows in the parts	Disk
Before	200,012	200,012	447.68 MiB
After the delete	132,617	200,012	447.68 MiB
After applying the mask	132,617	132,617	299.11 MiB

The delete frees not one byte. The rows stop being visible and the space stays occupied. The disk only goes down at the second step, applying the deleted mask, which frees 33.2 %.

And that second step is run by a different cron that ships disabled, just like the retention one. With the default values, whoever switches retention on in the interface will see the traces disappear from the view and will never see the disk go down, until an ordinary merge drags in the affected parts, which can take however long it takes.

There is a third detail. The cleaner works table by table, with an independent deleter for each one. In my test I deleted in the full table and the listings one kept its 200,012 rows, because the materialized view does not propagate deletes. In a real installation both deleters run, but they are two separate operations that can diverge if one fails.

On top of this you have to add what the fourth article already covered: neither of the two tables carries an expiry clause. There is no automatic expiry in the engine. The whole lifecycle depends on those crons. And per-project retention requires an enterprise edition entitlement, so in the open edition the only retention possible is the one you write yourself.

Projection

With the measured 2,782 bytes per observation, one replica and no retention:

Observations per day	Agent turns per day	Per day	Per month	Per year
100,000	~11,300	0.26 GiB	7.8 GiB	94.6 GiB
1,000,000	~112,800	2.59 GiB	77.7 GiB	945.6 GiB
10,000,000	~1,128,000	25.91 GiB	777.2 GiB	9,456 GiB

The conversion to turns uses the benchmark’s 8.86 observations per trace, which depend on the depth of the agents. With deeper agents, the same turn figure produces more observations.

And there is still a multiplication left. Langfuse recommends three minimum replicas of ClickHouse in Kubernetes. With one million observations a day and three replicas, a year is 2.8 tebibytes of raw disk. The official sizing of a hundred gibibytes per replica covers about thirty-eight days at that rate.

That is the number that is published nowhere and that has to go into the spreadsheet before committing to a service.

What this benchmark does not measure

For the sake of honesty, and because anyone repeating this is going to get different numbers:

The data is synthetic. The shape of the row and the topology of the traces are taken from the code and from the measurements in the previous articles, but the text is generated by a sampler, not by a model. I have calibrated its entropy so that compression resembles that of natural language, and that calibration is the largest source of error in the whole article. A load with a lot of structured JSON will compress better; one with a lot of code or a lot of base64, worse.

It is a single instance, with no replication. In a replicated installation there is a different engine and there are copies.

It is ClickHouse 26.9.1, above the 25.12 minimum. The text index implementations have changed between versions and I have not compared.

It does not measure performance: not ingestion per second, not query latency, not the CPU cost of merges. Disk only. The merge cost of full-text indexes is a topic with a life of its own and is left for the saturation runbook.

And it does not measure object storage, which in version 4 is mandatory and keeps a copy of the original event. That is additional disk this benchmark does not touch.

Checklist

Do not size using the official container sizing: it covers CPU and memory, not disk growth.
Count 2,782 bytes per observation as a starting point, and adjust by measuring your own load.
Multiply by the number of replicas before asking for the volume.
Multiply what the columns suggest by 1.7, because the indexes are 41.8 % of the full table’s disk.
Measure your own installation with the system tables instead of trusting this article: active parts for the total, columns for the breakdown, skip indexes for what is not data.
Watch the metadata, which is the only thing the materialized view’s truncation does not control, because it trims each element and not the whole.
Size by turn depth, not by number of requests.
Switch on both crons, the retention one and the deleted mask one, and check that the disk really goes down after the second.
Reckon on per-project retention requiring a licence, and budget the work of writing it if you do not have one.
Remember that the tables have no expiry in the engine: with no cron, they grow for ever.

Traps and things that are not what they look like

Langfuse publishes no capacity figure at all, neither in its code nor in its documentation.
A compression ratio without saying what was compressed means nothing: the same schema gives 38 times with repeated text and 3.68 with realistic text.
41.8 % of the full table’s disk is indexes, not data.
Indexing the input costs almost as much as storing it: 523 bytes per observation against 717.
The 78-column skeleton is irrelevant: seventeen bytes per observation across the sixty-seven small columns.
In the listings table metadata costs more than input and output together.
The materialized view’s truncation trims each element of the metadata, not the whole array.
The listings table is not an appendix: it adds 18.5 % of the first table’s disk despite storing two hundred characters.
A delete frees no disk. The mask has to be applied, and that cron ships disabled.
The retention one also ships disabled out of the box.
The cleaner works table by table; the materialized view does not propagate deletes.
The event tables carry no expiry clause.
Per-project retention is a paid feature when self-hosted.
Generations are 11 % of the observations and 65 % of the bytes.
Bloom filters are free as far as disk goes; the full-text ones are not.

Closing

What I take from this benchmark is not the figure, although the figure was needed. It is that the split is not where you look for it.

When somebody sizes a trace database, they look at the size of the prompts. It is the natural thing to do: they are the large fields and they are the ones you see in the interface. And it turns out that in this architecture the prompts are half the compressed data, and the compressed data is a little over half the disk. The rest is search structures nobody counts because they do not show up when you look at a row.

There is a pattern there that goes beyond this tool. An observability database is, by definition, a database queried in unforeseen ways: by free text, by user, by session, by whatever is needed when something goes wrong at three in the morning. That ability to search by anything is paid for in disk, always, and it is paid at write time and not at search time. Langfuse’s design pays it fairly explicitly, with four full-text indexes declared in the migration, in plain sight for anyone who wants to read them.

What was missing is for somebody to put the number alongside. Now it is there, with its margin of error and its method on the table, so that the next person to measure it can tell me where I got it wrong.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost (this article): how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse, migraciones de ClickHouse (packages/shared/clickhouse/migrations/canonical/, ficheros 0039 a 0047): https://github.com/langfuse/langfuse.
Langfuse, escritura del worker en ClickHouse (worker/src/services/IngestionService/, worker/src/services/ClickhouseWriter/): https://github.com/langfuse/langfuse.
Langfuse, limpiador de retención por lotes y limpiador de máscaras de borrado (worker/src/features/batch-data-retention-cleaner/, worker/src/features/deleted-mask-cleaner/): https://github.com/langfuse/langfuse.
Langfuse, guía de autoalojamiento y requisitos de infraestructura: https://langfuse.com/self-hosting.
ClickHouse, tablas del sistema parts, columns y data_skipping_indices: https://clickhouse.com/docs/en/operations/system-tables.
ClickHouse, índices de salto y de texto completo: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree.
ClickHouse, borrados ligeros y máscara de borrado: https://clickhouse.com/docs/en/guides/developer/lightweight-delete.

deepagents on a cluster of your own: the SDK is free, the server is not, and what you have to build in between

Tue, 15 Sep 2026 07:00:00 +0200

Opening of the agentic platform vertical. The previous articles dealt with the pieces separately: the inference gateway, its second door, what a fleet of agents costs and isolation. This one deals with what goes on top. Verified against deepagents 0.7.14 and the LangGraph repository, commits of 14 September 2026.

TL;DR

deepagents is not a runtime. It is a factory that assembles middleware and delegates to LangChain’s agent factory. It returns a compiled LangGraph graph and nothing else. The repository’s own documentation says so without decoration: it introduces no new runtime. That is good news for whoever deploys it, because it runs wherever Python runs.

The planning tool no longer exists. It disappeared in 0.7.0 as a breaking change, along with the state channel that backed it. Almost everything written about this harness still describes it as one of its three legs. Today you have to ask for it explicitly, and you have to ask for it in each subagent too.

With the default configuration, the agent’s files live inside the checkpoint. The state backend stores the full content, binaries in base64, in the same place the conversation is persisted. Changing that is one line and it is the first architecture decision of the deployment.

The database grows per loop step, not per turn. Four tables, one row per superstep, one per channel and version, another per task write. And of the four cleanup operations the base class declares, the Postgres implementation has only one: delete an entire thread. The other three raise not-implemented.

Pruning by hand can empty the history silently. The code itself warns that a partial delete breaks the delta chain and leaves the channels rebuilding empty without raising any error. The only safe pruning is by whole thread.

Tenant isolation is the thread identifier, and there is no owner check. Whoever can send a thread identifier reads that thread. The on-disk filesystem backend fixes its root when the agent is built, so it separates nobody. And the documentation’s example for computing the per-user namespace does not work outside the managed platform.

There is a configuration trap that leaves a short-window model uncompacted. If the model does not expose its window, the compaction threshold falls back to 170,000 tokens. Served on a vLLM with a 32,768 window, the agent blows up on context length long before it compacts. The failure is silent and is fixed by passing the profile by hand.

Observability without the platform does exist, and it goes through a package that is not obvious. LangChain’s core does not emit OpenTelemetry: zero references. The exporter lives inside the LangSmith SDK, and there is a mode that emits over OTLP only, without talking to the service. It is the road to a Langfuse of your own, and it is worth knowing where it passes.

And here is where the free part ends: the server. The SDK, the graph engine and the checkpointers are MIT. The server that publishes them over HTTP carries an Elastic 2.0 licence, asks for a commercial licence key for production, adds Redis to the stack and reports metadata to an external endpoint unless there is an isolation agreement. The development tool is explicitly for development.

What is left is a list of things you have to write. Per-thread authorisation, retention, a run queue, cancellation, clean draining and migrations. None of them is hard. All of them are work, and none of them appears in the twenty-line front-page example.

You are here: the harness arrives after the model

The sequence repeats on every project. First you set up inference, which is the visible part. Then the gateway, because you need keys and budgets. Then observability, because nobody knows what is going on. And when all of that works, somebody asks why the assistant cannot do multi-step tasks, and the word agent enters the scene.

At that point the temptation is to write the loop by hand, and for two weeks it looks like a good idea. The loop is easy. What is not easy is everything you discover afterwards: what happens when the context fills up, where the intermediate files are stored, how you resume a task that was left half done, how you ask a human for confirmation without losing state.

An agent harness is exactly that: the packaged answer to those questions. And choosing one looks like a library decision, about the size of choosing an HTTP client. It is not. It decides where your users’ state lives, which database grows, what can be isolated and what cannot, and in the case at hand, whether there is a commercial licence waiting at the end of the road.

This article looks at deepagents with those eyes. It is not a usage guide, and I do not assess whether its agents are any good. I look at what it forces you to build.

The analogy: the scaffolding and the crane

A building site needs scaffolding and it needs a crane, and they are not the same thing even though both are metal structures surrounding the building.

You put up the scaffolding yourself with parts you bought. It adapts to the façade, it can be extended, and the day the work finishes you take it down and put it away. Nobody charges you for using it once it is bought.

The crane is another matter. You hire it, it comes with an operator, it has a serial number and you have to notify somebody when it goes up. Without a crane the work still moves forward, more slowly and with more people, but it moves forward.

Here the scaffolding is the SDK and the graph engine, and it belongs to whoever buys it. The crane is the server that exposes the agent over HTTP, and that one comes with a contract. This article is about where exactly the line between the two sits, because you cannot see it in the brochure, and about how much work can be done without a crane.

What deepagents actually is

The first thing to get out of your head is that it is a system. It is a function.

The main factory assembles a list of middleware and calls LangChain’s agent factory, which is what builds the graph. What it returns is a compiled graph, with an added configuration on top that raises the recursion limit to a very high number and adds an integration tag in the metadata. The repository’s own architecture document states it: no new runtime is introduced.

Compiled with the default options, the graph has the nodes you would expect from a reasoning-and-action loop: start, a prior node that repairs incomplete tool calls, the model node, the tools node and the end. The conditional edges run from the model to tools, to the model itself or to the end. There is nothing more. Everything that looks like a system is middleware that adds nodes only when it declares hooks of its own.

The tools that ship by default are the filesystem ones, plus the subagent delegation one: list, read, write, edit, delete, search by pattern, search by content, execute, and the task tool.

The good consequence of this is portability. If something is a LangGraph graph and nothing else, it runs in any Python container, with the checkpointer you choose, against the model you choose. There is no hidden service behind it.

What is no longer there

It deserves its own section because it contradicts almost everything published. The planning tool, the one that wrote a task list and kept it in a state channel, was removed in 0.7.0 as a breaking change declared in the changelog. The tool is gone, the channel is gone, and the prompt fragment that went with it is gone.

It survives as LangChain middleware, and you have to ask for it by hand. With one detail that bites: you have to ask for it in each subagent too, because each subagent assembles its own stack. All that remains from the previous stage are archaeological leftovers, such as the state key that is still on the list of keys excluded when passing context to a subagent.

The plan, today, rests on files. Which is consistent with the rest of the design, and is another reason to look closely at where those files live.

Where state lives

This is the first architecture decision and the one that is most expensive to get wrong.

The agent’s state has two large channels. Messages, with a reducer of its own that deduplicates by identifier and treats deletions as tombstones. And files, with its own reducer where a null value means deleted. Both use a channel type that writes deltas and stores a full snapshot every fifty updates, and the code comment explains why: to take checkpoint growth from quadratic to linear with respect to the number of messages.

That channel type is marked beta in its own code, with a warning that the on-disk representation may change. Worth knowing before you rest a platform on top of it.

The backends

There is a common interface and several implementations. The ones that matter for deciding:

Backend	Where the files live	Persistence
State	Inside the graph state, that is, inside the checkpoint	Per thread, in the checkpointer’s database
Filesystem	In a real directory anchored to a root	Wherever you mount the volume
Store	In LangGraph’s store, with a namespace	Crosses threads, with a time to live
Composite	Routed by path prefix among the above	Mixed

The default backend is the state one. That is: by default, everything the agent writes ends up inside the checkpoint, and binaries go base64-encoded inside the same JSON. For a demo it is convenient, because you have nothing to set up. For a platform with real users it is a sizing decision taken by omission.

There is a detail that rounds the matter off. The filesystem middleware automatically evicts large tool results to a file when they exceed a token threshold, so that they do not occupy the context window. It is a good mechanism. But with the state backend, evicting to a file means moving the content from one place in the checkpoint to another place in the same checkpoint. It leaves the model’s window and it does not leave the database.

What grows in the database

The Postgres checkpointer creates four tables: one for migrations, one for checkpoints, one for blobs by channel and version, and one for writes by task. The primary keys all start with the thread identifier, and there is an index per table on that column.

What you have to internalise is the rhythm. One checkpoint row is written per engine superstep, not per conversation turn. A turn with four tool calls is several supersteps. None of that is recycled on its own.

And here comes the uncomfortable part. The checkpointer’s base class declares four maintenance operations: delete a thread, delete by run, copy a thread and prune with a strategy. In the Postgres implementation only the first is implemented. The other three inherit the not-implemented exception from the base class. Deleting a thread is three delete statements by identifier, and that is it.

In other words: retention is yours to build. And with a care that the code itself documents and that is worth quoting in full, because it is the kind of warning you read after the incident. A naive pruning that removes intermediate checkpoints and their writes can cut the delta chain; the surviving checkpoint is rarely a snapshot point, so its channels would rebuild empty without any error being raised.

Translated into operations: the periodic sweep deletes whole threads and only whole threads. Never loose rows. And since the checkpoints table has no date column, to know which thread is old you have to keep count outside or rely on the checkpoint identifier being a time-sortable UUID.

There is a variant of the checkpointer that stores only the last state and retains no history, which caps growth at the root. With one important reservation to verify before using it: it is not clear that it is compatible with the delta channels this harness uses by default for messages and files. Combining the two without checking is asking for a silent problem.

The store, which does have cleanup

Curiously, the sibling piece is better resolved. The store has a configurable time to live, with refresh on read, skipping of expired entries and a sweep interval, and the sweep is a delete by expiry date.

Two warnings. The sweeper is a thread inside the process, so with several replicas there are several sweepers competing for the same delete; it is idempotent, but it is free contention and you avoid it by switching it off and putting the delete in a scheduled cluster job. And the store’s semantic search requires the vector extension in the database, which constrains the managed Postgres image, which normally does not carry it.

The isolation that does not exist

If the platform is going to have more than one tenant, this is the section that decides the deployment topology.

The only thing separating one thread from another is the thread identifier. It is the first column of the primary key in all three tables. There is no owner check anywhere in the checkpointer. Whoever manages to send a thread identifier in the call configuration reads that thread. Authorisation is the responsibility of the layer you write on top, and since that layer also has to be written, it is worth noting down now.

By backend, the situation differs:

The state one inherits thread isolation, which is enough if the HTTP layer validates that the user owns the thread.
The store one computes the namespace per call, through a function that receives the runtime context. It is the correct option for multi-tenant.
The filesystem one fixes its root in the constructor, when the agent is built. There is no path in the code that recomputes it per request, per thread or per tenant. Sharing one deployment between tenants with this backend means everybody sees the same directory.

On that last point, do not confuse it with the protection that does exist. Virtual mode, which now comes enabled by default, blocks paths that escape the root. That protects against directory traversal. It does not protect against the neighbour. The project’s own threat document admits it: that mode exists to support the composite backend’s prefix routing, not as a security boundary.

There is also a documentation trap worth flagging because it costs an afternoon. The example the documentation proposes for computing the per-user namespace reads the identity from a server-information field of the runtime context. That field is annotated in the LangGraph code as metadata injected by the managed server, and null when running open-source LangGraph without managed deployments. On a cluster of your own with an HTTP layer of your own, that example fails. What you have to use is the context you fill in yourself, with the identifier that comes validated from the identity provider.

And one more boundary that gets crossed without warning: the memory and skills middleware interpolate file content into the system prompt as is, without sanitising. If two tenants share a directory that serves as the source for that, the first writes instructions that the second’s agent executes. It is prompt injection through shared storage, and it is avoided with the same measure as everything above: do not share the filesystem backend between tenants.

The topology conclusion is short. Real multi-tenant with files means one deployment per tenant, with its namespace, its volume and its quota, or else the store backend with a namespace computed per request. There is no third way.

The model, and the trap you need to know about

The default model is a specific commercial provider, and it is marked deprecated: passing an empty model warns and will stop working. You can inject any LangChain chat model, which is what anyone serving their own weights will do.

With two integration details that are not obvious.

The first. To point at an OpenAI-compatible endpoint, which is how a vLLM behind a gateway looks, you have to pass the already-built instance, not the string with the provider prefix. With the string, a provider profile is applied that forces the use of the responses API, which a vLLM normally does not implement. Passing the built object applies no profile and the agent stays clean.

The second is the trap, and in my view it is the most likely configuration failure in the whole deployment.

The compaction middleware computes its thresholds from the model profile. If the model declares its input window, it compacts on reaching 85 % of the window and keeps 10 %. If it does not declare it, it falls back to a fixed value: 170,000 tokens.

The chain that leads to not declaring it is the usual one in a deployment of your own. The OpenAI client resolves the profile by looking the model name up in a static table of the provider’s models. A model served as qwen3-30b-a3b is not in that table, so the profile comes out empty, and empty becomes null. The method that assigns it swallows any exception, so there is no warning.

Result: a vLLM with a 32,768-token window will not compact until 170,000. The server will return a context-length error long before that, and it will do so intermittently, only when conversations get long. There is a safety net that catches the error and trims, but that is reactive recovery after a failed call, not planning.

The fix is one line, because the profile is a public field and only fills itself in when it is empty: you pass it when building the model, with a value below the real window to leave room for generation. And it deserves an assert at start-up, because it is the kind of thing nobody looks at until it fails.

A related sizing note. With the default thresholds and a 32,768 window, compaction would trigger at 27,852 tokens, while eviction of large tool results is fixed at 20,000. That is, a single tool result can occupy almost the whole budget before anything evicts it. With 128,000 windows the defaults are proportionate; below 64,000 you have to lower those two thresholds by hand.

The tools, and the MCP that is not where it seems

This harness has its filesystem tools and its delegation tool, and the rest are passed as an ordinary list of LangChain tools.

The fact that matters to anyone with an MCP gateway in place: the SDK does not integrate MCP. There is no reference to the protocol in the package, nor any dependency on adapters. The integration lives in the command-line tool that accompanies the project, which does depend on the MCP adapters for LangChain, resolves configurations in the style of the desktop clients, and turns each remote tool into a LangChain tool that then goes through the ordinary list.

So the path exists and is proven, but you have to walk it yourself: load the gateway’s tools with the adapters and pass them in the list. Which, incidentally, fits well with what we already know about the catalogue: the credential-based filtering the gateway does is applied in the listing, so each agent receives the subset that belongs to it without the harness having to know anything.

The subagents

They are declared in three ways and invoked with a task tool that takes a description and a type. What you need to know to design with them:

The subagent receives the parent’s state except the messages and a few private keys, and it starts with a single human message which is the task description. Files are inherited, and it shares the same backend object, so parent and child work on the same logical filesystem. The result comes back as a tool message with the structured response or the child’s last text.

Depth is effectively one level. Verified by compiling the graph: the stack assembled for a declarative subagent does not include the subagent middleware, so the child does not have the task tool and cannot delegate in turn.

And there is no concurrency limit of its own. The parallelism is that of the engine’s parallel tool calls, and the task tool’s prompt explicitly invites launching them in parallel. Anyone with a per-tenant GPU quota will want to put the brake where it belongs, because it is not here.

Code execution

The sandbox protocol is well defined and the packaged implementations are nearly all paid services: four commercial providers, plus an embedded JavaScript interpreter and a sandbox from the platform vendor itself.

That leaves a local shell backend that is worth looking at closely before considering it. It runs the command with the system shell, and its own docstring lists what it does not do: no isolation, no process separation, no resource limits, and it lists production and multi-tenant environments as inappropriate use cases. It also has an option to inherit the process environment which, enabled, hands the model every variable, including the gateway credentials and the database connection string. It comes disabled, and it is an easy foot to shoot.

The reasonable way out on a cluster of your own is to implement the protocol yourself, and it is smaller than it looks. The sandbox base class requires four things: an identifier, execute, upload files and download files. Everything else, list, read, write, edit, delete and search, is built by the base class on top of execute. Looking at one of the commercial implementations, it is on the order of a hundred and fifty lines.

One image requirement that constrains the Dockerfile and is worth knowing beforehand: the base class helpers inject small encoded Python scripts to resolve searches and checks, so the sandbox image needs Python and a POSIX shell or half the calls fail.

On how to implement it, the shape that fits a Kubernetes of your own is an ephemeral container per session with a bounded time to live, resource limits, no service account token mounted and a network policy that only allows egress towards the inference gateway. Execute is resolved against the cluster’s exec API, and upload and download through the same channel. The alternative with less code is an isolated runtime class for the agent pod, one pod per tenant, and the shell backend inside; you lose multi-tenancy in a single pod and you have nothing to write.

Observability, and the package it goes through

Here there is a fact that surprises and that is worth being clear about before designing the trace pipeline.

LangChain’s core does not emit OpenTelemetry. Zero references in the whole package. Its native tracer talks to the vendor’s observability service and that is that.

The OTLP exporter does exist, but it lives inside that service’s SDK. And there is the good news: that SDK has a tracing mode that emits only over OTLP, without talking to the service, reading the destination and headers from OpenTelemetry’s standard environment variables. The transport is HTTP with protobuf, which is exactly what a self-hosted Langfuse accepts on its ingestion route.

That is: to take the agent’s traces to your own observability you have to install the vendor’s package and ask it not to talk to the vendor. It works, it is supported and it is documented, but it is a dependency you have to declare in the supply chain analysis, not a configuration detail. And if the OpenTelemetry packages are missing, it emits a warning and silently stops tracing, so start-up should check for it.

There is the alternative of the callback handler that Langfuse itself publishes, with fewer pieces. The practical difference is correlation: over the OTLP route the agent’s spans share a trace identifier with the gateway’s and the inference engine’s, and you see a whole request end to end. Over the callback route, you do not. For anyone who already has a collector deployed, the OTLP route is the one that pays.

The server: here is where the free part ends

Everything above is an architecture decision. This is a licence decision, and it is the one that gives the article its title.

The harness SDK is MIT. The graph engine is MIT. The memory, SQLite and Postgres checkpointers are MIT. All of that runs anywhere, with no key, no registration and without calling anybody.

The server that exposes a graph over HTTP is another matter. The package that implements it carries an Elastic 2.0 licence, whose relevant clauses forbid offering the software to third parties as a managed service and forbid modifying or circumventing the licence key functionality. The command-line tool’s own command prints the conditions: for local development it asks for a service API key, and for production use it asks for a licence key in an environment variable.

There is also usage reporting. The server code contains a beacon endpoint to which it sends metadata, with a header carrying the licence key. The self-hosting documentation confirms it from the other side, listing among the requirements network egress to that domain for licence verification and usage reporting if not running in isolated mode. Isolated mode is a contractual variant, not a box you tick.

And it drags infrastructure along: besides Postgres, it asks for Redis.

The development tool, which is the one that appears in every tutorial, describes itself as development mode with hot reload and an in-memory server. It is not a production server and it does not claim to be.

None of this is a reproach. It is a legitimate and fairly common business model: permissive core, commercial control plane. What is not legitimate is finding out late. Anyone building a platform with a sovereignty requirement, with no dependencies on external services and with the argument that the whole stack is open, has to know that the piece that turns the graph into a service does not meet that requirement.

What you have to build

The way out is to write the HTTP layer yourself, and the good news is that it is boring. A service that compiles the graph once with the checkpointer pointing at the database, and exposes an endpoint that validates the identity provider’s token, derives the thread identifier and the tenant context, and invokes the graph. Event streaming if needed.

What you additionally have to write, and which the commercial server gave you ready-made:

Per-thread authorisation. The engine does not have it. It is the most important and the easiest to forget, because nothing fails if it is missing.
Retention and deletion. A scheduled job that deletes whole threads, never loose rows, for the reasons in the state section.
Migrations. Schema setup has to be called explicitly and needs data-definition permissions, so it goes in a separate job with a different role from the runtime one. And since it creates indexes concurrently, it cannot go through a connection pooler in transaction mode; point it at the direct write service.
Health probes and draining. The engine has a cooperative draining mechanism, but wiring it to the container’s termination signal is your own work.
Run queue and cancellation, if you need long background runs. This is where the absence is most felt, and the reasonable answer is a queue on the same database rather than another system.
Durability decided deliberately. There are three modes: persist before the next step, persist in parallel, or persist only on exit. The last one is no good on Kubernetes: if the pod dies, the whole turn is lost.

On that last point it is worth being explicit, because it affects tool design. The engine stores each task’s writes as soon as it finishes, and on resume it reapplies those that were already there and does not repeat them. A tool that finished is not run again. A tool that was in flight when the pod died is, because its write never landed. The semantics are at-least-once, so every tool with an external effect has to be idempotent or carry an idempotency key. There is no exactly-once.

And the same applies to interrupts for human confirmation: on resume, the node is re-executed in full, so any side effect that sat before the interrupt inside the same node is repeated. The rule is that the interrupt goes first.

How it fits with activity logging

Two things about this setup touch compliance head on, and both come out of earlier sections.

The first is that the checkpoint is, de facto, a repository of personal data. It contains the whole conversation and, with the default backend, the files the agent has written. If the platform serves identifiable people, that database needs encryption, retention of its own and separate access control, exactly like the gateway’s spend table. The difference is that here there is no redaction switch that helps: the state is the state.

The second is traceability. An agent that delegates to subagents and executes tools produces a chain of actions that you have to be able to reconstruct, and the only place where that chain is complete is the trace. Traces going out over OTLP to your own observability stops being a technical preference and becomes the logging mechanism. It is worth designing it that way from the start, and not adding it afterwards.

As always in this series, the annex’s specific codes have to be checked against the current text before taking them into a compliance document; the approach is developed in the technical controls article.

Checklist

Decide the filesystem backend before the first demo, because the default one puts the content in the database.
Pass the model profile by hand with the real window, and check it at start-up.
Lower the eviction thresholds if the model’s window is below 64,000 tokens.
Build the model as an instance, never as a string with a provider prefix, if there is an OpenAI-compatible endpoint behind it.
Write per-thread authorisation before opening the service to more than one user.
One deployment per tenant if you use the filesystem backend; a namespace computed per request if you use the store one.
Compute that namespace from your own context, not from the server-information field, which is null outside the managed platform.
A scheduled retention job that deletes whole threads and only whole threads.
Schema migrations in a separate job, with a role of their own and against the direct write service.
Synchronous durability, and tools with external effects made idempotent.
Traces over OTLP with the mode that does not talk to the vendor’s service, and a start-up check that the packages are there.
Put a concurrency limit on subagents if there is a GPU quota involved.
If you need code execution, implement the sandbox protocol against an ephemeral container; the local shell backend is advised against by its own documentation.
Note in the supply chain analysis the hard dependencies on commercial providers that the package drags in even when unused.

Traps and things that are not what they look like

The planning tool was removed in 0.7.0. You have to add it by hand, and in each subagent too.
The default backend stores files inside the checkpoint, binaries included in base64.
Evicting a large result to a file does not get it out of the database with that backend, only out of the context window.
One checkpoint is written per superstep, not per turn.
Of the four declared cleanup operations, only delete-a-thread exists in Postgres.
A partial pruning can empty the history without raising any error.
The delta channel is in beta and its on-disk representation may change.
There is no owner check in the checkpointer. The thread identifier is not a credential.
The filesystem backend’s root is fixed when the agent is built and it separates no tenants.
The documentation’s example for the per-user namespace does not work outside the managed platform.
The filesystem middleware and the subagent middleware cannot be removed.
If the model does not declare its window, compaction does not trigger until 170,000 tokens, silently.
Passing the model as a string with a provider prefix forces an API that a vLLM does not implement.
The SDK does not integrate MCP; the integration lives in the command-line tool.
Subagents do not delegate: depth is one level.
LangChain’s core does not emit OpenTelemetry; the exporter lives in the vendor’s service SDK.
The HTTP server carries an Elastic 2.0 licence, asks for a licence key for production and reports usage unless there is an isolation agreement.
The development tool is for development, with an in-memory server.
Resumption is at-least-once for in-flight tools.

Closing

The question this article started with was whether deepagents fits into a platform of your own. The answer is that it does, and that it fits better than I expected, precisely because it is less than it appears. A middleware stack on top of a graph is easy to host, easy to understand and easy to replace. What does not fit is the piece next to it.

That asymmetry is the pattern I have been seeing all year across the ecosystem, and it deserves naming. The core is published under a permissive licence because the core is where you compete for adoption. The control plane is published under a restrictive licence because the control plane is where you charge. It happened with the gateways, it is happening with observability, and it happens here. Anyone building a sovereign platform cannot assess a whole project by the licence of its main repository: you have to look package by package, and look at which of them is the one you are going to need the day the thing has users.

The good part is that the bill, in this case, is paid in work and not in money. What you have to build to do without the commercial server is an HTTP service with no mystery to it, a retention job, a migration job and an authorisation layer. None of that is research. All of it is a week of an engineer who knows what they are doing, and in exchange you are left with a system that deploys the same way in a school as in a classified datacenter, with no internet egress and with nobody counting runs on the other side.

What I do recommend is making that list before the first demo and not after. Because the front-page example works in twenty lines, and the twenty lines have the default backend, the model without a profile and no authorisation at all.

Sources

langchain-ai/deepagents, código y documento de arquitectura: https://github.com/langchain-ai/deepagents.
langchain-ai/deepagents, registro de cambios de la rama del SDK (eliminación de la herramienta de planificación en la 0.7.0): https://github.com/langchain-ai/deepagents/blob/main/libs/deepagents/CHANGELOG.md.
langchain-ai/langgraph, motor, canales y checkpointers: https://github.com/langchain-ai/langgraph.
langgraph-checkpoint-postgres, esquema de tablas y operaciones de mantenimiento: https://github.com/langchain-ai/langgraph/tree/main/libs/checkpoint-postgres.
langgraph-checkpoint, clase base del checkpointer y aviso sobre poda con canales de deltas: https://github.com/langchain-ai/langgraph/tree/main/libs/checkpoint.
LangChain, despliegue de servidor autónomo (requisitos de licencia, Postgres, Redis y salida de red): https://docs.langchain.com/langsmith/deploy-standalone-server.
LangChain, trazado con OpenTelemetry: https://docs.langchain.com/langsmith/trace-with-opentelemetry.
Elastic License 2.0: https://www.elastic.co/licensing/elastic-license.
Langfuse, integración nativa OpenTelemetry (ruta de ingesta y autenticación): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, integración con LangChain por retrollamadas: https://langfuse.com/integrations/frameworks/langchain.
langchain-ai/helm, chart del servidor: https://github.com/langchain-ai/helm.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.

Making the model prefer our tools: the priority that exists in no layer, the two primitives that do, and the `_meta` that crosses the gateway

Tue, 15 Sep 2026 03:30:00 +0200

A follow-up to LiteLLM’s MCP gateway, which treated the tool catalogue as cost and as surface. This one treats the next question, the one that arrives in the meeting afterwards: so then, how do I get the model to use mine. Verified against LiteLLM 1.102.0 and MCP specification 2026-07-28, with a test of our own run on 14 September 2026.

TL;DR

Tool priority does not exist as a concept in any layer of the stack. Not in the specification, not in the clients, not in the providers’ APIs. What exist are two distinct primitives: removing the alternative from the catalogue, and leaving it behind a search step. Everything else is persuasion, and the model ignores it when it suits.

The specification has no ranking field. The Tool object is name, title, icons, description, schemas, five annotations that are unreliable hints by their own declaration, and _meta. There is a trap I see repeated: MCP does define a numeric priority, but in another structure, the one that applies to content blocks, resources and prompts. It does not govern tool selection.

The only filtering the specification blesses is by credential. The current revision says the set of tools must not vary by connection, and that it may vary according to the authorisation presented. That is exactly what a gateway with per-key lists does.

The client is the one that decides, and only one of them has anything resembling priority. Claude Code lets you exempt a server from deferral with alwaysLoad, so that its tools load in full at start-up while the rest require a prior search. It is not called priority and it works as such.

We proved that the per-tool mark crosses the gateway. We set up an MCP server with one marked tool, served it through LiteLLM 1.102.0 and captured the wire: the _meta arrives whole and with the correct key. The chain is clean on purpose, with a comment in the code that says so.

And it is lost on exactly three routes, which are the gateway’s three catalogue-trimming modes. Proxy mode, tool search driven by key permissions, and the REST listing. In other words: the per-tool mark and LiteLLM’s catalogue trimming are incompatible. You have to choose one.

There is a library trap that makes the field disappear silently. The SDK’s Tool model declares the alias without allowing population by field name, so building it with the field name puts the dictionary in the extras and serialises it under the wrong key. There is no error. The field simply does not arrive.

The cheap lever is the description, not the name. Swapping descriptions shifts the selection distribution substantially; changing only the name has minimal and inconsistent effects. And the gateway’s overrides really do replace what is sent to the client.

Reordering the catalogue is a weak lever. The model is already attending to the correct tool eighty per cent of the time when it fails. Interventions on prompt order repair at most 23 % of the failures.

And the asymmetry you build is attack surface. Whoever controls the order controls the selection: with injection rates of 1.2 % you can take over the head of a tool ranking between 91 % and 97 % of the time.

You are here: the question that arrives after the third server

The previous article closed with a decision ladder for the catalogue: trim, rewrite, split by route, and use progressive disclosure when the catalogue grows. That ladder answers how much catalogue to show.

The question that arrives afterwards is a different one, and it is more uncomfortable. A real agent has native tools from the client hosting it, it has the in-house gateway with the house servers, and it has two or three third-party servers that somebody connected. All of them compete. And what you want is not to show less catalogue: you want that, faced with a query three tools could serve, yours wins. The audited one, the one with access control in place, the one that leaves a trace.

The answer everybody gives is to write a rule in the agent’s instruction file politely asking it to prefer the in-house server. There is even a rule published in a template marketplace that does exactly that. It is text in the prompt and the model follows it when it feels like it.

This article is what lies underneath.

The analogy: the counter with two trays

Let us go back to the switchboard operator from the previous article, the one who also hands out keys. Now the building has three key providers: the house cabinet, the maintenance contractor’s, and the cleaning service’s. All three have a key to the store room and all three open it.

The head of security wants the house one used, because it is the one with a log. And he discovers he can do three things, not one more.

He can take the other two off the counter. It always works and it annoys whoever needed them.

He can leave his own in the front tray and the other two in a drawer, so that to take one you have to ask. It works almost always and annoys nobody.

And he can change the label on his key so that it describes better when it is the right one. It works sometimes, it is the cheapest, and it is the only one that also protects against the contractor changing his label without warning.

What he cannot do is put a number from one to ten on each key and trust the operator to respect it. That number does not exist. The rest of the article is why it does not exist and what to do instead.

What the specification says: nothing

The current revision is 2026-07-28. Read against the source schema, the Tool object has name, title, icons, description, input schema, output schema, annotations and _meta. There is no field for priority, weight, rank, cost, group or tag.

The annotations are exactly five: title, and the read-only, destructive, idempotent and open-world hints. The schema itself warns that all of them are hints and that a client should not make tool-use decisions based on them when they come from untrusted servers.

The `priority` that exists and is no use

Here is the mistake I see repeated most. MCP does define a priority field, numeric between zero and one. It is in the annotations interface that applies to content blocks, to resources and to prompts, alongside the audience and the last-modified date.

It is not in Tool. It has no relation whatsoever to which tool the model picks. Anyone who finds it by searching the word in the schema and concludes that tool priority is standardised has the wrong structure.

The only precedence the specification defines for a tool is one of presentation: to display the name, first the title, then the annotations title, then the name.

The only blessed filtering is by credential

There is a new sentence in the current revision that is genuinely useful and that goes unnoticed. The set of tools must not vary by connection nor as a side effect of other requests, and it may vary according to the authorisation presented in the request.

Translated: segmenting the catalogue by key, by team or by token is the legitimate way to do it. Segmenting it by session state stopped being legitimate when the July revision removed protocol sessions.

Discovery does not discover tools

server/discover, which the current revision adds and which servers have to implement, does not return tools. It returns supported versions, capabilities, server instructions, identity and the cache freshness fields. It accepts neither filters nor a cursor.

Listing is still asking for everything and paginating. The listing request accepts only a cursor: there is no query, no filter, no limit, no groups, no tags.

And on collisions between servers, the specification explicitly and reasonedly washes its hands: clients or proxies that aggregate tools from several servers will encounter collisions and should implement a disambiguation strategy, for example prefixing. And it adds that the server name is not guaranteed to be unique and should not be used to disambiguate. There is no precedence rule between servers. There is not going to be one soon.

What there is in proposals

Nothing accepted. The official proposals index only lists those in a final state, and none of them deals with priority, groups or search. In draft there is one asking for a free-text query in the listing and a filtering capability; another on groups and tags that ended up superseded; and one carrying the word priority but which is per-tool model routing, which is another thing.

The interest group exploring the grouping of primitives declares in its own minutes that it is not going to pick a canonical pattern soon, and the documents in its repository are empty. The year’s roadmap does not mention the problem.

The layer that decides: the client

If the specification gives you nothing, what is left is whatever each client has built on its own. And here there is a clear winner.

Claude Code

Tool search is on by default, and what it does is defer: MCP tools are not loaded in full at start-up, the model sees just enough and discovers them when it searches. It requires a model that supports tool reference blocks, that is Claude Sonnet 4.5, Haiku 4.5, Opus 4.5 and later.

On top of that, the lever:

{
 "mcpServers": {
 "gateway-casa": {
 "type": "http",
 "url": "https://litellm.interna.svc/mcp",
 "alwaysLoad": true
 }
 }
}

The documentation describes it bluntly: if a server’s tools must always be visible without a search step, you set alwaysLoad to true and all of that server’s tools are loaded into context at the start of the session, regardless of the global search setting. And it recommends using it for a small number of tools that the model needs on every turn, because each tool loaded up front consumes context.

That is visibility asymmetry, and it works as priority even if it is not called that. Ours in front, the rest behind one step.

There is an operational consequence that the documentation mentions and that matters more in a gateway than in an ordinary server: setting alwaysLoad makes start-up wait for that server to return its tools, capped by the standard five-second connection timeout. A gateway that aggregates eight upstream servers and does not cache the listing, as is the case, lists live against all eight every time. If one of them is slow, the session’s start-up feels it.

There is also the fine-grained variant. An MCP server can mark individual tools as always loaded by including "anthropic/alwaysLoad": true in the tool’s _meta object, with the same effect for that tool alone. That would let you load three tools from our gateway up front and leave the other forty from the same gateway deferred. It is the lever that matters, and it is the one that motivated the test in the next section.

The rest of this client’s arsenal is pure exclusion, and it is worth knowing one detail that decides whether it works or not. A deny rule with the bare tool name removes it from the model’s context; with parentheses, it does not. The documentation puts it in a table: plain WebFetch removes the tool and the model cannot search at all, whereas WebFetch(domain:*) keeps the tool and rejects every request. The difference is not cosmetic: in the second case the model still sees the tool, still tries it and still spends the context its definition occupies.

The --tools flag is the scalpel: it accepts the empty string to disable all the built-ins, and it does not affect MCP tools. That is, --tools "" leaves the model with only what the gateway serves. It is a command-line flag and has no equivalent as a settings-file key; the functional equivalent there is deny rules.

The other clients

Client	Disable the native ones	Filter the MCP ones	Priority
Claude Code	`--tools ""`, deny by bare name	deny by pattern, subagents	`alwaysLoad`
Codex CLI	`features.shell_tool` to false	`enabled_tools` and `disabled_tools` per server	No
Gemini CLI	`tools.core` as an allowlist	`includeTools` and `excludeTools` per server	No
Copilot in VS Code	Not documented	picker and tool sets	No
Cursor	No	per-server switches	No

Two notes. Copilot cuts off at 128 tools per request and its virtualisation threshold does not go above that; its embedding-based routing is internal and has no configuration surface. And Cursor’s rules are purely indicative: they are text prepended to the context.

The API layer

If the client is in-house, the margin is wider, though narrower than it looks.

Forcing a set is not possible. In Anthropic’s case, tool choice accepts auto, any, a specific one by name, or none, and the documentation says explicitly that pointing at a set of MCP tools or at a member is not supported. In vLLM it is the same: a specific one, all of them or none. There is no way to say “any of mine from the gateway”. The proposal that would introduce grammars restricting tool names to the request’s set has been open for months and is not merged.

Filtering the catalogue is possible, in all three of the big ones. In Anthropic, with an MCP tool-set block where the default configuration disables and specific ones are enabled. In OpenAI, with the allowed tools list inside the MCP block itself, which the documentation justifies on latency and cost grounds, avoiding the model seeing unnecessary definitions. And in Gemini, the remote MCP server configuration accepts restricting which of the server’s tools the agent may call.

It is worth not confusing two things that are named almost identically in OpenAI: the allowed tools list inside the MCP block filters the catalogue, and that is what matters here; the identically named tool-choice type is another thing, it is a selection restriction, and there is evidence that it does not combine well with hosted tools, without the current documentation recording it. If anyone depends on that, let them test it against their model: it fails as a request error, so it shows up on a cold start.

And there is a 2026 lever that almost nobody is using. Anthropic has in beta a mechanism for mid-conversation tool changes, with add and remove blocks that accept referencing an entire MCP set. The reason it exists is in its documentation: the tools array sits even earlier in the chunked prefix than the system field, so editing it invalidates the cache for the whole conversation; by declaring the full set at the start and using the blocks, the array never changes and the cached prefix stays intact.

Applied to our case: you can remove a rival server’s set hot, mid-session, without paying for the whole prefix. It is not available on every model.

The test: the `_meta` crosses the gateway

The fine-grained variant of Claude Code’s lever depends on a question no documentation answers: whether a gateway that aggregates upstream servers propagates each tool’s _meta all the way to the client, or loses it along the way.

We checked.

The setup

A minimal MCP server with the Python SDK, serving two tools over HTTP, one of them with the mark in place:

types.Tool(
 **{
 "name": "always_loaded_tool",
 "description": "tool with _meta",
 "inputSchema": {"type": "object", "properties": {}},
 "_meta": {"anthropic/alwaysLoad": True, "probe": "UPSTREAM_META_MARKER"},
 }
)

In front of it, a LiteLLM from the main branch with that server declared in config.yaml, with no database, with a master key. And an MCP client querying the listing against /mcp.

The result

This is what comes out on the wire:

{
 "_meta": {"litellm.ai/server_outcomes": {"probe": {"status": "ok", "tool_count": 2}}},
 "tools": [
 {
 "name": "probe-always_loaded_tool",
 "description": "tool with _meta",
 "inputSchema": {"type": "object", "properties": {}},
 "_meta": {"anthropic/alwaysLoad": true, "probe": "UPSTREAM_META_MARKER"}
 }
 ]
}

The mark arrives whole and with the correct key. The client parses it without trouble.

Tracing the code, the chain is clean on purpose. LiteLLM keeps the SDK objects as they are when it lists against the upstream, instead of rebuilding them. The name prefixing does a deep copy and mutates only the name, with a comment in the code declaring the intention to preserve every field including the _meta by avoiding mutation. The two permission filters are list comprehensions that reuse the same objects. And the name and description overrides mutate in place, so they do not lose anything either. Somebody thought about this.

One operational detail: the name arrives prefixed with the server alias, governed by the configurable separator and the short prefix mode.

The three routes where it is lost

And here is the finding that changes the previous article’s recommendation.

Proxy mode returns only its three fixed tools, with no _meta and with no upstream tool at all. Verified against the wire as well.

Tool search activated by key permissions does the same with four virtual tools.

And the REST listing builds its response object by hand and discards the field: the tool that did carry a mark comes out with the field null. The class inherits from the type that has the field; it simply is not filled in. The fix fits on one line.

Those three routes are, exactly, the three ways LiteLLM has of trimming the catalogue. The operational conclusion is uncomfortable and it is worth saying plainly: either you use the gateway’s catalogue trimming, or you use the client’s per-tool mark. They cannot be combined. The previous article recommended proxy mode as the fifth rung of the ladder; with this in hand, that rung and the fine-grained mark are mutually exclusive.

The library trap

This deserves its own section because it takes out anyone writing a server or a proxy, and it gives no warning.

The SDK’s tool model declares the field with an alias, and does not enable populating it by name. The consequences, both checked by running them:

Building with the field name populates nothing. The dictionary slips into the model’s extras and comes out serialised under the wrong key, without the leading underscore, which is a key no client interprets.

And dumping and revalidating without asking for aliases loses the field, for the same reason.

There is no exception, no warning in the log, nothing. The field disappears. If someone marks their tools and does not see them marked in the client, this is the first place to look, before the gateway.

What we have not tested

For honesty’s sake, and because it is the link that remains: we have shown that the mark survives the gateway. We have not shown that the client acts on it when the tool arrives with the name prefixed by the gateway. Claude Code’s documentation describes the per-tool mark in a single sentence and says nothing about aggregating servers or about prefixes.

It is a half-hour test for anyone who has the setup in front of them: two tools from the same gateway, one marked, and look at which one appears loaded at the start of the turn. If someone does it before I do, I am interested in the result.

What LiteLLM can and cannot do

With the above, the gateway’s inventory of levers comes out like this.

Lever	Serves external MCP clients	Preserves the `_meta`
`allowed_tools` and `disallowed_tools` per server	Yes	Yes
Name and description overrides	Yes	Yes
Tool sets per route	Yes	Yes
Proxy mode and virtual tools	Yes	No
Search by key permissions	Yes	No
REST listing	Yes	No
Semantic filter	No	Not applicable

Two clarifications on the previous article’s table, now that the code has been read again.

The allowed and disallowed lists are applied in the listing, not only on the call. They are the blunt, effective instrument, and they do not break the cache because they are static.

And the overrides have a limit that was not accounted for: they are applied in the MCP protocol listing, but they are skipped in proxy mode and not applied on the REST route. Anyone rewriting descriptions and also turning on proxy mode is not serving what they think.

What the gateway cannot do is inject _meta. There is no setting equivalent to the ones for name and description. The mark has to be put there by the origin server, which for in-house servers is trivial because we write them. To mark third-party tools you would have to patch, and it is small: one field in the type and three lines in the overrides function if configuring it by file is enough.

There is also an ordering lever that did not appear in the previous article. In the proxy’s search mode there is a core tools setting that puts them at the front of the ranking and that also does not count against the results cap. It is the closest thing to a declarative priority in the whole stack, and it lives inside the one mode that discards the _meta.

And a nuance about proxy mode’s results cap. The client cannot ask for more than five, that is true, but the operator can raise it by configuration. The previous article implied it was immovable.

The cheap lever: the description, not the name

If you cannot exclude, what is left is to bias. And bias comes in through the description.

The work that measures selection bias between functionally equivalent tools puts the combined bias of the evaluated models between 0.25 and 0.38, that is, you would have to redistribute between 25 % and 38 % of the probability mass for equivalent tools to be picked equally. On that basis, swapping two tools’ descriptions shifts selection substantially, whereas perturbations that touch only the name produce smaller and more inconsistent effects.

Another piece of work measures that a single pass of automatic description rewriting improves the metric over a large corpus, almost as much as iterative refinement, and with two orders of magnitude less time. And it adds a warning worth retaining: descriptions tuned against a fixed set of candidates do not generalise to the dynamically retrieved set. You have to optimise for the regime you serve in.

Applied: the gateway’s description override is at once token trimming, selection bias and mitigation of description change after approval. It is the lever with the best effort-to-effect ratio in the whole article, and it is static, so it does not touch the prompt cache.

Name prefixing, by contrast, costs tokens and probably does not change which tool the model goes to.

What does not work

Reordering the catalogue. A June paper measures that, when the model fails, it was already attending to the correct tool eighty per cent of the time, well above chance. The bottleneck is not at the input but in the late layers. Interventions on prompt order recover at most 23 % of the failures, against the 59 % to 91 % of those acting on the final readout. Reordering is cheap and that is why it gets recommended a lot; it is also weak.

Rules and instructions. They are text. They help and they do not decide.

Your own per-request top-K filters. This is covered in the previous article and it still holds: four out of five measured strategies perform worse than not filtering, and on top of that rewriting the tool block on every turn invalidates the entire cached prefix.

The ladder, updated

Exclude in the client. It is the only deterministic thing. --tools "" or deny by bare name for the competing native tools.
Mark your own as always loaded. At server level with alwaysLoad, or per tool with the mark in the _meta if you control the origin server. Counting on start-up waiting for the server.
Trim with static lists per server and per key, and split a catalogue by agent profile across routes.
Rewrite the descriptions with the gateway’s overrides.
Remove hot the rival sets, if the client is in-house and the model supports it.
Choose: the gateway’s catalogue trimming, or the per-tool mark. Not both.
Do not: reorder, rename, or trust prompt rules.

The risk that has to be declared

The asymmetry you build is also a surface. Whoever controls which tools head a ranking controls the selection, and that can be attacked: there is work measuring that by injecting adversarial tools at rates of 1.2 % you take over the head of the ranking between 91 % and 97 % of the time.

In a system under ENS (Esquema Nacional de Seguridad, Spain’s national security framework) this fits under change management, not access control: the set of tools a model sees and the order in which it sees them are security configuration, and today nobody versions them. Neither does the specification fix a hash of the description or of the schema, nor does the gateway. The specific Annex II codes are worth checking against the current text before taking them into a compliance document; the approach is developed in the MCP gateway article.

There is a precedent within the ecosystem itself that points the way. The MCP skills extension, which is in a final state, requires a manifest with a per-file hash, mandatory verification before use, and approval tied to the set of files and their digests, so that any change revokes the approval. And it obliges hosts to prevent two skills with the same name from silently replacing one another. It is exactly the pattern missing for tools. Anyone who needs it today has to implement it in their gateway.

Checklist

Decide the strategy before touching anything: catalogue trimming in the gateway, or the per-tool mark in the client. They are mutually exclusive.
If you pick the mark, put it in the origin server, because the gateway cannot inject it.
Build the tool object with the alias key, never with the field name, or the field disappears without warning.
Do not revalidate tool objects from a dump without asking for aliases.
Count on marking a server as always loaded making session start-up wait for that server, and on a gateway listing live against all of its upstreams.
Exclude in the client the native tools that compete, by bare name and not with a pattern in parentheses.
Rewrite the descriptions of your own tools, and do not waste time renaming.
Do not turn on proxy mode if you depend on the description overrides, because they are not applied there.
Version the exposed tool set and its order as security configuration, with a hash of the description and the schema.
If you use the gateway’s REST route for anything, know that it discards the _meta.

Traps and things that are not what they look like

MCP’s priority exists, and it is not about tools. It lives in the annotations for content, resources and prompts.
Tool annotations are hints and the schema itself says they are not reliable from untrusted servers.
The specification gives no precedence rule between servers, and it says besides that the server name is no use for disambiguating.
The tool set may indeed vary by authorisation, and that is the only blessed segmentation.
server/discover does not return tools.
A deny rule with parentheses does not remove the tool from the context, it only rejects the calls.
--tools is a command-line flag, with no equivalent in the settings file.
Marking a server as always loaded delays start-up by up to five seconds per server.
Tool(meta=...) does not populate the field and serialises it under the wrong key.
Proxy mode, search by permissions and the REST listing discard the _meta.
Description overrides are skipped in proxy mode and are not applied on REST.
Proxy mode’s cap of five results can be raised by the operator, even though the client cannot ask for more.
The core tools of the proxy’s ranking are the only declarative priority in the stack, and they live in the mode that loses the _meta.
You cannot force “any of mine from the gateway” in any API.
In OpenAI, the allowed list inside the MCP block and the identically named tool-choice type are different things.
Gemini does accept a per-tool allowlist on its remote MCP servers.
Reordering the catalogue repairs at most 23 % of the failures.

Closing

The starting question had a trap in it, and the trap is the word. When someone asks how you prioritise an MCP server, they are assuming a dial exists somewhere. It does not, and I have spent the whole article showing the places where it is not.

What there is instead is poorer and more manageable: you remove what competes, you leave your own in front, and you write the description better. Three things, none of them elegant, all three effective in that order.

What does seem to me worth taking away is the shape of the problem. The tool catalogue a model sees is security configuration in the full sense, because it determines what the agent can do and which system it is going to talk to. And today it is managed like a list of connections: things get added, it is not versioned, it is not signed, and nobody detects a change. The ecosystem has already solved that problem once, for skills, with hashes and tied approval. For tools, not yet.

In the meantime, the asymmetry has to be built by hand, server by server, and knowing that it rests on the _meta of an object that a library can empty without saying anything.

Sources

Model Context Protocol, herramientas en la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/server/tools.
Model Context Protocol, server/discover: https://modelcontextprotocol.io/specification/2026-07-28/server/discover.
Model Context Protocol, paginación: https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/pagination.
Model Context Protocol, caché de listados: https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/caching.
Model Context Protocol, esquema fuente de la revisión: https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/schema/2026-07-28/schema.ts.
Model Context Protocol, extensión de habilidades: https://modelcontextprotocol.io/extensions/skills/overview.
Model Context Protocol, grupo de interés sobre agrupación de primitivas: https://modelcontextprotocol.io/community/interest-groups/primitive-grouping.
Anthropic, Claude Code y MCP (alwaysLoad, búsqueda de herramientas): https://code.claude.com/docs/en/mcp.
Anthropic, Claude Code, referencia de línea de comandos (--tools): https://code.claude.com/docs/en/cli-reference.
Anthropic, Claude Code, permisos (denegación por nombre pelado): https://code.claude.com/docs/en/permissions.
Anthropic, conector MCP y conjuntos de herramientas: https://platform.claude.com/docs/en/agents-and-tools/mcp-connector.
Anthropic, cambios de herramientas a mitad de conversación: https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages.
Anthropic, búsqueda de herramientas: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, MCP y conectores (lista de herramientas permitidas): https://developers.openai.com/api/docs/guides/tools-connectors-mcp.
OpenAI, búsqueda de herramientas y caché: https://developers.openai.com/api/docs/guides/tools-tool-search.
Google, llamada a funciones en la Interactions API: https://ai.google.dev/gemini-api/docs/function-calling.
vLLM, llamada a herramientas y decodificación restringida: https://docs.vllm.ai/en/latest/features/tool_calling.html.
vLLM, propuesta de decodificación guiada por regiones y gramáticas de herramienta: https://github.com/vllm-project/vllm/issues/39848.
LiteLLM, código del gateway MCP: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, tool_search.py, litellm/experimental_mcp_client/tools.py: https://github.com/BerriAI/litellm.
SDK de Python de MCP, modelo Tool y serialización por alias: https://github.com/modelcontextprotocol/python-sdk.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307v2.
Looking Is Not Picking, atención frente a lectura final y techo de las intervenciones sobre el prompt: https://arxiv.org/html/2606.16364v1.
A Single Rewrite Suffices, reescritura de descripciones y generalización al conjunto recuperado: https://arxiv.org/html/2606.30775.
ToolFlood, saturación adversaria de la capa de recuperación: https://arxiv.org/html/2603.13950.

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

Sat, 12 Sep 2026 09:00:00 +0200

A follow-up to Keycloak in an AI platform, which flagged the gap: resource indicators are not supported and protected resource metadata belongs to the MCP server. This post tries to close it. Verified against revision 2026-07-28 of the specification, the official Python and TypeScript SDKs, Keycloak 26.7.3 and LiteLLM 1.102.0.

TL;DR

The specification splits three obligations and the authorisation server covers only one. The MCP server must publish its protected resource metadata. The client must send the resource indicator in both requests, and must do so even if the authorisation server does not support it. The MCP server must validate that the token was issued for it. Keycloak does not understand the indicator and considers the metadata someone else’s, so all three end up in the resource layer.

The Python SDK mounts the metadata on its own, but degraded. As soon as the resource server URL is configured, the route appears. What that automatic route publishes carries a single authorisation server, reuses as its scope catalogue the ones the middleware requires, and leaves the name and the documentation null. To publish a complete document you have to mount the route by hand.

Audience validation comes switched off in Python and does not exist in TypeScript. In Python, if the resource URL is configured and the corresponding flag is not enabled, the SDK emits a deprecation warning and behaves as if it were disabled; the docstring itself promises that version 3 will set it to true. In TypeScript the bearer verification does three things, and none of them is looking at the audience. Neither of the two ships a signature verifier with public keys.

Token passthrough is forbidden in writing, and the substitute has a name. The specification says the MCP server must not accept or relay tokens that were not issued for it, and that if it calls upstream APIs the token must be a different one. The mechanism is token exchange, which Keycloak supports in its standard version with a limitation you need to know: the audience is a client identifier, not a resource URL.

There is an official extension that changes the whole picture, and it is stable. Enterprise-managed authorisation replaces the redirect to the MCP authorisation server with an exchange at the corporate identity provider, which evaluates the policy and issues a grant based on an identity assertion. It has been stable since June 2026 and there are servers in production.

Keycloak implements it halfway and in experimental. It only knows how to act as receiver, not as issuer, behind a feature flag, and against draft 01 when the working group is on 04. The official documentation literally says not to use it in production.

LiteLLM already implements it on the client side and does not document it. In 1.102.0 there is an authentication mode towards MCP servers that runs the two stages of that grant. There is not a single documentation file that mentions it.

You are here: the resource side

The previous post walked through the identity provider. This one walks through what sits on the other side of the token, which is where the MCP specification puts almost all its normative obligations.

The natural reading order is When MCP grows up for the basic setup, Keycloak in an AI platform for the identity piece, and this one for the missing layer. The MCP gateway seen from the inside is in its own post.

The analogy: the visa and the border checkpoint

A consulate issues visas. A border post checks them. They are two different offices, and the classic mistake consists of assuming that because the visa is genuine it is good for entering through any door.

The visa has written on it which country it is good for. That is the token’s audience, and here the problem appears: the consulate we work with does not know how to write the destination the traveller asks for, because it does not understand that field on the form. It knows how to write a destination if asked for it under a different name, by way of a pre-agreed stamp, which is the workaround of scopes and the audience mapper.

And then there is the other half, the one almost nobody builds. The border post has to exist, it has to announce where it is and from which consulates it accepts visas, and it has to read the destination written on the visa before letting anyone through. If the border post limits itself to checking that the stamp is genuine and does not look at the destination, any valid visa from the same consulate is good for entering. That is exactly what the two official SDKs do by default.

Part 1. The split of obligations

From revision 2026-07-28, with the normative words as they are:

Obligation	Whose	Status with Keycloak
Publish protected resource metadata (RFC 9728)	MCP server, MUST	Out of its scope, by design
Send the resource indicator (RFC 8707) in authorisation and in token	Client, MUST, even if the authorisation server does not support it	It does not understand it
Validate that the token was issued for oneself	MCP server, MUST	The resource’s responsibility
Not accept or relay other tokens	MCP server, MUST NOT	The resource’s responsibility
Validate the issuer of the authorisation response (RFC 9207)	Client, MUST; authorisation server SHOULD emit it	Supported
Consent for each dynamically registered client	MCP proxy with a static identifier, MUST	Configurable

The most confusing part of that split is the second line. The specification obliges the client to send the parameter regardless of whether the authorisation server supports it. With Keycloak, that parameter is lost: the official documentation says it cannot recognise it, and the standard behaviour towards an unknown parameter is to ignore it. The client complies, the token comes out, and what does not come out is the correct audience. The failure appears later and somewhere else, which is the worst kind of failure.

It is worth knowing where that work stands. The matter is issue 14355 of the project, open, with a milestone in 26.8.0. There was a complete implementation in pull request 35711, with a mapper of its own and an extension point for resolving resources, which went back to draft in October 2025 because of a total lack of tests and because it was decided to redo it in phases, starting with a single resource per client. In March 2026 a new issue was opened for experimental support, also with a milestone in 26.8.0 and with no associated pull request. In 26.7.x there is no feature flag that enables it.

What is new in the revision and affects the setup

Four things changed with respect to the previous revision and are worth collecting:

RFC 9207. A new section on validating the authorisation response. The client must record the issuer from the validated metadata document, and the comparison is literal: you are not allowed to normalise case, elide the default port, add or remove the trailing slash or re-encode characters before comparing.
Offline tokens leave the catalogue. A new section on refresh tokens: MCP servers should not include the corresponding scope either in the challenge header or in the supported scopes of their metadata.
Dynamic registration becomes deprecated. It goes from optional to optional and deprecated, retained for compatibility with authorisation servers that do not support client identifier metadata documents. And a new requirement is added to registration: the application type is mandatory, and a client must not reuse credentials from another authorisation server, it has to register again.
Step-up elevation is rewritten. The menu of three server strategies disappears and is replaced by two rules: the challenge’s scope attribute describes what is needed for the requested resource, with no obligation to include what has already been granted, and scope accumulation becomes the client’s responsibility. With a new requirement for the server: it must take scope hierarchies into account, where a broad one implies the narrow ones.

Part 2. What the SDKs give, and what they do not

Here is the uncomfortable part, and it is the main reason for writing this post. Verified by reading the code of the Python SDK and of the TypeScript 2.0.0-alpha one.

Protected resource metadata

In Python the complete model exists and it mounts on its own. The metadata class carries the RFC fields: resource, authorisation servers with a minimum of one, key set URL, supported scopes, bearer methods with a default value of header, name, documentation, policy, terms, and the client certificate and DPoP fields. The handler serves the document with a one-hour cache directive, and the URL is built by inserting the well-known path in front of the resource path, as the RFC requires.

As soon as the resource server URL is configured, the route appears with no further work. But what that automatic route publishes is degraded in three points: it passes a single authorisation server, it takes as its scope catalogue the ones the middleware requires (which are not the same as the ones the resource supports), and it passes neither the name nor the documentation, which come out null. To publish a complete document you have to call the route creation function by hand.

In TypeScript it exists and does not mount on its own. The function that builds the document emits only five fields: resource, authorisation servers, supported scopes, name and documentation. It does not emit the supported bearer methods, nor the key set URL, nor anything about DPoP. And serving it is explicit: either you call the response function from your own handler, or you mount the metadata router. There is no point in the SDK that does it on its own.

Audience validation

This is the one to fix on day one.

In Python it comes disabled. The token verifier is a single-method protocol, that is to say a hole the implementer fills. The access token’s resource field is the indicator you set yourself. And the comparison only happens if you ask for it: there is a function that normalises as a URL and ignores the trailing slash, but the resource server URL reaches it empty unless the validation flag is enabled.

The detail to read twice is in the configuration: if the resource URL is set and the flag is not, the SDK raises a deprecation warning and behaves as if the flag were false. The docstring says version 3 will set the default to true. Until then, a Python server configured with authorisation accepts tokens issued for another resource, unless the implementer enables the flag or validates the audience inside their own verifier.

In TypeScript it is not validated at all. Bearer token verification does exactly three things: strip the prefix, check the required scopes, and require that the expiry is present and not past. A search for audience or key set across the server and middleware packages returns no validation. It is all left in the hands of whichever verifier you plug in.

Neither of the two ships a signature verifier with public keys. In Python the only example is introspection against the authorisation server, and its audience check sits behind a flag that also comes out false. In TypeScript the interface is empty. That is: the component that validates the token, which is the one holding up the most important normative requirement of the specification, is your own code in both cases.

The authentication challenge

Python builds the challenge with the error and its description, and adds the metadata URL only if it is configured. It returns 401 with invalid token and 403 with insufficient scope, both from the same function, so both carry the metadata URL. What it never emits is the scope parameter, which revision 2026-07-28 asks to be included, and which is exactly what the client needs in order to know what to ask for.

TypeScript does emit the scope when there are required scopes, in addition to the metadata URL, and it correctly maps invalid token to 401 and insufficient scope to 403.

That is, each SDK gets a different half right. The Python one validates better and warns worse; the TypeScript one warns better and does not validate.

Scopes, which are not per tool

In both SDKs scopes are a static list at the transport mounting level, not per tool. There is no per-tool scope declaration and no step-up elevation help on the server side. Emitting a 403 with the specific scope that call requires, which is what the specification describes, is your own code.

That is the origin of almost all the fine-grained authorisation you have to build, and it is why there is a Part 6.

Which revision each one announces

Python is already on 2026-07-28. The revision appears in the known versions and in the list of modern versions, described as the ones that use the stateless envelope per request. The server discovery method is registered with a default handler, and the method and name headers are validated against the body.

TypeScript has it, but in a separate list. The latest protocol version constant is still 2025-11-25, because that list is only the one for the initial handshake. The modern era lives in a separate module, with a comment explaining the reason: keeping them deliberately separate so that adding a revision there never leaks a modern version string into a 2025-era handshake.

And it is worth recalling the mismatch that already came up in the MCP gateway post: LiteLLM 1.102.0 still announces 2025-06-18.

Part 3. Building the protected resource

With the above, the list of what you have to write is short and concrete.

1. A token verifier that validates the signature and the audience. Against the realm’s key set, checking issuer, expiry and that the audience contains the MCP server’s canonical URL and only things that concern it. Here the side effect of the Keycloak workaround comes in: if a client asks for two scopes from two different resources, out comes a token with two audiences, because each mapper contributes its own to the array. There is no client policy condition or executor in 26.7 that limits the number of audiences per token. The practical defence is the resource’s: reject tokens whose audience includes other resources, instead of merely checking that its own is present.

2. The complete metadata document, with all the authorisation servers you accept, the real catalogue of scopes the resource understands, and the name and documentation filled in. In Python, mounting the route by hand instead of leaving the automatic one. In TypeScript, mounting it, full stop.

3. The authentication challenge with the scope. In Python you have to add it, because the SDK does not emit it. And you have to take scope hierarchies into account, which is a new requirement of the revision.

4. The decision between local validation and introspection. Validating the signature locally costs no network, but revocation does not take effect until the token expires: there are no revocation lists for signed tokens. Introspection costs a round trip per request and only confidential clients can invoke it. The reasonable combination is a short token life and introspection on the operations that change state.

5. Binding state to the user. The new revision is explicit about this, because with protocol sessions gone, state is carried in handlers that travel as an ordinary tool argument. Servers must verify every incoming request and must not treat possession of a handler as authentication; and they should bind the handler to the user on the server side, for example storing the state under a key that combines the user identifier derived from the verified token with the handler, and reject the handler if someone else presents it. It is a change in the way of working compared with the sessions of before.

Part 4. The hop from the gateway to the MCP server

The real shape in an inference platform is not client against MCP server: it is client against gateway, and gateway against MCP server. That second hop is where it is decided whether the architecture is correct.

The specification leaves no room. The MCP server must not accept or relay tokens that were not issued for it. And if it makes requests to upstream APIs, it may act as an OAuth client of theirs, but the token it uses there is a different token, issued by the upstream authorisation server, and it must not forward the one it received.

The standard mechanism for that is token exchange, and here you need to know three Keycloak details.

First, the permission model changed. The old version required fine-grained administration permissions and an explicit exchange authorisation on the target client. The standard version, supported since 26.2, does not require them: it is enough for the requesting client to be confidential and to have the corresponding switch enabled. In exchange there is a surprising condition: the subject token has to carry the requesting client in its audience, unless it exchanges its own token. That is, the gateway needs to appear in the audience of the user token, which is achieved with an audience mapper in a default scope of the gateway. Only then can it narrow the audience to that of the specific MCP server.

Second, the audience is a client identifier, not a resource URL. The parameter filters audiences, that is to say it narrows them, which is what you want. But it takes the identifier of a client registered in the realm. The practical consequence is that the MCP server has to be registered as a client, and its identifier should be its canonical URL, so that the resulting audience matches what the resource validates. It is a naming trick, and it has to be documented for whoever comes after. The documentation itself acknowledges it: token exchange does not yet support the resource parameter.

Third, the subject token type is limited. The standard version only accepts access tokens as subject.

There is also a new experimental feature in 26.7 that adds a parameterised scope type for validating whether the requesting user is authorised to act on behalf of another. Interesting for delegation, but experimental.

What the gateway does today

In LiteLLM 1.102.0 the authentication modes towards MCP servers number twelve, with a default value of none. The ones that matter for this discussion are four.

oauth2_token_exchange implements the standard exchange. It sends the correct grant type, the subject token and its type, and the audience, never the resource. The omission is deliberate according to the code itself: fabricating a target risks an invalid target error. It caches the resulting token under a key that combines the subject token with the whole configuration, and with a lifetime equal to the token’s minus one minute. And it blocks, never degrades: with no incoming token it returns 401, which the edge turns into a challenge with the resource metadata URL; a rejection from the identity provider returns 401; a gateway configuration error returns 500; a transport failure, 503.

true_passthrough and oauth_delegate forward the client’s authorisation header as is. That is token passthrough, by the name the specification gives it, and its legitimate use is narrow: when the token the client presents was already issued for the target MCP server and the gateway is a mere transport. Outside that case, it is non-compliant.

oauth2_id_jag is the surprise, and it deserves a part of its own.

Part 5. The enterprise extension, which changes the picture

In June 2026 the MCP project published an authorisation extension that solves a different problem from the one the standard flow solves, and which in an organisation with an identity provider of its own is the real problem.

The normal flow is per user: each employee authorises each client against each MCP server. It works for consumer applications and it does not work in a company, because onboarding a person requires authorising dozens of services one by one and offboarding requires revoking them one by one.

The io.modelcontextprotocol/enterprise-managed-authorization extension inverts that. It is in the stable specifications directory of the extensions repository, it comes from SEP-990, and the flow is this:

The MCP client authenticates the user against the corporate identity provider through the normal flow, and stores the identity assertion, which may be an OpenID identity token or a SAML assertion.
When the server indicates that enterprise-managed authorisation is needed, the client exchanges that assertion at the corporate provider for an authorisation grant based on an identity assertion. The provider evaluates the organisation’s policy there: group membership, roles, conditional access.
The client presents that grant to the MCP authorisation server and obtains an access token.
The sentence that defines the extension, verbatim: the user is not redirected to the authorisation endpoint of the MCP authorisation server.

The MCP authorisation server validates the signature against the corporate provider’s key set, plus audience, issuer and expiry, and uses the subject as the user’s stable identifier, with the email as a fallback for linking earlier accounts.

What that buys is exactly what a compliance file asks for: policy in a single place, an auditable decision at the identity provider, and centralised revocation that takes effect across all clients at once. The employee who loses access stops receiving grants, without touching any server.

The underlying standard is a draft from the IETF’s OAuth working group, under the name of JWT identity assertion authorisation grant, revision 04 of May 2026, signed by people from Okta, Ping Identity and an independent author. It profiles identity chaining across trust domains by combining token exchange with the JWT profile.

And here comes the problem

Keycloak implements it only halfway. It has its own documentation page, behind a feature flag. And three limitations you need in front of you before designing anything:

It only acts as receiver. It accepts assertions issued by an external provider and issues local tokens. Native support for acting as issuer, the documentation says, is not yet fully implemented. In the extension’s flow, the issuer is the corporate provider: if that provider is Keycloak, the missing piece is exactly the one you need.
It is based on draft 01 while the working group is on 04.
The official documentation says not to use it in production.

The asymmetry is striking and worth pointing out: Keycloak does not support a standard published eight years ago such as resource indicators, and does partially support a draft from a few months back.

What is already in the gateway, undocumented

In LiteLLM 1.102.0, the oauth2_id_jag mode runs the two stages of the flow. In the first it asks the corporate provider for a token exchange with the requested token type specific to this grant, sending the user’s identity token as subject and, optionally, audience, resource and scope. In the second it presents the resulting assertion to the resource’s authorisation server with the JWT bearer grant type. In both the gateway authenticates as a client with a JWT signed by private key.

Two operational warnings that come out of the code and are nowhere else:

The subject comes from the caller’s identity token, or from the one captured at login. And only the generic OIDC provider captures assertions: with Google, Microsoft or SAML there is none, and all users fail. The code raises a warning when loading the configuration.
A badly configured server refuses with a 500 instead of falling back to the static credential. It is the right decision, and it is worth knowing before it happens.

And the main warning: there is no documentation. Not a single file in the repository that mentions this mode. Anyone using it is reading code, like this post.

Part 6. Per-tool authorisation

None of the previous pieces answers the question the business ends up asking: whether this person may call this tool with these arguments. The identity provider gives roles and scopes, and with that you do not model a graph of relations, as was already argued in the previous post.

What is published today, with sources:

OpenFGA is the only one with a dedicated product guide, updated on 9 September 2026. The pattern is a tool type with an invocation relation, checked on each call, and a listing query so the client only sees the tools it may invoke. It is a pattern and a modelling language, not a reference implementation.
Pomerium ships a per-tool policy criterion, with exact, prefix, suffix and list matching.
Kong offers it as a product, with per-tool control lists.
Traefik Hub goes as far as parameter-level restrictions, which is the finest published grain.
Envoy has an MCP filter that extracts protocol attributes for fine-grained access control, declared to be in active development and with no OAuth of its own: the hook is external authorisation towards a policy engine.

The lowest-friction option in a platform that already has a service mesh is external authorisation against a policy engine. The most expressive one, the relations model.

Part 7. What still has no standard

Two gaps the specification does not cover and that have to be closed from outside.

Tool definition change. Neither revision 2026-07-28 nor the SDKs pin a hash of the description or of the input schema. The best practices page covers confused deputy, token passthrough, server-side request forgery in discovery, state handler hijacking, server confusion and validation of the authorisation URL scheme, but there is no integrity or pinning requirement. The tool that does it is the Invariant Labs scanner, under the Apache 2.0 licence, which pins hashes to detect definition change after approval and also ships a proxy mode with guardrails. It already came up in the MCP gateway post and it remains the only answer.

The public guidance that does get into this is the NSA’s on MCP security design considerations, from May 2026, done with Carnegie Mellon’s software engineering institute. Two of its observations are directly actionable: that authorisation in MCP is optional and that the specification imposes no requirement on token lifecycle management, so expiry and rotation are left in the organisation’s hands; and the warning against dynamic tool discovery with no origin verification and no authorisation checks, which is the most direct acknowledgement of the risk in a state guidance document.

Part 8. Who covers what

With all of the above, the decision table for the layer you put in front of the MCP server:

Piece	Publishes resource metadata	Validates audience	Token exchange	Per-tool authorisation
Python SDK, automatic route	Yes, degraded	Not by default	No	No
TypeScript SDK	Does not mount on its own	No	No	No
Traefik Hub (commercial)	Yes, automatic	Not documented	Not documented	Yes, down to parameter
Kong, OAuth plugin for MCP	Not documented	Yes	Not documented	Yes, per-tool lists
Pomerium	Partial	Not documented	Not documented	Yes, per-tool criterion
agentgateway (Solo.io)	Yes	Yes, by JWT policy	Yes, including the assertion grant	Yes, with expressions
mcp-context-forge (IBM)	Not in release notes	Not documented	Yes, since 1.0.6	Yes, per-role control
Envoy, MCP filter	No	No	No	Via external authorisation
LiteLLM 1.102.0	Yes, and authorisation server metadata too	Not on the incoming JWT	Yes, and the assertion grant	Yes, per-key and per-team permissions

Two warnings about that table. The Kong plugin requires a minimum version of 3.12, is in technical preview and needs a licence for its AI edition; it also introduces a breaking change in 3.13, which starts treating all traffic as MCP in order to close a possible authentication bypass. And on oauth2-proxy I found no primary documentation either for or against protected resource mode: I am not claiming it does not have it, I am claiming it is not documented where I could look.

Reference configuration

Keycloak: the per-resource scope, optional, with its audience. That it is optional and not default is the key piece, because it is what makes the client have to ask for it explicitly and makes it work as a substitute for the resource indicator:

kcadm.sh create client-scopes -r plataforma \
 -s name=mcp:inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

And the MCP server registered as a client with its canonical URL as identifier, so that token exchange produces the right audience:

kcadm.sh create clients -r plataforma \
 -s clientId=https://mcp.ejemplo.es/inventario \
 -s enabled=true -s publicClient=false -s consentRequired=true

The gateway, with token exchange instead of token passthrough:

mcp_servers:
 inventario:
 url: "https://mcp.ejemplo.es/inventario"
 transport: "http"
 auth_type: "oauth2_token_exchange"
 client_id: "litellm-gateway"
 client_secret: os.environ/GW_SECRET
 token_exchange_endpoint: "https://sso.ejemplo.es/realms/plataforma/protocol/openid-connect/token"
 audience: "https://mcp.ejemplo.es/inventario"
 subject_token_type: "urn:ietf:params:oauth:token-type:access_token"
 token_exchange_profile: "rfc8693"

The resource verifier, which is what nobody hands you ready-made. The essential bit is the last check, the one that rejects other resources’ audiences instead of settling for finding its own:

RECURSO = "https://mcp.ejemplo.es/inventario"

async def verificar(token: str) -> AccessToken | None:
 claims = jwt.decode(
 token, await jwks(),
 algorithms=["RS256"],
 audience=RECURSO, # validates that I am there
 issuer=ISSUER,
 )
 aud = claims["aud"]
 aud = [aud] if isinstance(aud, str) else aud
 # and that nobody else is: the scopes workaround allows
 # tokens with two audiences, and that reopens the confused deputy
 if set(aud) - {RECURSO, CLIENTE_GATEWAY}:
 return None
 return AccessToken(
 token=token,
 client_id=claims["azp"],
 scopes=claims.get("scope", "").split(),
 expires_at=claims["exp"],
 resource=RECURSO,
 )

And when mounting the server, the flag that does not come set:

auth = AuthSettings(
 issuer_url=ISSUER,
 resource_server_url=RECURSO,
 validate_token_resource=True, # without this there is only a warning
 required_scopes=["mcp:inventario"],
)

Checklist

Enable audience validation in the Python SDK, or write it into the verifier if it is TypeScript. Without that, the central normative requirement is not met.
Reject tokens with other resources’ audiences, not just check that your own is present.
Mount the metadata document by hand, with all the authorisation servers and the real catalogue of scopes.
Add the scope parameter to the challenge, in Python, and account for scope hierarchies.
Register each MCP server as a Keycloak client with its canonical URL as identifier, so that token exchange produces the right audience.
Replace any token passthrough mode with token exchange, except the narrow case where the token was already issued for the target.
Check that the gateway appears in the audience of the user token, or the exchange will fail.
Bind state handlers to the user on the server side, and never treat them as authentication.
Decide between local validation and introspection knowing that the first delays revocation until expiry, and shorten the token life accordingly.
Pin a hash of the description and the input schema from outside, because the specification does not cover it.
Put a policy engine in for per-tool authorisation, and do not try to model it with roles.
If you are considering the enterprise extension, check first who issues the grant: Keycloak today only knows how to receive it, in experimental and against an earlier draft.

Traps and things that are not what they look like

The Python SDK mounts the metadata document on its own, and that is why it looks solved. What it publishes carries a single authorisation server and no name and no documentation.

The audience validation deprecation warning does not enable anything. It warns and carries on behaving as disabled.

The TypeScript SDK does not validate the audience at all. It is not configuration, it is that there is no code for it.

Python’s challenge does not carry the scope, so a client that receives a 403 does not know what to ask for.

The scopes workaround allows two audiences in the same token. Each mapper contributes its own, and there is no client policy that limits it. The defence is in the resource.

The token exchange audience is a client identifier, not a URL. If the MCP server is not registered with its URL as identifier, the resulting audience will not match what the resource validates.

Token exchange requires the gateway to be in the audience of the user token. It is the condition that wastes the most time the first time round.

Dynamic client registration is deprecated as of this revision, and the substitute, client identifier metadata documents, is experimental in Keycloak and has an open bug with documents carrying unknown fields, which blocks login with real clients.

LiteLLM’s identity assertion grant depends on the generic OIDC provider. With Google, Microsoft or SAML no assertion is captured and all users fail.

Step-up elevation is no longer the server’s responsibility. Scope accumulation moved to the client in this revision.

The authentication elevation signal goes in a 401, not a 403. Insufficient scope is 403; asking for a higher authentication level is a different thing and a different code.

Closing

The summary of this post fits in one sentence: the identity provider is the cheap half of the problem, and the expensive half is the protected resource, which no SDK hands you ready-made and which in the two official ones comes with audience validation disabled or absent.

From that comes an order of work. First the token verifier, with the audience validated in both directions, because without it everything else is decoration. Second, the complete metadata document, mounted by hand. Third, replacing any token passthrough with exchange, with the MCP server registered with its URL as identifier. Fourth, a policy engine for what roles do not model.

And a scheduling decision worth taking with the information in front of you. The enterprise-managed authorisation extension is the right route for an organisation with a provider of its own, it is stable and it has servers in production behind it. What is not ready is Keycloak as issuer. Anyone who wants that flow today has two honest options: delegate the mechanics to a gateway that implements it, or wait. Putting it into production with a feature the official documentation marks as experimental and expressly advises against is not one of them.

Sources

Especificación MCP, revisión 2026-07-28: autorización, registro de clientes y buenas prácticas de seguridad.
Extensión de autorización gestionada por la empresa y su especificación estable, del SEP-990. Anuncio de 18 de junio de 2026.
draft-ietf-oauth-identity-assertion-authz-grant, revisión 04 de 21 de mayo de 2026, grupo de trabajo OAuth del IETF.
Código de los SDK oficiales de MCP: python-sdk y typescript-sdk 2.0.0-alfa, clonados el 12 de septiembre de 2026.
Integración de Keycloak con Model Context Protocol, intercambio de tokens y concesión de autorización por aserción de identidad, consultados el 12 de septiembre de 2026.
Incidencias de Keycloak 14355, 47117 y 51039, y petición de cambios 35711.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
NSA y SEI, Model Context Protocol: Security Design Considerations, mayo de 2026.
OpenFGA, autorización de servidores MCP, actualizado el 9 de septiembre de 2026.
mcp-scan de Invariant Labs, Apache 2.0.
Documentación de agentgateway, Traefik Hub, Kong y Pomerium, consultadas el 12 de septiembre de 2026.

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

Sat, 12 Sep 2026 07:30:00 +0200

This post opens the identity block of the operations track. The previous one covered the gateway side: the JWT that does not validate audience, the user who never reaches the trace. This one covers the piece on the other side. Verified against Keycloak 26.7.3, revision 2026-07-28 of the MCP specification and the text of Royal Decree 311/2022 in the BOE.

TL;DR

Keycloak 26.7.3 was released on 31 August 2026 and is still incubating at the CNCF, which it entered in April 2023. The cadence is four minor releases a year and support in practice covers a single branch: the twenty security bugs fixed in 26.7.3 were not backported to the 26.4, 26.5 or 26.6 branches at the time of release.

The Quarkus-based distribution separates build and runtime configuration. What is set in the build phase is frozen into the image, and changing it at startup forces a rebuild. In containers, the pattern is to build into the image and start with the optimised flag.

Storage is relational and only relational. The alternative engine that was in development was withdrawn in version 25.0. Since 26.0 sessions are persisted to the database by default, which changes the load profile: the dominant cost in the official benchmarks is database CPU.

Organisations solves multi-tenancy inside a realm, fully supported since 26.0. It groups members, associates email domains and links identity providers per organisation. It is the alternative to replicating realms per customer.

The measured bottleneck is password hashing. Since 25.0 the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. The official sizing guide gives one vCPU for every 15 password logins per second, against 120 per second for client credentials and for refreshes.

Keycloak does not implement two standards the MCP specification marks as mandatory. Protected resource metadata belongs to the MCP server, not the authorisation server, so there is nothing to hold against it there. Resource indicators do belong to it and are not supported. Keycloak’s own documentation declares partial conformance with the last three MCP revisions for that reason. The practical substitute is scopes plus an audience mapper.

There is a recent critical bug worth looking at today. CVE-2026-18963, scored 9.1, allowed resetting any user’s password without authenticating, administrators included. Fixed in 26.7.2, released on 19 August 2026.

You are here: the cross-cutting layer

Identity is not a stage of the pipeline, it is a layer that crosses all of them. It shows up in stack hardening as one of the seven layers of defence, in the model’s chain of trust as workload identity, in the gateway as token validation, and in the post about authenticating MCP servers as authorisation server.

What was missing was the post about the piece itself, and above all the decision of where to put it and what to expect from it.

The analogy: the civil registry and building permits

A small town hall has a population register, which says who lives there, and it has permits, which say who can do what. They are two different things and it is worth not confusing them, because the register is good for the whole town while a permit is good for one specific job on one specific street.

Keycloak is the register. It knows who each person is, identifies them reliably, knows which groups they belong to and issues a credential document that expires. What it does not do is decide whether this person can modify this particular file: that is the permit, and it lives in each service.

Confusing the two produces the most expensive design mistake in platforms of this kind, which consists of putting into the token everything that will need to be decided later. The register document grows, gets handed out everywhere, and when a permission changes you have to wait for everyone’s document to expire.

There is a third element the analogy captures well. A credential document issued for presenting at the tax office should not be good for entering the municipal depot. That is the token’s audience, and it is the central subject of the second half of this post.

Part 1. What Keycloak is today

Project status

Item	Value
Latest stable version	26.7.3, released on 31 August 2026
Previous ones	26.7.2 (19 August), 26.7.0 (9 July), 26.6.0 (8 April)
Cadence	four minor releases a year
CNCF	incubating since 10 April 2023, not graduated
Support	the current branch; earlier ones do not receive fixes reliably

That last point deserves underlining because it conditions the upgrade policy. The analysis of 26.7.3 shows that its twenty security fixes were not backported to earlier branches at release time. Operating Keycloak means keeping up to date with the current branch, not staying on a “stable” version for a year.

Since 26.0 there is a backward compatibility guarantee across minor versions for features marked as fully supported, and breaking changes are opt-in.

The internal architecture in five pieces

Two startup modes. The development one enables HTTP, does not resolve the hostname strictly and disables the theme cache. The production one requires a hostname and TLS, or it will not start. Confusing them in production is the first antipattern on the final list.

The build phase. The build command materialises the optimisations into the image. Build options, such as the database engine, the enabled features or metrics, are frozen. Runtime ones, such as the database password or the hostname, are applied at startup. Changing a build option at startup forces a rebuild with the corresponding latency cost. The correct pattern in containers is to build into the image and start with the optimised flag.

Extensions via SPI. Authenticators, protocol mappers, user storage, event listeners, hashing algorithms and secret vaults. They are packaged as JARs and resolved in the build phase.

Infinispan cache. Embedded by default, with database-based discovery as the recommended mechanism. The discovery alternatives via Kubernetes DNS, TCP and UDP are deprecated. There are local caches, with default capacities of 10,000 entries in several of them, and replicated caches for sessions, client sessions, offline sessions, authentication sessions, login failures and action tokens. Remote mode, with an external Infinispan, is needed for first-generation multi-site deployment.

Relational storage. JPA over a database. The alternative engine was removed in version 25.0. The databases supported today include PostgreSQL from 14 to 18, MariaDB and MySQL in their extended support versions, SQL Server 2019 and 2022, Oracle 19c and 23.x, and the managed variants Aurora PostgreSQL and Azure SQL.

The minimum conceptual model

Before touching anything you need to be clear on eight concepts, and the order in which they relate:

Realm: the isolation boundary. Its own users, its own signing keys, its own policies. Two realms share nothing.
Client: an application that asks for tokens. Public if it cannot keep a secret, such as a browser application; confidential if it can, such as a service; and bearer-only if it merely validates tokens without requesting them.
Client scope: a reusable bundle of scopes and mappers, assigned to clients as default or as optional. It is the unit with which you bound what a token carries.
Protocol mapper: what projects an attribute, a role or an audience into the token. The audience mapper is the one that solves the central problem of this post.
Role, realm or client, and composite role, which aggregates others.
Group, hierarchical, with inherited roles and attributes.
Identity provider: federation towards another external OIDC or SAML. User federation: LDAP, Active Directory and Kerberos, with synchronisation and attribute mapping.
Authentication flow: a sequence of executions with required, alternative, conditional or disabled requirements, and subflows. It is where you build a second factor or a check of your own.

And one feature that changes architectures: organisations, fully supported since 26.0. It solves multi-tenancy inside a single realm, grouping members, associating email domains and linking identity providers per organisation, so that provider discovery is done by the user’s email domain. The classic alternative, one realm per customer, multiplies administration and shares no configuration.

Part 2. Who can authenticate against it

This is the table you need before drawing anything. The right-hand column is usually the one that decides.

Piece	Protocol	Cost
Kubernetes API	native OIDC, or the structured authentication configuration file	Free
LiteLLM	OIDC and JWT, plus UI SSO	JWT and SSO above five users: Enterprise
vLLM	static key only	No OIDC
Langfuse	generic OIDC and Keycloak provider	SSO free; per-project access control: Enterprise
Grafana	generic OAuth yes in the open edition	Team synchronisation: Enterprise
Backstage	generic OIDC and community-maintained Keycloak provider	Free
Open WebUI	OIDC, with role management by claim	Free
LibreChat	generic OIDC, with Keycloak documented	Free
Argo CD	native OIDC or Dex, with groups into its RBAC file	Free
MinIO	OIDC with role assumption via web identity	See the note below

Two clarifications that change decisions.

Kubernetes. The modern mechanism is structured authentication configuration, which allows several JWT issuers plus validations and mappings with CEL expressions. The corresponding proposal, KEP-3331, went alpha in 1.29, beta in 1.30 and is stable since 1.34. It is the route to use today, and it replaces the loose API server flags.

vLLM has no identity. All there is is a static key via environment variable. This is not a defect to fix, it is an architectural fact: the engine sits behind the gateway and its access control is network-based, not identity-based. Any design that exposes vLLM directly to users is giving up per-person traceability.

About MinIO a note of honesty is in order: the public documentation now redirects to the commercial edition, so the detail of how it maps policies from a claim is verified there and not in the open edition. Anyone depending on that mechanism should confirm it against their own version before designing on top of it.

Part 3. Humans and machines

They are two different problems and it is worth solving them separately.

People

Authorization code with PKCE, with the S256 method. The MCP specification requires it when the client is technically capable, and it goes further: it mandates rejecting the flow if the authorisation server metadata does not publish the supported challenge methods.

Workloads with no person behind them

Here the obvious option is client credentials with a Keycloak service account, authenticated with a secret or, better, with a JWT signed by private key.

But there is a less known and better option for workloads inside the cluster: federated client authentication, introduced in January 2026 under the banner of doing away with secrets. It lets a client authenticate with a token issued by an external provider, and one of those providers is Kubernetes. The announcement placed full support for the Kubernetes service account variant and for OIDC in version 26.6; the SPIFFE variant, by contrast, expressly says it will stay in preview until the SPIFFE client authentication standard is finalised at the IETF.

That means a pod can authenticate against Keycloak with the token the kubelet already projects into it, with no additional secret to rotate. It is worth confirming the exact status and the flag name against the release notes of whichever version you have deployed, because the feature is recent and has changed status between 26.5 and 26.6. It is preferable to token exchange, whose standard version is supported but is internal-to-internal only, while the old one, which covered external-to-internal and impersonation, is deprecated and in preview.

Keycloak against SPIFFE

The question always comes up and the answer is that they solve different things with a small overlap.

SPIRE attests the node and the process, and issues short-lived identity documents with no prior secret. Keycloak does not attest workloads: it trusts a credential someone had to place there, except in the federated authentication case just mentioned, where the one attesting is Kubernetes. In exchange, Keycloak federates people, issues tokens with business data and governs consent, things SPIFFE does not do.

The practical rule: SPIFFE for service-to-service identity inside the perimeter, Keycloak for anything that has a person behind it or crosses the perimeter. The SPIFFE detail is in the close of the chain of trust series.

Part 4. MCP, and the two missing standards

This part has the most consequences and is the worst documented outside the primary sources.

What the specification requires

From revision 2026-07-28, with the normative words as they appear:

The MCP server must implement RFC 9728 protected resource metadata, and the client must use it to discover the authorisation server.
The client must implement RFC 8707 resource indicators, including the resource parameter in both the authorisation request and the token request, and must send it regardless of whether the authorisation server supports it.
The MCP server must validate that the access token was issued specifically for it as the audience, and must not accept or relay any other token.
Authorisation servers and clients should support client identifier metadata documents. RFC 7591 dynamic registration becomes optional and is declared deprecated, retained for compatibility.

The revision also adds the issuer in the authorisation response, from RFC 9207, and a stateless model with state handlers where it is expressly stated that holding a state handler is not equivalent to being authenticated.

What Keycloak supports

From its own MCP integration guide:

Standard	Keycloak
OAuth 2.1 and authorisation server metadata	Supported
RFC 9207, issuer in the response	Supported
RFC 7591, dynamic registration	Supported
Client identifier metadata document	Supported, experimental, after enabling the feature
RFC 8707, resource indicators	Not supported
RFC 9728, protected resource metadata	Out of scope, it belongs to the MCP server

The conformance declared by the project itself is “partially supported, without resource indicators” for the last three MCP revisions.

Two practical conclusions:

The protected resource metadata document is published by the MCP server or a proxy in front of it, never by Keycloak. That is correct and there is nothing to fix, but you have to remember to implement it, because the client needs it to discover where to go.
Without resource indicators, the audience is bounded with scopes. The official route is to use the scope parameter instead of resource, and to hang from that client scope an audience mapper with the MCP server URL as the included custom audience.

It is a workaround, it works, and it has to be documented in the design because anyone coming after will look for the standard parameter and will not find it.

Part 5. The confused deputy

With the above on the table, the classic mistake becomes obvious on its own. A token with a generic audience turns any compromised service into a master key: whoever obtains the token a user presented to the dashboard can present it to the gateway, and from there to the MCP server.

The MCP specification is explicit in three sentences worth quoting verbatim in any architecture document:

MCP servers must not accept tokens that were not explicitly issued for them.
If the MCP server calls upstream APIs, it must not forward the token it received from the client.
MCP proxies that use static client identifiers must obtain user consent for each dynamically registered client before forwarding to third-party authorisation servers.

Translated into the specific figure this track has been dealing with: the gateway must not forward the user’s JWT to third-party MCP servers. It has to obtain a different token with the right audience, or act as an OAuth client in its own right. By default, as already documented in the MCP gateway post, the behaviour is to use the server’s own credential, which is the correct one; there are two modes that forward the caller’s token and both have a name in the specification.

Part 6. Two hours of task against five minutes of token

The specification pushes towards short-lived tokens and mandatory refresh token rotation in public clients. A two-hour agentic task comfortably exceeds the life of an access token and may also exceed session idle time if nobody refreshes.

Three real options, with their cost:

Offline tokens. They survive session expiry, and expire through their own idle timeout or through a maximum if that limit is enabled. The risk is obvious: a near-permanent credential, hard to inventory. And the MCP specification adds that MCP servers should not include that scope either in the authentication header or in the supported scopes of their metadata.

Automatic refresh in the client. It is the right answer when there is a client that can implement it, with refresh token rotation and secure storage. For an agent with no person behind it, it is better to renew via client credentials.

Long-lived virtual keys in the gateway. It works, and it is what most people end up doing. The price has to be written down in the risk analysis: identity ends up decoupled from the provider, so deactivating someone in Keycloak does not invalidate their key, and traceability back to the real person depends on the discipline with which the keys were created. The levers that compensate are expiry, automatic rotation and a mandatory alias, covered in the virtual keys post.

A note of honesty about the default values. The numbers going around (five minutes of access token, thirty of session idle time, ten hours of maximum) do not appear published in the administration guide and I did not locate them in the code. Treat them as a console observation and confirm them in your own realm before quoting them in a document.

Part 7. Sizing and operating

On Kubernetes

The operator exposes three custom resources: the server, the realm import and the OIDC client. Probes and metrics live on management port 9000, with the health and readiness paths, and metrics in Prometheus format provided the corresponding option was enabled in the build phase. That port is not exposed publicly.

For high availability there are three shapes. A single cluster spread across zones, with no additional infrastructure. First-generation multi-cluster, with an external load balancer and an external Infinispan at each site. And second-generation multi-cluster, introduced in 26.7.0 and still in preview, which connects clusters without external Infinispan, using the synchronously replicated database as the source of truth, with less cache and more database load.

The official sizing figures

From the CPU and memory sizing concepts guide, which is still published:

1 vCPU for every 15 password logins per second, tested up to 300 per second.
1 vCPU for every 120 client credential grants per second, tested up to 2,000.
1 vCPU for every 120 token refresh requests per second, tested up to 435.
Leave a 150 % CPU headroom for spikes, startup and failover.
1,250 MB of memory per pod with realm data and 10,000 cached sessions. Keycloak allocates 70 % of the limit to the heap and around 300 MB outside it, so the limit comes from subtracting those 300 MB from the expected usage and dividing by 0.7.

The stated assumptions behind those figures include Argon2 with five iterations, sessions in the database and the default cache of 10,000 entries.

The difference between 15 and 120 is the whole story of Keycloak’s performance: the cost is in password hashing. Since 25.0, the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. Before it was PBKDF2, whose iterations were multiplied by ten in version 24. In FIPS mode PBKDF2 is still used.

From that comes an architectural consequence for an AI platform: machine workloads, which go through client credentials, cost eight times less than a human login. Sizing by number of services is cheap; sizing by morning login peaks is the expensive part.

The benchmark

From the 26.4 performance report, published on 1 October 2025: up to 2,000 logins per second and 10,000 refreshes per second with three pods, 74 vCPU in total and 8 GB per pod, over an Aurora PostgreSQL database, with almost linear vertical scaling.

Two useful findings from that report. With 20 ms of round-trip time between sites, the 99th percentile rises from 51 to 130 ms, and the project advises against deploying across different regions. And raising the cache from 10,000 to 200,000 entries brought the database CPU peak down from 77.77 % to 63.77 %, in exchange for somewhat longer garbage collection pauses.

Security

The urgent matter, if the deployment is below 26.7.2:

CVE-2026-18963, score 9.1, critical. A bypass of the email action token in the credential reset flow allowed resetting any user’s password with no prior authentication, administrators included. Fixed in 26.7.2, released on 19 August 2026. The temporary mitigation, if you cannot upgrade, is to disable password recovery in all realms. No known exploitation was recorded as of the end of August.

In 26.7.3 there are another nineteen, among them the missing hostname verification in the TLS certificate towards LDAP with a score of 8.8, two tenant restriction bypasses in token exchange with Microsoft and Google brokers, and two access control bypasses in organisations.

Official hardening comes down to four decisions:

Strict hostname, which is the default value since the second generation of that configuration. The server does not infer the URL from headers, which prevents poisoning email links and redirects.
Administration console on a different hostname, and network-restricted.
Proxy headers declared explicitly, trusting only a controlled reverse proxy.
Separate management port, not exposed.

Part 8. The mapping to the ENS

The codes and titles below are verified against the official text of Royal Decree 311/2022 published in the BOE, Annex II.

Code	Literal title	What a central IdP contributes
op.acc.1	Identificación	A unique identifier per entity, user or process that gains access
op.acc.2	Requisitos de acceso	Permissions are granted by role and group, not by loose credential
op.acc.3	Segregación de funciones y tareas	It supports it with roles, but the control is organisational
op.acc.4	Proceso de gestión de derechos de acceso	Least privilege and a specific remote access policy
op.acc.5	Mecanismo de autenticación (usuarios externos)	Flows and factors configurable per flow
op.acc.6	Mecanismo de autenticación (usuarios de la organización)	Reinforcements required already at the low level
op.exp.8	Registro de la actividad	It contributes part of the events, not all of them
op.exp.10	Protección de claves criptográficas	Management and rotation of the realm signing keys

Two nuances an auditor is going to look for.

The first, about op.acc.1: the text requires every entity that accesses the system, whether user or process, to have a unique identifier. That includes agents. A virtual key shared between several agents does not comply, and that is exactly what usually happens when JWT authentication falls outside the budget.

The second, about op.exp.8: the control requires recording the identifier of the user or entity associated with the event, the date and time, what information it is performed on, the type and the result. Keycloak contributes the authentication events. The usage events are contributed by the gateway and the MCP servers, and there the two already documented gaps reappear: adding and removing MCP servers generates no audit log and the trace user is set by the client.

The correspondence between control and product in this table is my own interpretation and must be validated by the certification body. The codes and titles are not: those are literal from the BOE.

Part 9. The configuration to leave written down

Three concrete pieces, so as not to stay at the level of concepts.

Hardened startup, as container arguments:

kc.sh start --optimized \
--hostname=https://sso.ejemplo.es \
--hostname-admin=https://sso-admin.interna.ejemplo.es \
--proxy-headers=xforwarded \
--health-enabled=true \
--metrics-enabled=true \
--http-management-port=9000 \
--cache=ispn --cache-stack=jdbc-ping \
--db=postgres

The metrics, health, cache and database options are build options: they go into the image, and passing them here forces a rebuild if they do not match.

The client scope with audience, which is the workaround forced by the lack of resource indicators. Created with the administration tool:

# one scope per protected resource
kcadm.sh create client-scopes -r plataforma \
 -s name=mcp-inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

# the mapper that sets the token audience
kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocol=openid-connect \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

The client asks for that scope and receives a token whose audience is only that MCP server. The MCP server validates the audience and rejects any other token, as the specification requires.

Realm import via GitOps, with the operator’s custom resource, so that the configuration does not live only in the console:

apiVersion: k8s.keycloak.org/v2beta1
kind: KeycloakRealmImport
metadata:
 name: plataforma
 namespace: identidad
spec:
 keycloakCRName: sso
 realm:
 realm: plataforma
 enabled: true
 # without this, the recovery email is attack surface
 resetPasswordAllowed: false
 bruteForceProtected: true
 sslRequired: all

Part 10. What Keycloak does not solve

Fine-grained authorisation over resources. The question “can this person see this particular trace” is a relation between objects, not a role. That problem is solved by Zanzibar-style engines, with OpenFGA and its object, relation and user tuples, or a policy language such as Cedar. Keycloak gives roles and scopes, and with those you do not model a graph of relations.

Machine secrets. Storage credentials, provider keys, database passwords. That is a secrets manager, covered in stack hardening.

Cryptographic workload identity with attestation. That is SPIFFE and SPIRE. In Keycloak it exists as an identity provider in preview, which is a different thing.

Authorisation inside the model. Which MCP tool an agent may invoke, and with what arguments. The specification offers scopes, the insufficient scope error and step-up elevation, but the per-tool and per-argument decision is taken by the MCP server or a policy engine, not by the identity provider.

Antipatterns

Forwarding the user’s token from the gateway to third-party MCP servers. Explicitly forbidden by the specification.
Static client identifier with dynamic registration and no per-client consent, which is the exact recipe for the confused deputy.
General scopes, of the full-access kind, and publishing the entire catalogue of supported scopes in the metadata.
Treating a state handler as authentication, which the new revision expressly forbids.
Following OAuth discovery towards any URL, which opens the door to requests towards internal addresses. It also applies to the authorisation server when it downloads a client metadata document.
Trust headers badly exposed in chat front ends. The Open WebUI documentation warns that an incorrect configuration allows authenticating as any user.
Starting in development mode in production, with the hostname not resolved strictly.
A single shared static key as the only control between the gateway and the engine.

Startup checklist

Check the version. Below 26.7.2 there is a critical password reset bug.
Build the image with the build phase done and start with the optimised flag.
Strict hostname, administration console on another hostname, separate management port and not exposed.
Decide between organisations and one realm per tenant before creating the second tenant.
One client scope per resource, with an audience mapper, because resource indicators are not supported.
In the gateway, define the two audience and issuer variables. Without them none of that is validated.
For workloads inside the cluster, look at the Kubernetes identity provider before handing out client secrets.
Size by login peaks, not by number of services: the difference is eightfold.
Export the Keycloak events to the same place the gateway’s go, because the activity logging control needs them together.
Write into the design what the system does with a two-hour task, before someone solves the problem with an offline token.

Closing

Keycloak is a mature piece that solves one bounded problem well: knowing who the caller is, and saying so verifiably in a document that expires. In a sovereign inference platform that is worth a lot, because the alternative is an inventory of static keys spread across ten services that no audit can walk through.

What is not advisable is asking more of it. Fine-grained authorisation lives in each service, workload identity with attestation lives in SPIFFE, and the decision of which tool an agent may call lives in the MCP server.

And there is one concrete gap worth noting down, because it does not close on its own: the MCP specification marks two standards as mandatory, and of the two that would fall to an authorisation server, resource indicators are not implemented. The workaround with scopes and an audience mapper works, but it is a workaround, and whoever comes after will look for the standard parameter. Leaving it written in the design saves an afternoon.

Sources

Keycloak 26.7.3, 31 de agosto de 2026, y descargas.
Configuración del servidor, caché, bases de datos y nombre de host.
Integración con Model Context Protocol, intercambio de tokens y autenticación federada de clientes, 27 de enero de 2026.
Dimensionado de CPU y memoria e introducción a alta disponibilidad.
Bancos de pruebas de rendimiento de Keycloak 26.4, 1 de octubre de 2025.
Especificación de MCP, revisión 2026-07-28, autorización y buenas prácticas de seguridad.
KEP-3331, configuración estructurada de autenticación.
Real Decreto 311/2022, Anexo II.
Registro de CVE-2026-18963 en la base de datos de Red Hat y notas de la versión 26.7.3 en GitHub.
CNCF, proyecto Keycloak.

Langfuse v4, day 2 (1 of 10): the data model changed underneath, and what really goes into a trace

Sat, 12 Sep 2026 07:00:00 +0200

First article in a series of ten about operating Langfuse v4 in production. The architecture of version 3 and its ten performance levers were already covered in Langfuse from the inside; the seam with the gateway, in the operational pair. This series is about day 2. Verified against Langfuse 4.35.0, commit of 11 September 2026.

TL;DR

Version 4 is observations first. There is no trace table. A trace is a set of observations that share an identifier, and the trace attributes live denormalised in every row: name, user, session, tags, version, release and environment (clickhouse/migrations/canonical/0039_create_events_full.up.sql). The whole intuition of “first I create the trace and then I hang things off it” stops describing the storage.

There are ten observation types, not three. Alongside the classic SPAN, EVENT and GENERATION come AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (packages/shared/src/domain/observations.ts:5). Using the right type changes what the interface knows how to group.

The real limits are in the code and they do not match what people assume. The body of a request to the classic ingestion route is capped at 4.5 MB (api/public/ingestion.ts:40). The OTLP body, at 512 MiB, compressed and decompressed, with a 413 when it is exceeded. An OTLP span above 9.5 MB is only written to the log, not rejected. Identifiers accept 800 characters. The trace name, 1,000.

The ingester looks for the user in five places and the session in six. With strict precedence order, starting from Langfuse’s own attributes and ending in the metadata of other frameworks. Among the session ones sits gen_ai.conversation.id, which is the OpenTelemetry standard, so a conformant instrumentation works without translation.

Scores do not travel over OTLP. Zero occurrences of the word across the 3,864 lines of the OTLP ingestion processor. They come in through the classic ingestion route, through the scores endpoint or through the interface. Any design that assumes an evaluator can write its score down the same channel as the trace is wrong.

The classic masking function does not cover what almost everyone thinks it covers. It only acts on data that passes through the SDK calls. The attributes produced by a third-party automatic instrumentation come out unmasked. There is a new function that does act on the batch attributes, but it leaves out span events, which is where the recent conventions put the messages.

Filtering by tags or by status message is a scan. In the listings table, the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the dates, the model and the metadata names. The name, the tags, the version and the status message have no index.

You are here: OBSERVE, day 2

The previous series about this pair covered selection and setup. The six-service architecture explained version 3; tracing with OpenTelemetry explained the pipeline; the pair with the gateway explained where the trace gets lost between the two pieces.

What is left uncovered is the work of the next three years: migrating, sizing ClickHouse, retaining and deleting, taking backups, alerting, getting the data out. This series is about that, and it starts with the question that conditions everything else, which is what gets stored.

The analogy: the archive that changed system

An archive with a folder per case file and documents inside each folder is easy to understand. You open the folder, you look at what is there. The problem shows up when you have to find every document of one kind over the last two years: you have to open every folder.

Large archives solve that the other way around. There is no folder: each document carries, copied into its header, the case reference, the client name and the date. Searching by client is immediate. Reconstructing a case file is grouping by reference. In exchange, each document weighs a little more, and what gets written into that header matters a great deal, because everything you can search for comes from there.

Langfuse v4 made exactly that change. What used to be a folder with attributes of its own now travels repeated in the header of every document. The two operational consequences of this article come from there: which fields are worth filling in and which fields are not worth filling with content.

The data model, as it is in the code

The tables

Three new pieces in ClickHouse:

events_full (migration 0039): the table of record, with the full input and output, compressed with ZSTD level 3 and indexed with full-text indexes over the lowercase versions. ReplacingMergeTree engine ordered by event timestamp and deletion flag, partitioned by month of the start time.
events_core (migration 0040): the listings table, lighter.
events_core_mv (migration 0041): the materialised view that fills the previous one, and which truncates to two hundred characters the input, the output and each metadata value.

That truncation to two hundred characters is the first thing to know. What you see in a listing is not what is stored; it is a clipping. And when reading a full trace a second cap applies on top, LANGFUSE_SERVER_SIDE_IO_CHAR_LIMIT, with a default value of a thousand (packages/shared/src/env.ts:513).

In the events_full table there is a configuration line that says a lot: index_granularity_bytes = '64Mi', with the comment that it avoids very small granules because of large rows. It is the schema itself acknowledging that the rows of this system are bulky.

The ten observation types

SPAN, EVENT, GENERATION, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (domain/observations.ts:5), with their corresponding ingestion events (server/ingestion/types.ts:279).

For a platform with agents, the difference between marking a tool call as a generic SPAN or as a TOOL is the difference between being able to count tool calls per session and having to reconstruct it by hand. It is worth reviewing the instrumentation with this list in front of you.

The level is still DEBUG, DEFAULT, WARNING and ERROR (domain/observations.ts:31), with a free-form status message.

The limits, with their source

Field	Limit	File
Trace, span and observation identifiers	800 characters, no carriage return	`ingestion/types.ts:10`
Trace name	1,000 characters	`ingestion/types.ts:428`
Environment	40 characters, pattern of lowercase, digits, hyphen and underscore	`ingestion/types.ts:226`
Text-type score	500 characters	`domain/scores.ts:44`
Score config name	35 characters	`domain/score-configs.ts:11`
Classic ingestion body	4.5 MB per request	`api/public/ingestion.ts:40`
OTLP body	512 MiB, adjustable	`web/src/env.mjs:583`
OTLP span considered large	9.5 MB, only logged	`packages/shared/src/env.ts:193`
Name, status message, version, release, tags	no declared limit	`ingestion/types.ts:424`
Metadata	no declared limit	`ingestion/types.ts:425`

The last two are the dangerous ones, because they have no schema cap and the real ceiling ends up being the one on the request body. A three-megabyte metadata value passes validation and reaches ClickHouse.

A note on the langfuse prefix in the environment field: it is reserved for the product’s own internal traces and is stripped idempotently from public inputs (ingestion/types.ts:228), including the case of someone trying to sneak it in duplicated. The default value is default.

On rate limits: in the cloud there are per-plan quotas, but in self-hosted they are not applied. The service returns directly that there is no limiting (features/public-api/server/RateLimitService.ts). It is good news for bulk exports and bad news for anyone relying on that mechanism as protection.

The two instrumentation routes

What gets lost over OTLP, and it is one specific thing

The check is simple and the result is blunt: there is not a single occurrence of the word score in the OTLP ingestion processor, a file of 3,864 lines. Scores only come in through three places: the score-create event of the classic ingestion route, the POST /api/public/scores endpoint, or the interface.

This has a design consequence worth being clear about before building evaluators: the trace can travel through the OpenTelemetry collector with everything good that brings, on-disk queue, retries and sampling, but the score an evaluator puts on it has to go down a different channel and with the project keys.

Everything else does travel over OTLP. Whatever is not in the mapping lists is kept as metadata converted to a string, visible but outside the indexed fields.

The precedence chain, which is the map you need to have

Verified in packages/shared/src/server/otel/OtelIngestionProcessor.ts:

User identifier (line 2545), in this order: langfuse.user.id, user.id, the observation metadata, the trace metadata, and lastly the Vercel SDK metadata.

Session identifier (line 2565): langfuse.session.id, session.id, gen_ai.conversation.id, observation metadata, trace metadata, Vercel metadata.

Environment (line 2383): langfuse.environment, deployment.environment.name, deployment.environment; for each key the span attributes are checked first and the resource attributes after.

Model (line 2753): langfuse.observation.model.name, gen_ai.response.model, ai.model.id, gen_ai.request.model, llm.response.model, llm.model_name, model.

Level (line 1250): the native attribute, with accepted aliases where SUCCESS and OK become DEFAULT, WARN becomes WARNING and FATAL or CRITICAL become ERROR; if there is no attribute and the span status is an error, the result is ERROR.

Tags (line 3227): langfuse.trace.tags, langfuse.tags, observation and trace metadata, Vercel metadata, and tag.tags.

Input and output (line 1790): a chain of some twenty frameworks in strict order, starting from Langfuse’s own attributes and going through Genkit, the Vercel SDK, the standard inference operation details event of the conventions from 1.37 onwards, the per-role message events, the prompt and completion attributes, Logfire, MLflow, TraceLoop, OpenInference and the gen_ai.input.messages and gen_ai.output.messages attributes.

That this chain has twenty entries says something about the real state of the conventions: it was already noted that the generative AI conventions were extracted into their own repository without any published version. While that remains the case, the backend has to guess, and it guesses with a precedence list.

The practical part: if Langfuse’s own attributes are emitted, they always win. If you want backend-neutral instrumentation, gen_ai.conversation.id and deployment.environment.name are the two standard attributes that Langfuse does understand.

Metadata, and how it ends up in the table

A metadata value is flattened into two parallel arrays, one of names and one of values, with dot-separated paths and everything converted to a string (otel/utils.ts:239, applied in worker/src/services/IngestionService/index.ts:363). A nested object with twenty keys is twenty entries in each array.

There is a cap of 10,001 positions when reconstructing arrays from OTel attributes (OtelIngestionProcessor.ts:271), which is high but it exists.

Sessions, users and cardinality

A session is a text column with a Bloom filter index at 1 %. The same goes for the user. The Postgres table that accompanies sessions stores only whether it is bookmarked, whether it is public and its environment; there is no counter and no cap. There is no limit on traces per session in the code, and the documentation recommends session identifiers below 200 ASCII characters.

The cardinality difference matters and it is not intuitive: user and session are plain text, so high cardinality does not break the schema, it only inflates the Bloom filters. The environment, by contrast, is declared low cardinality. Generating a dynamic environment value per tenant degrades ClickHouse measurably.

Aggregated cost comes out of a cost details map with materialised input, output and total columns. The classic sessions endpoint is deprecated; the replacement is the version 2 observations one with a filter.

Scores: five types and three sources

The types are NUMERIC, CATEGORICAL, BOOLEAN, CORRECTION and TEXT (domain/scores.ts:45). The boolean requires a value of zero or one. The correction does not accept an associated config. The text one is capped at 500 characters.

The sources are API, EVAL and ANNOTATION. The EVAL source is reserved for the internal evaluators: the public API only accepts the other two. Annotation scores require an associated config except for the correction one.

On what costs money, a clarification that corrects a widespread belief: in 4.35.0 the entitlement for the number of model-based evaluators is unlimited on every plan, including the open edition (features/entitlements/constants/entitlements.ts:60). What is limited by plan is the number of annotation queues and the days of data access. And there is a version nuance: trace-level evaluators are discontinued in version 4.

Masking, and what it does not cover

This is the part to read twice if the deployment has to comply with anything.

The classic Python SDK function has the signature masking_function(*, data, **kwargs) and covers only the data that passes through the SDK calls: observation creation, update, and setting the trace input and output. It does not cover the raw OTel attributes produced by a third-party automatic instrumentation.

The replacement is a function that acts on the attributes and the resource attributes in the export batch. That one does reach automatic instrumentation, with a limitation declared in the documentation: it cannot change the span name, the identifiers, the parent relationship, the resource attributes, the events or the links.

And there is the gap that matters: the generative AI conventions from 1.37 onwards put messages and responses into span events. That is, the route by which prompts arrive in a modern instrumentation is precisely the one the new masking does not touch.

For a deployment with personal data, the conclusion is that client-side masking is not enough and a redaction processor has to go into the collector. It is the same reasoning that led in the previous article to recommending the collector in the middle, and it adds to the already documented problem of MCP tool arguments being written in clear, bypassing the message redaction switch.

On the server side there is no safety net: Langfuse has no server-side personal data masking. The encryption key that gets configured protects secrets, that is, model provider credentials, integration headers and SSO configurations. Trace inputs and outputs are not encrypted at application level.

Volume control

Three levers, in order of how high up they act:

In the SDK: sample_rate, or the LANGFUSE_SAMPLE_RATE variable, with head sampling at trace level. If the trace is not sampled, neither its observations nor its scores travel. LANGFUSE_TRACING_ENABLED set to false switches the whole thing off.

In the collector: probabilistic sampling and tail sampling. It is the only layer where you can decide based on the outcome, that is, keep every error and a fraction of the rest.

In the server: a deterministic per-project sampling, with a SHA-256 hash of the trace identifier (server/ingestion/sampling.ts). Deterministic per trace, so it does not cut traces in half.

What not to put into a trace

With the schema in front of you, the list is concrete.

Do not filter by tags or by status message. In the listings table the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the creation and update dates, the model name, the experiment identifier and the metadata names. The name, the tags, the version, the release and the status message are not. Filtering by them scans.

Do not use metadata as a store. It gets flattened into string arrays, truncated to two hundred characters in the listings table but stored whole in the full table, and on top of that it is indexed with a full-text index segmented on non-alphabetic characters. Bulky metadata multiplies the inverted index, not just the data.

Do not store full prompts without thinking about it. The input and the output carry ZSTD level 3 compression and two full-text indexes over their lowercase versions. Storing the entire context of a 142,000-token agentic call on every turn inflates the inverted index disproportionately. Client-side truncation is the cheap lever.

Do not use the environment to separate tenants. It is low cardinality, it has a 40-character cap and it isolates nothing: same project, same keys, same limits, same retention. It is there to segment dashboards. Real isolation means separate projects, as was already argued in the article on the seams.

Version 3 and the calendar date

The constant is in the code: V3_SUNSET_DATE = "2026-11-16" (features/public-api/server/deprecations.ts:11).

It is worth being precise about what it means, because the public documentation and the code do not say exactly the same thing. The code comment is literal: the classic ingestion route is never switched off and continues to accept score events; trace and observation events fail only in version 4’s exclusive write mode, not in the mixed or classic modes. And when they fail, they fail per event, not per batch.

The x-langfuse-ingestion-version: 4 header activates direct writing to the events table, which is the real-time route. Without it the deferred path is taken. Recent SDKs send it by themselves. Any value above four is rejected.

As for names, trace-level input and output attributes are deprecated in favour of the observation ones on the root span. The classic read endpoints carry a deprecation marker in the response body, and their replacements are version 2 of observations, version 3 of scores and version 2 of metrics. And dataset runs are renamed experiments.

There is a curious piece that deserves a mention because it shows up in the Redis dashboards and is misleading: a queue called version 4 classic API usage, which ingests nothing. It is a periodic task every fifteen minutes that scans the ClickHouse query log to detect which projects are still using the old API, and materialises the result into Redis for the interface’s migration notice.

How this looks in code

Two pieces, the client one and the collector one.

On the application side, what changes compared with the usual instrumentation is the observation type and where the identity comes from:

from langfuse import get_client, propagate_attributes

lf = get_client()

# identity propagates by context, not observation by observation
with propagate_attributes(
 session_id=id_conversacion, # the same across every turn
 user_id=sujeto_del_token, # from the claim, not from the body
 environment="produccion",
 tags=["soporte", "n2"],
):
 # the type matters: the interface groups by it
 with lf.start_as_current_observation(
 as_type="agent", name="resolver-incidencia"
 ) as raiz:

 with lf.start_as_current_observation(
 as_type="tool", name="buscar_en_inventario"
 ) as herramienta:
 resultado = mcp.call("buscar", args)
 # truncate here, not on the server
 herramienta.update(output=str(resultado)[:4000])

 # the score goes down its own channel, never over OTLP
 lf.score_current_trace(name="resuelta", value=1, data_type="BOOLEAN")

The as_type parameter accepts nine literals: span, generation, embedding, agent, tool, chain, retriever, evaluator and guardrail. The tenth type in the model, the event, has its own constructor because it does not open a scope.

On the collector side, the redaction processor that compensates for what the SDK masking does not reach:

processors:
 redaction:
 allow_all_keys: true
 blocked_values:
 - '\b\d{8}[A-HJ-NP-TV-Z]\b' # national ID number
 - '\b[\w.+-]+@[\w-]+\.[\w.]+\b' # email
 summary: silent
 tail_sampling:
 policies:
 - name: errores
 type: status_code
 status_code: {status_codes: [ERROR]}
 - name: resto
 type: probabilistic
 probabilistic: {sampling_percentage: 5}

exporters:
 otlphttp/langfuse:
 endpoint: http://langfuse-web.observabilidad.svc:3000/api/public/otel
 headers:
 Authorization: "Basic ${LANGFUSE_BASIC}"
 x-langfuse-ingestion-version: "4"
 sending_queue:
 enabled: true
 storage: file_storage
 retry_on_failure:
 enabled: true

The endpoint carries the base path without the signal suffix: the exporter adds its own. And the on-disk queue is the main reason for putting the collector in the middle, because the SDK’s batch processor drops when it fills up.

Instrumentation checklist

Review the type of every observation against the list of ten. Tool calls as TOOL, agents as AGENT, guardrails as GUARDRAIL.
Decide where the user identifier comes from and set it in a single place, high up the precedence chain.
Emit gen_ai.conversation.id if you want neutral instrumentation, or Langfuse’s own attribute if coupling does not matter.
Set the environment with deployment.environment.name, with few distinct values and never one per tenant.
Get scores out through their own channel, with the project keys, and do not expect them to travel with the trace.
Truncate input and output on the client before sending them, especially with long agentic contexts.
Put a redaction processor in the collector, because the SDK masking leaves out span events.
Do not filter by tags in dashboards that get queried often.
Send the ingestion version header, or use recent SDKs that send it by themselves.
Review the size of the metadata: there is no schema cap, the cap is the one on the request body.

Traps and things that are not what they look like

What you see in a listing is truncated to two hundred characters by the materialised view, and the full read has another cap of a thousand. The data is whole in the full events table.

Metadata has no size limit in the schema. A blob of megabytes passes validation.

An OTLP span over 9.5 MB is not rejected, it is only logged. There is no signal back to the client.

The environment is declared low cardinality. One value per tenant degrades ClickHouse.

Scores do not travel over OTLP. It is by absence in the code, not by configuration.

The classic masking does not cover automatic instrumentation, and the new one does not cover span events, which is where the recent conventions put the messages.

Langfuse does not encrypt trace inputs or outputs. The encryption key protects integration credentials.

In self-hosted there are no rate limits on the API. It is no use as protection.

The classic ingestion route is not switched off on 16 November, it changes behaviour depending on the write mode, and it fails per event instead of per batch.

Model-based evaluators are not limited by plan in this version, contrary to what is usually assumed. What is limited are the annotation queues and the days of data access.

The series: the ten articles

This is the first. The order is designed so that each one can be read on its own and so that together they cover the whole of day 2.

What goes into a trace (this article): version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Anuncio de Langfuse v4, 17 de agosto de 2026.
Compatibilidad y migración, consultado el 12 de septiembre de 2026.
Sesiones, muestreo y enmascarado, consultados el 12 de septiembre de 2026.
Integración nativa con OpenTelemetry, consultado el 12 de septiembre de 2026.

The gateway does not live alone: the JWT that does not validate the audience, the user who never reaches the trace, and the seams you have to write by hand

Sat, 12 Sep 2026 06:30:00 +0200

Eighth article in the operational track of the control layer. The previous seven deal with the gateway from the inside: the pair with Langfuse, day 2, virtual keys, humans and agents, prefix routing, the MCP gateway and sizing for agents. This one deals with what surrounds it. Verified against LiteLLM 1.102.0 and Langfuse 4.35.0, both with a commit of 11 September 2026.

TL;DR

The gateway has five neighbours: a database, a cache, an identity provider, a trace backend and a collector. Each neighbourhood has its own physics, and these are the seven things it decides.

The proxy starts without Postgres, but then only the master key works. The DATABASE_URL variable is optional (proxy_server.py:1163); without it, any key other than the master one returns a 400 for a missing connection (user_api_key_auth.py:1886). With a database configured but down, the default behaviour is to return 503, unless general_settings.allow_requests_on_db_unavailable is switched on.

Redis is never mandatory, and that is precisely the trap. There is a circuit breaker with a threshold of five failures and a recovery of sixty seconds (constants.py:456). When it opens, the proxy fails no request: it degrades to local memory. What is lost silently is that rate limits stop being global and become per pod.

No observability backend can bring a request down. All callbacks are launched with asyncio.create_task and their exceptions are caught and logged (litellm_logging.py:3235). If Langfuse does not respond, events are discarded as the queue fills up. There is no added latency, and there is no reliable record either: the pair’s observability is a statistical figure, not an audit record.

JWT authentication validates neither the audience nor the issuer unless two variables are defined. In _build_decode_kwargs (handle_jwt.py:1003) the audience comes from JWT_AUDIENCE and the issuer from JWT_ISSUER; if they are empty, the code sets verify_aud=False and verify_iss=False and emits a warning once only. Any valid token signed by that provider, issued for another service, is accepted.

Role-based access control ships disabled. enforce_rbac has a default value of False (_types.py:4848 onwards). With that value, a token whose role resolves to nothing does not produce a 403: it carries on through the flow. Real authorisation, then, comes from team membership, if there is a team claim configured at all.

The user identifier that reaches Langfuse is not the one from Keycloak. The user_id field of the trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604), that is, with the user field of the OpenAI body, which the client sets. The subject of the token and the owner of the key do not travel. Anyone who wants per-person traceability needs to configure end_user_id_jwt_field or make the client send that field.

Langfuse provisions nothing from a claim. There is no variable that maps an OIDC claim to an organisation or to a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID and LANGFUSE_DEFAULT_PROJECT_ID, the same for everyone. On top of that, per-project access control sits behind the Enterprise licence in the self-hosted edition, as do audit logs and per-project retention.

You are here: between the four pieces

The LiteLLM and Langfuse pair walked through the seam between the gateway and the trace backend: integration routes, cost that arrives as zero, correlation and queues with discarding. This article widens the focus to the other three neighbourhoods, with identity at the centre, because that is the one that orders everything else and the one an auditor asks about.

The identity piece itself, that is, Keycloak, is dealt with in the next article. Here we deal with the gateway side.

The analogy: the building site with five trades

On a small building site, the problem is rarely inside each trade’s work, it is at the joins: where the plumber leaves the pipe sleeve and the bricklayer closes up the partition wall, where the electrician wants to run through the same gap. Each trade does its part well and the join ends up badly resolved because it belongs to nobody.

An inference platform has five trades. The gateway knows about tokens, the identity provider knows about people, the trace backend knows about events, the database knows about spend and the cache knows about counters. Each one has good documentation. The joins have nobody’s documentation, and that is where the user’s identity gets lost between the token and the trace.

There is one more detail of the analogy that is useful. On a building site, the part that causes the most problems is not the one that looks badly finished, it is the one that gets covered up. The same happens here: the five seams in this article fail silently.

Seam 1. What brings things down and what degrades them

It pays to have written down what happens when each neighbour falls over, because intuition fails in almost every case.

Neighbour	Does it start without it?	If it falls over
Postgres	Yes, master key only	503 by default; degraded with `allow_requests_on_db_unavailable`
Redis	Yes	Circuit breaker after 5 failures; limits become per pod
Langfuse	Yes	Events discarded as the queue fills; no impact on latency
OTel collector	Yes	Same as the previous one
Identity provider	Yes	Access to the interface and the JWT flow goes down; virtual keys carry on

Postgres

Without DATABASE_URL, _setup_prisma_client returns None without error (proxy_server.py:10375). The proxy serves /chat/completions with the master key. Any other key gets a 400. There is no global budget either, and start-up itself warns about it: Redis does not replace the database (proxy_server.py:9240).

With a database configured and down, the exception handler re-raises unless the degradation flag is active (db/exception_handler.py:56). There is also a DISABLE_PRISMA_HEALTH_CHECK_ON_STARTUP and a watcher that reconnects. The detail that matters for diagnosis: a transient database failure returns 503, not 401. If during an incident you see 401s, the problem is credentials; if you see 503s, it is the database or admission.

Redis

The circuit breaker is enabled by default, with a threshold of five failures, a recovery of sixty seconds and a minimum duration of five (constants.py:456, logic in caching/redis_cache.py:153). The socket has a timeout of 0.1 seconds (constants.py:453). With the circuit open, DualCache serves from memory without propagating an exception (dual_cache.py:215).

The consequence has to be measured, not assumed: with four workers and three replicas, a limit of one hundred requests per minute becomes one thousand two hundred. It is a failure mode that generates no errors and that can last for days without anyone noticing, until the bill arrives or the engine saturates.

The right alert is not about Redis, it is about litellm_service_latency labelled by service (integrations/prometheus_services.py:119), which exposes Redis and database latency separately.

The callbacks

Everything behind, that is, Langfuse and the collector, is launched outside the response path and its failures are caught. The classic integration uses the v2 SDK with its own background thread and a flush interval of one second, adjustable with LANGFUSE_FLUSH_INTERVAL (langfuse.py:198). There is a cap of fifty instantiated Langfuse clients (constants.py:554), because each client is a thread.

The OTLP integration uses the batch processor of the OpenTelemetry SDK (opentelemetry.py:3060) with its default values. In both cases, saturation means discarding, and discarding generates no error towards the client.

Seam 2. Identity: the three doors and what each one validates

Everything goes through a single authentication constructor (user_api_key_auth.py:1245), in this order:

Custom authentication from the enterprise edition.
Public routes.
Opaque OAuth2, if enabled, with a licence check.
JWT, if general_settings.enable_jwt_auth is active and the token has three parts.
Master key, with constant-time comparison.
Virtual key, with SHA-256 hash and lookup in the verification table.

A JWT works for /chat/completions, not just for administration: the routes allowed by default for a team include openai_routes (_types.py:4886), which contains the chat endpoint. Administration, on the other hand, is only reachable by a token with an administrator role.

The first thing to know: it is paid for

The JWT branch carries an explicit licence check, with the literal message that JWT authentication is an enterprise-only feature (user_api_key_auth.py:1416). SSO for the administration interface is free up to five billable users and requires a licence above that (ui_sso.py:976).

This conditions the architecture of any sovereign deployment that does not want to pay for a licence: the Keycloak integration is left for the administration interface and for provisioning, while inference traffic authenticates with virtual keys. Virtual keys are not worse, but they are a different thing, and it has to be said in the risk analysis: they are credentials decoupled from the identity provider, so removing a person from Keycloak does not invalidate their key.

The `litellm_jwtauth` fields that decide security

From the LiteLLM_JWTAuth class (_types.py:4848 onwards), these are the ones that matter, with their default values:

Field	Default	What it does
`enforce_rbac`	`False`	If false, an unresolved role does not produce a 403
`enforce_scope_based_access`	`False`	Without this, `scope_mappings` is not applied
`enforce_team_based_model_access`	`False`	Model access by team
`team_id_jwt_field`	`None`	Claim the team comes from
`team_id_upsert`	`False`	If false, a team that does not exist gives a 404
`user_id_upsert`	`False`	Same with the user
`public_key_ttl`	`600`	Cache of the public keys
`public_key_stale_ttl`	`3600`	Extra margin if the provider is down
`admin_jwt_scope`	`litellm_proxy_admin`	Scope that grants administration
`user_allowed_email_domain`	`None`	Restriction by email domain

The accepted algorithms are RS256/384/512, PS256/384/512, ES256/384/512 and EdDSA (handle_jwt.py:163). There is no symmetric one, which eliminates the algorithm confusion class of attack. It is a good design decision and it deserves recognition.

Key download supports several comma-separated URLs in JWT_PUBLIC_KEY_URL, resolves discovery if the URL points to the well-known OpenID configuration, retries three times with increasing waits and memoises the failure for thirty seconds (handle_jwt.py:680, :751, :88). If the provider is down, it serves the stale copy up to the sum of the two TTLs, that is, up to seventy minutes. For availability that is fine; for revocation it is not, and it has to be written down.

One detail of key rotation: with more than one key in the set, an exact match of the key identifier is required, but with a single key and no identifier it is accepted without checking (handle_jwt.py:913).

The validation that does not happen

This is the part to fix on day one. In _build_decode_kwargs (handle_jwt.py:1003), the audience is taken from the JWT_AUDIENCE environment variable and the issuer from JWT_ISSUER. If they are not defined, the code sets verify_aud=False and verify_iss=False, and emits a warning once only.

What that means in a real deployment with Keycloak: a token issued for the Grafana client, or for the Backstage one, signed by the same realm, is accepted by the gateway as a valid credential. Expiry is always validated, with zero margin. So is the signature. The audience is not.

There is a second route, issuers, with per-issuer configuration, which does validate issuer and audience unless explicitly disabled (handle_jwt.py:1186). That is the one to use when there is more than one provider.

The minimum fix is two lines of environment:

JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma

And on the Keycloak side, an audience mapper in a dedicated client scope that injects that audience. The detail is in the Keycloak article, because the standard that would resolve this cleanly, resource indicators, is not supported.

From claim to permission

The administrator role comes from admin_jwt_scope appearing in the scope claim, and it returns a result without going through a team (handle_jwt.py:309, :1373). The rest is resolved by team.

If the team in the claim does not exist in the database, with team_id_upsert false a 404 is returned with the message that the team has to be created (auth_checks.py:3035). With the flag active, it is created by invoking team creation with a synthetic administrator identity (auth_checks.py:2874). That is the only automatic provisioning route that exists from Keycloak towards the gateway, and it works well. It pays to know that it creates teams with no budget and no limits, so a default value is needed in the configuration.

The vulnerability worth knowing about

CVE-2026-35030, scoring 9.1 on CVSS 3.1 and 9.4 on CVSS 4.0, advisory GHSA-jjhc-v7c2-5hh6. With JWT authentication active, the cache of the OIDC user info endpoint used the first twenty characters of the token as the key, so that two tokens with the same prefix were confused with each other. Fixed in 1.83.0.

In 1.102.0 the code uses the full SHA-256 hash (handle_jwt.py:962). A search for similar patterns across the tree finds no authentication cache indexed by token prefix: the truncations that remain are masked for logging. And it is worth recalling what has already been published about the MCP gateway: below 1.83.14 there is unauthenticated remote execution or SQL injection.

Seam 3. Identity does not reach the trace

Here lies the most expensive design flaw in the arrangement, because it is discovered at the first audit.

The user_id field of a Langfuse trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604 and :724), which comes from the user field of the OpenAI request body or from end_user_id_jwt_field (litellm_pre_call_utils.py:1584). It is not the subject of the token nor the owner of the virtual key.

The knock-on consequences:

If the client does not send user, the trace has no user.
If the client does send user, the trace has whatever value the client cares to put there, unverified.
An agent calling with a service key leaves traces with no person behind them.

There are three ways to fix it, in order of robustness:

Configure end_user_id_jwt_field so that the end user comes from a token claim and not from the body. It is the only option in which the value is not controlled by the client, and it requires JWT authentication, that is, a licence.
Force the trace_user_id metadata field, which overwrites the field (langfuse.py:726; on the OTLP route, langfuse_otel.py:96). It can be set per team.
Contractually require the client to send user and validate it in a guardrail. It works, but it is a client declaration, not a proof.

Something similar happens towards the engine. The caller’s identity does not travel to vLLM unless litellm.add_user_information_to_llm_headers is switched on, whose default value is None (litellm/__init__.py:223). With it, x-litellm-* headers are injected with user identifier, team identifier and key hash (litellm_pre_call_utils.py:1356). Forwarding client headers requires general_settings.forward_client_headers_to_llm_api, and the authorization header is always filtered out, which is correct.

Towards MCP servers, by default the server’s own credential is used, not the user’s token. Only two modes forward the caller’s token, and doing so has a name in the MCP specification: token passthrough, which is explicitly forbidden. It comes up again in the Keycloak article.

Seam 4. Langfuse as a neighbour

What can be automated and what cannot

The exact list of variables for SSO in Langfuse 4.x, taken from web/src/env.mjs:

With the Keycloak provider: AUTH_KEYCLOAK_CLIENT_ID, AUTH_KEYCLOAK_CLIENT_SECRET, AUTH_KEYCLOAK_ISSUER, AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING, AUTH_KEYCLOAK_CLIENT_AUTH_METHOD, AUTH_KEYCLOAK_CHECKS, AUTH_KEYCLOAK_ID_TOKEN_SIGNED_RESPONSE_ALG, AUTH_KEYCLOAK_SCOPE, AUTH_KEYCLOAK_ID_TOKEN, AUTH_KEYCLOAK_NAME.

With generic OIDC: the same set with the AUTH_CUSTOM_ prefix, plus AUTH_CUSTOM_FETCH_USERINFO and claim mapping with LANGFUSE_CUSTOM_SSO_SUB_CLAIM, LANGFUSE_CUSTOM_SSO_EMAIL_CLAIM, LANGFUSE_CUSTOM_SSO_NAME_CLAIM and LANGFUSE_CUSTOM_SSO_IMAGE_CLAIM.

Global ones: AUTH_DISABLE_USERNAME_PASSWORD, AUTH_DISABLE_SIGNUP, AUTH_DOMAINS_WITH_SSO_ENFORCEMENT, AUTH_SESSION_MAX_AGE.

And now what is missing. There is no variable that maps a claim to an organisation or a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID, LANGFUSE_DEFAULT_ORG_ROLE with a default value of VIEWER, LANGFUSE_DEFAULT_PROJECT_ID and LANGFUSE_DEFAULT_PROJECT_ROLE (features/auth/lib/createProjectMembershipsOnSignup.ts:61). The documentation itself confirms it: enterprise SSO does not provision roles automatically at sign-up.

On top of that, in the per-edition permissions table (features/entitlements/constants/entitlements.ts:139), the open and self-hosted professional editions do not include per-project access control or the administration API. They belong to the enterprise edition, as do audit logs and per-project retention. In the open edition, every user inherits the role of their organisation.

The declarative bootstrap, which does exist

For on-premise without a licence, the workable route is bootstrapping through variables (web/src/initialize.ts), which is idempotent: LANGFUSE_INIT_ORG_ID (mandatory, if missing the rest is ignored with a warning), LANGFUSE_INIT_ORG_NAME, LANGFUSE_INIT_PROJECT_ID, LANGFUSE_INIT_PROJECT_NAME, LANGFUSE_INIT_PROJECT_RETENTION, LANGFUSE_INIT_PROJECT_PUBLIC_KEY, LANGFUSE_INIT_PROJECT_SECRET_KEY, LANGFUSE_INIT_USER_EMAIL, LANGFUSE_INIT_USER_NAME, LANGFUSE_INIT_USER_PASSWORD.

With that, one project per tenant is created from GitOps with one initialisation container per tenant and the keys in Kubernetes secrets.

Langfuse credentials per team

This part is well resolved on the gateway side and few people use it. TeamCallbackMetadata.callback_vars (_types.py:2152) accepts langfuse_public_key, langfuse_secret_key, langfuse_host and langfuse_environment, with an allow list in initialize_dynamic_callback_params.py:67. The gateway caches one logger per credential set (langfuse_handler.py:24) and, on the OTLP route, builds a full exporter per key (langfuse_otel.py:400).

That is to say: one Langfuse project per LiteLLM team is possible today, without a licence, and it is the correct way to isolate traces between tenants. What does not exist is automatic project creation, nor the correspondence between the team and the project. That is glue you have to write.

The ingestion route

Langfuse accepts only OTLP over HTTP, in http/protobuf or http/json. gRPC is not supported. The base route is /api/public/otel and the signal lives at /api/public/otel/v1/traces, verified in the repository. Authentication is Basic with the project’s key pair, plus the x-langfuse-ingestion-version: 4 header for the version 4 route.

The gateway builds it exactly like that (langfuse_otel.py:328, :347, :359) and then normalises the endpoint by adding the signal suffix (opentelemetry.py:3237). The result matches the real route, so the integration works without touching anything. The detail matters when a collector is put in the middle and somebody copies the endpoint by hand.

And the calendar warning that already appeared in the operational pair still stands: the classic callback is tied to the v2 SDK and the date marked is 16 November 2026. With a nuance worth reading before scheduling an emergency migration, which is detailed in the data model article: that date applies to Langfuse Cloud, and in the code of the self-hosted version the old route is not switched off, it changes behaviour according to the write mode.

When to put a collector in

Direct if Langfuse is the only trace destination. Via a collector if there is more than one consumer, or if any of these three things is needed that the proxy’s batch processor does not provide:

Retries with increasing waits and an on-disk queue, with the sending_queue and file_storage processors. With that, a Langfuse maintenance window stops losing traces.
Tail sampling, which lets you keep 100 % of errors and 1 % of the rest. The proxy cannot sample by outcome, because at the point it decides it does not yet know how things ended.
Attribute redaction before they leave the namespace. It is the only layer where you can delete the MCP tool arguments which, as has already been documented, are written in the clear, bypassing the message redaction switch.

Seam 5. Postgres, Redis and the memory policy

Separate Postgres, always. Each piece applies its own Prisma migrations over the public schema of the database it is given. Two Prisma clients migrating the same schema collide over table names and over migration history. On top of that, Langfuse 4.x requires Postgres 15 as a minimum, recommends 16 and wants UTC by default. The load profile is also the opposite: the gateway writes spend records at high frequency, while Langfuse does light transactional work because trace data goes to ClickHouse.

Separate Redis, and the argument is not the one it looks like. There is no key collision by design: the gateway uses hash tags of the form {api_key:...}:requests and an optional namespace, and Langfuse uses the BullMQ prefix plus its own prefix for its key cache. The decisive argument is another one: Langfuse requires maxmemory-policy=noeviction because its queues are data, not cache, while the gateway assumes a disposable cache. Sharing an instance means that a spike of rate limit keys can fill the memory and cause write errors in the ingestion queues instead of a harmless eviction.

If it is shared anyway, two minimum measures: REDIS_KEY_PREFIX in Langfuse, namespace in the gateway, and different logical databases.

And a reminder about surface area: Langfuse 4.x does not start without ClickHouse or without an S3-compatible bucket. CLICKHOUSE_URL and LANGFUSE_S3_EVENT_UPLOAD_BUCKET are mandatory and have no default value (packages/shared/src/env.ts:120 and :271). Its failure surface is far larger than the gateway’s, and that changes where to put the high availability effort.

Ports, for the NetworkPolicy

Piece	Port
LiteLLM (API, interface and metrics)	4000
Langfuse web	3000
Langfuse worker	3030
ClickHouse	8123 HTTP, 9000 native
S3-compatible storage	9000
Redis	6379
Postgres	5432
OTel collector	4317 gRPC, 4318 HTTP

The pairs to allow under a default-deny policy, following the method from hardening the stack:

litellm → postgres:5432, redis:6379
litellm → collector:4318 (or langfuse-web:3000 if going direct)
litellm → vllm:8000
litellm → keycloak:8443 (interface and JWT flow only)
collector → langfuse-web:3000
langfuse-web → postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-worker→ postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-web → keycloak:8443
ingress → litellm:4000, langfuse-web:3000
prometheus → litellm:4000

The Langfuse worker needs no inbound traffic beyond the health probe.

The glue you have to write

Put into a table, the real state of the tenant chain:

Segment	Automatic?	How
Keycloak → LiteLLM team	Yes	`team_ids_jwt_field` plus `team_id_upsert: true`
Keycloak → LiteLLM user	Yes	`user_id_jwt_field` plus `user_id_upsert: true`
Keycloak → Langfuse organisation	No	Static assignment at sign-up only
Keycloak → Langfuse project	No	Same
LiteLLM team → Langfuse project	No	`callback_vars` by hand, or an operator of your own
Keycloak user → trace `user_id`	No	`end_user_id_jwt_field`, or a contract with the client

The minimum viable architecture without a licence, then, is this:

One Langfuse project per tenant, created with the initialisation variables from GitOps.
That project’s keys in a Kubernetes secret.
A periodic job of your own that reads the teams through the gateway’s administration API and writes callback_vars with the corresponding keys.
People entering Langfuse through Keycloak, but assigned by hand to their organisation.
The end user identifier resolved by contract with the clients, and validated.

That is about one hundred and fifty lines of your own code. It pays to budget for them from the start, because the gap does not close by itself.

Reference configuration

general_settings:
 # only with a licence; without one, virtual keys
 enable_jwt_auth: true
 litellm_jwtauth:
 team_ids_jwt_field: "groups"
 team_id_upsert: true
 user_id_jwt_field: "sub"
 user_id_upsert: true
 end_user_id_jwt_field: "preferred_username"
 user_allowed_email_domain: "ejemplo.es"
 admin_jwt_scope: "litellm_proxy_admin"
 # the three that ship switched off
 enforce_rbac: true
 enforce_scope_based_access: true
 enforce_team_based_model_access: true
 allow_requests_on_db_unavailable: false
 proxy_batch_write_at: 60

litellm_settings:
 callbacks: ["langfuse_otel", "prometheus"]
 # without this, the engine does not know who is calling
 add_user_information_to_llm_headers: true
 turn_off_message_logging: true
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

# the two that enable the validation that by default does not happen
JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma
JWT_PUBLIC_KEY_URL=https://sso.ejemplo.es/realms/plataforma/.well-known/openid-configuration

LANGFUSE_HOST=http://langfuse-web.observabilidad.svc:3000
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...

And on the Langfuse side:

AUTH_KEYCLOAK_CLIENT_ID=langfuse
AUTH_KEYCLOAK_ISSUER=https://sso.ejemplo.es/realms/plataforma
AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING=true
AUTH_DISABLE_USERNAME_PASSWORD=true
AUTH_DISABLE_SIGNUP=true
LANGFUSE_DEFAULT_ORG_ID=plataforma
LANGFUSE_DEFAULT_ORG_ROLE=VIEWER
ENCRYPTION_KEY=... # 64 hexadecimal characters

Checklist

Define JWT_AUDIENCE and JWT_ISSUER on day one, or use the per-issuer configuration. Without that, neither audience nor issuer is validated.
Set enforce_rbac, enforce_scope_based_access and enforce_team_based_model_access to true, and test that a token from another client of the same realm is rejected.
Decide where the end user of the traces comes from and write it into the design. If it is not decided, traces come out with no person.
Separate Postgres. Separate Redis. If Redis is shared, different prefix and logical database, and review the memory policy.
Check that the proxy starts with Postgres down only if that is what you want, and that the on-call team can tell an admission 503 from a credential 401.
Alert on litellm_service_latency per service, not on Redis availability, because the circuit breaker hides the problem.
One Langfuse project per tenant with the initialisation variables, and the per-team keys in callback_vars.
Put a collector in the middle if you need an on-disk queue, tail sampling or attribute redaction.
Check that the gateway version is above 1.83.14 because of the earlier critical vulnerabilities.
Write into the risk analysis that virtual keys are credentials decoupled from the identity provider, and define the offboarding procedure.

Traps and things that are not what they seem

A properly signed JWT is not a JWT addressed to this service. Without the two environment variables, the audience is not checked.

enforce_rbac set to false does not mean there is no authorisation, it means that authorisation comes only from team membership, if there is a team claim. With no team claim and without the flag, a valid token reaches /chat/completions.

The stale copy of the public keys lasts up to seventy minutes, adding the two TTLs. Revoking a key at the provider does not take effect immediately.

With a single key in the JWKS and no key identifier, the identifier is not checked.

The trace user is set by the client. Any per-person spend report built on that field is a declaration, not a measurement.

Redis going down generates no errors, it generates per-pod limits. It is the most expensive of the failures that give no warning.

Langfuse does not start without ClickHouse or without an S3 bucket. Its failure surface is bigger than the gateway’s, and that changes how the high availability effort is shared out.

Langfuse’s per-project access control is paid for in self-hosted, as are audit logs. For an ENS file, that gets decided before building, not after.

Sharing Redis does not break through key collision, it breaks through memory policy. It is a failure that shows up under load and gets diagnosed badly.

Langfuse’s ingestion route is OTLP over HTTP only. A collector configured with the gRPC exporter points at somewhere that does not exist.

Closing

The five seams share a common pattern: they fail without warning. The token from another service gets in, the trace user comes from the client, the rate limit stops being global, the spend queue discards on reaching 64 MB and the Langfuse project does not create itself. None of those five things produces a visible error.

Day 2 work, then, consists above all of turning silences into signals. Two environment variables so that the audience is validated, three flags so that authorisation is real, an alert on service latency instead of on Redis availability, and an explicit contract on where the end user’s identity comes from.

And one decision worth taking early, before the budget is closed. The gateway’s JWT authentication and Langfuse’s per-project access control are both behind a licence. A sovereign platform can live perfectly well without either, with virtual keys and one project per tenant, but it is a different architecture and it has to be drawn as such from the start.

Sources

Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Autenticación y SSO en Langfuse, consultado el 12 de septiembre de 2026.
SCIM y API de organización, consultado el 12 de septiembre de 2026.
Integración OpenTelemetry de Langfuse, consultado el 12 de septiembre de 2026.
Postgres y caché en la documentación de self-hosting, consultados el 12 de septiembre de 2026.
Autenticación por token en LiteLLM y ajustes de configuración, consultados el 12 de septiembre de 2026.
GHSA-jjhc-v7c2-5hh6 y registro de CVE-2026-35030 en NVD.

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

Sat, 12 Sep 2026 06:00:00 +0200

Seventh article in the operational track of the control layer. The fourth dealt with the coexistence of human traffic and agentic traffic on a fixed fleet. This one deals with the prior question: how big the fleet has to be, and how big the gateway in front of it. Verified against LiteLLM 1.102.0 (commit of 11 September 2026) and vLLM 0.29.0, with load data published between May and September 2026.

TL;DR

Until this year, sizing for agents meant extrapolating a chat profile and adding margin. In 2026 the first production-scale characterisations were published and the profile turned out to be something else. These are the seven conclusions.

The median prompt of an agentic call is 68,000 tokens, of which 63,000 arrive cached. The measurement covers 13.5 million sessions, 3.2 million users, 760.5 million model calls and 44.9 trillion prompt tokens in one week of June 2026. The median input-to-output ratio exceeds 275 to 1. Any calculation done on a profile of 800 prompt tokens and 250 output tokens describes a different system.

The load is prefill-dominant when counting tokens and decode-dominant when counting time. The genuinely new tokens that have to be prefilled on each turn are only between 1.5 and 7.3 times the output, not 275. With the prefix cache working, the split of execution time runs from 91 % to 98.6 % in decode. You size by memory bandwidth and by KV cache capacity, not by prefill FLOPs.

The sizing risk is not the average, it is the collapse of the prefix cache. The median hit rate within a turn is 98 %, it falls to 55 % at the boundary between turns and to 8 % when the client changes model. With dynamically retrieved context, prefix reuse drops to 1 %. One miss turns 142,000 prefix tokens into real prefill. Session affinity per replica stops being an optimisation and becomes a capacity requirement.

The intra-turn concurrency measured in production is 1.15 at the median and 1.4 at the 90th percentile. 93 % of tool batches contain a single invocation. The image of the agent opening twenty calls in parallel does not appear in the data. The real multiplier comes from the duty cycle and from the fact that 87 % of calls are started by the agent, not by the person.

The gateway is sized per worker, and each worker is a universe. One vCPU and 4 GiB per worker, with requests equal to limits. The limit counters, the caches and the cooldowns live in the worker’s memory unless there is Redis, so a limit of 100 requests per minute with four workers and three replicas allows 1,200. The Prisma pool is also per process: the number of connections open against Postgres is workers times connection_limit.

The CPU cost that grows with agentic load is the token counter. It serialises the entire tool catalogue on every request and tokenises it. The project itself measured 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens, and the operation passes through a concurrency limiter of four per worker. With agents sending long catalogues and six-figure contexts, that limit is the ceiling before the CPU is.

Autoscaling by memory does not work on this piece and the startup probe takes five minutes. The official chart deliberately leaves memory without a target because Prisma’s query engine never returns the maximum reached. That leaves CPU at 60 %, with a startupProbe of 300 seconds that makes any new replica arrive late. For agents there are two better metrics, and they are in /metrics.

You are here: the arithmetic that comes before buying

Classic capacity planning starts from a chat SLO and crosses two budgets, the VRAM one and the time one. That method is still correct. What changes with agents are the inputs: the prompt length, the number of calls per unit of human work, the split between prefill and decode, and the very definition of an SLO. This article replaces those inputs with measured figures and drags the result through to two numbers: gateway replicas and GPUs.

Out of scope is the splitting of traffic once it is built, which was already covered by the coexistence of humans and agents and prefix routing.

The analogy: the repair shop and the front desk

An appliance repair shop sizes its staff by counting faults, not phone calls. The difference matters because the person who brings in the washing machine occupies the front desk for three minutes, while the technician who takes it apart goes back to consult the machine’s data sheet fourteen times over the afternoon. If the shop manager sizes by visits to the front desk, they end up short of technicians and overstaffed at reception.

The inference platform has the same asymmetry. The person sends one turn. The agent sends fifteen calls for that turn, and each one drags along the complete file of everything done so far. The front desk is the gateway and the technicians are the GPUs, and the usual mistake is to size both with the same counter.

There is a second detail the analogy captures well. The technician does not read the whole data sheet each time: they have it open on the bench. That is the prefix cache. The day somebody moves them to a different bench between consultations, they have to fetch it from the archive again, and the work multiplies without the number of faults having increased.

Part 1. The real load profile

For two years, any capacity calculation for agents was done on assumptions. In 2026 three pieces of work appeared that measure the matter, and it is worth reading them together because they measure different scaffolds and their figures are not averageable.

The three sources

The characterisation at scale was published by a research team working on GitHub Copilot production telemetry (arXiv 2608.00101, 30 July 2026): 13.5 million sessions, 3.2 million users, 95.1 million user turns, 760.5 million model calls and 44.9 trillion prompt tokens, in one week of June 2026. It is the largest sample published to date.

The characterisation by benchmark suites comes from an architecture team (arXiv 2605.26297, 25 May 2026), which instruments five of them (ADE-Bench, DABStep, GAIA, SWE-bench Pro and Terminal-Bench 2.0) and measures the internal split of time.

The production measurement on a different scaffold was published by the vLLM project itself together with AgentX (vLLM blog, 8 September 2026).

The three disagree on turns per session: 15 calls at Copilot’s median, 43 turns in AgentX, 62.4 in SWE-bench Pro with Qwen. It is not a contradiction, they are scaffolds with different loops. The sizing lesson is that the number of calls per unit of human work depends on the client that connects, and that is why you have to measure it at home before fixing the size.

The figures you can actually use

Quantity	Measured value	Source
Median prompt per call	68K tokens, 63K of them cached	2608.00101
Input:output ratio (median)	above 275:1	2608.00101
Median session	3 user turns, 15 calls, 4.2 min	2608.00101
Mean session	6.1 turns, 40.6 calls, 62.6 min	2608.00101
Calls per turn (median)	4.5	2608.00101
Calls started by the agent	87 %	2608.00101
Range across user profiles	50× (23K to 1.1M tokens per turn)	2608.00101
Median input context	142K tokens; output 444 tokens per turn	AgentX
Accumulated context in SWE-bench Pro	mean 68.7 to 80.1K, maximum 146 to 166K	2605.26297
Reasoning tokens over the output	29.0 % to 67.6 %	2605.26297

Two operational observations come out of that and appear in no sizing guide.

The first: agents that fail accumulate up to 1.8 times more average context than those that finish well (2605.26297). The cost does not correlate with success, it correlates with difficulty and with getting stuck. A sizing done on the task that goes well falls short.

The second: the range across user profiles is 50 to 1. A user who only converses spends 23,000 tokens per turn; one who leaves the agent in a long loop spends 1.1 million. Segmenting by usage profile pays off more than any difficulty-based routing, because the variance is in the people, not in the requests.

What to measure before calculating

Three of your own figures, taken from a week of real traffic, are worth more than any table in this article:

Model calls per hour of human work, by user profile.
Prompt length distribution (median and 95th percentile), not the mean.
Aggregate prefix cache hit rate, with the time series, not the single number.

All three come from what the gateway already stores. In LiteLLM_SpendLogs you have the tokens per request and the key and team identifiers; in proxy_server_request you have the tools block of each call; and in vLLM’s /metrics you have vllm:prefix_cache_queries and vllm:prefix_cache_hits.

Part 2. Prefill, decode and the cache trap

Here is the finding that corrects almost everyone’s intuition, including mine when I started this article.

Counting tokens, agentic load is overwhelmingly input: 275 to 1. The apparently obvious conclusion would be to size by prefill capacity, that is by FLOPs. It is false.

What has to be prefilled on each turn is not the whole prompt, but what was added since the previous turn. That addition, measured, sits between 1.5 and 7.3 times the output length (2605.26297). The rest is served by the prefix cache. And the real split of the model’s execution time comes out like this: between 91.0 % and 98.6 % in decode, between 1.4 % and 9.0 % in prefill (2605.26297).

The hit rate that sustains that split is high and has been measured in three places: theoretical from 87.9 % to 99.3 % and empirical from 84.6 % to 99.5 % (2605.26297); above 96 % in production (AgentX); and a median of 98 % within a turn in Copilot (2608.00101).

Where it falls over

The same Copilot work measures the degradation, and it is the most actionable figure in the article:

98 % median hit rate within a turn. The detail matters: the first call of a turn hits around 45 %, the second rises to 86 % and from the third onwards it stabilises between 92 % and 94 %.
55 % at the boundary between turns, with idle gaps of 4.1 minutes at the median.
8 % when the client changes model.

And there is a fourth, worse regime: with dynamically retrieved context, that is when the scaffold injects search or RAG results that change on every iteration, prefix reuse falls to 1 % or less (arXiv 2608.15127, 15 August 2026).

A cache miss with a context of 142,000 tokens turns a prefill of a few thousand tokens into a six-figure one. The work multiplies by a factor of between 20 and 60 without the number of requests having changed. That is the difference between a fleet that meets its targets and one that seizes up mid-afternoon.

Two sizing rules come from that:

Without session affinity towards the same replica, the calculated sizing is worthless. LiteLLM’s concrete mechanism is session_affinity, which is not active by default and which was already covered in the prefix routing article. Without it, the calculation has to be redone assuming a prefill regime, which is between 20 and 60 times more expensive.
Changing model halfway through a session costs the entire cache. Any difficulty-based routing policy that moves a live conversation from one model group to another pays an 8 % hit rate on the next call. If you want to route by difficulty, do it at the start of the session.

Co-batching, which is the other effect nobody measures

Mixing a short request into the same batch as long-context requests degrades the short one’s inter-token latency. The measured figure: one long request in the batch raises time per token by 38.3 %; two raise it by 79.7 % (2608.15127). It is the technical reason why separating human and agent pools improves the human experience more than the capacity split suggests, and it is also the reason not to raise --max-num-seqs above what the KV budget allows.

Part 3. The KV budget, with the numbers of the agentic case

The formula does not change with respect to classic capacity planning, but the inputs do, and the result is surprising:

$$\text{KV}_{\text{bytes/token}} = 2 \times n_{\text{layers}} \times n_{\text{kv heads}} \times d_{\text{head}} \times \text{bytes}_{\text{elem}}$$

For a 70B-class geometry (80 layers, 8 KV heads with grouped attention, head dimension 128, 2-byte elements) that gives 0.3125 MiB per token. With the median agentic context of 142,000 tokens:

$$142{,}000 \times 0.3125\ \text{MiB} \approx 43.3\ \text{GiB per sequence}$$

For a 30B-class geometry (48 layers, everything else the same) that gives 0.1875 MiB per token, that is around 26.0 GiB per sequence.

It is worth pausing on what that means. An 80 GB H100 with the weights of a 70B model in FP8 already taking up space does not have 43 GiB free for a single sequence. The conclusion is harsh and has to be written down: serving real 70B-class agentic context demands tensor parallelism of at least 2 and a quantised KV cache. And the number of simultaneous sequences that fit, which is what sets --max-num-seqs, comes from dividing the available KV by that figure, not from a chat heuristic.

$$\text{sequences} \approx \frac{\text{VRAM}_{\text{KV available}}}{\text{KV}_{\text{bytes/token}} \times \text{max context}}$$

With FP8 in the KV, the divisor halves. It is the cheapest capacity lever there is in this load profile, and its cost in quality is validated with evals, not assumed.

Part 4. From people to concurrent requests

This is where most calculations overshoot, because it is assumed that an agent equals many simultaneous requests. The data says otherwise.

The concurrency of calls within a turn, measured in production, is 1.15 at the median and 1.4 at the 90th percentile, and 93 % of tool batches contain a single invocation (2608.00101). Intra-turn parallelism is small. Subagents appear in 44 % of AgentX sessions, with a median of 4 deployments among those that use them, so there is a long tail, but it is not the usual regime.

What does multiply is the duty cycle. The mean session lasts 62.6 minutes and contains 40.6 calls. At 444 output tokens and an interactivity of around 60 tokens per second, each call occupies around 7.4 seconds of decode, which gives between 5 and 7 active minutes out of those 62.6.

$$C = D \times \rho \times A$$

where $D$ is the people working, $\rho$ the duty cycle and $A$ the amplification. With $\rho \approx 0.10$ to $0.12$ derived from the previous figures, twenty developers give between 2 and 2.4 concurrent sequences in the steady state. It is the peak, not the average, that has to be sized for.

The amplification

Three factors published separately:

Tool failures: 9 % of turns trigger them, with a compute amplification of 4 times (2608.00101).
Subagents: 44 % of sessions, median of 4 deployments (AgentX).
Internal variance: the same task from the same agent varies by up to 30 times in cost within a single trace (2608.15127).

The product of the first two gives $(1 + 0.09 \times 3) \times (1 + 0.44 \times 3) \approx 2.9$. Nobody publishes that product as a single figure, so it is flagged as my own derivation, but an amplification factor of 3 is the reasonable design provision and it fits the measured variance.

For the twenty developers in the example: between 6 and 7 concurrent design sequences, with 142K contexts. At 43.3 GiB of KV per sequence in the 70B class, that is around 300 GiB of KV cache, that is four H100s dedicated to KV alone with the weights on top. With KV in FP8, half. With a 30B-class model, a third.

That calculation, done with the chat profile of 800 prompt tokens, would have given a single GPU. That is the distance between the two regimes.

Part 5. The gateway, which also has to be sized

The proxy consumes no GPU, but it has its own arithmetic and it is the part that produces the most surprises in production, because almost all of its state is per process.

A worker is a universe

The default value of --num_workers is 1 (litellm/constants.py:57). The usual startup is uvicorn directly with uvloop (proxy_cli.py:1477), and there are alternatives with gunicorn, hypercorn and granian.

What lives in each worker’s memory, and therefore multiplies:

The in-memory cache, limited to 200 elements with a TTL of 600 seconds (litellm/caching/in_memory_cache.py:27). With many deployments, the cooldowns and the counters compete for that same space and get evicted before they expire.
The key authentication cache, with a TTL of 60 seconds (proxy_server.py:1650), adjustable with general_settings.user_api_key_cache_ttl.
The limit and budget counters when there is no Redis (parallel_request_limiter_v3.py, local_only=True paths).
The router cooldowns, with DEFAULT_COOLDOWN_TIME_SECONDS = 5 (constants.py:75).
A periodic task scheduler of its own per worker (proxy_server.py:9674), coordinated between pods with a lock in Redis.

The practical consequence: a limit of 100 requests per minute with four workers and three replicas allows up to 1,200 without Redis. With Redis, the v3 limiter registers seven Lua scripts at startup (parallel_request_limiter_v3.py:608) and the limits become global.

One detail of the v3 limiter that matters with agentic load: LITELLM_TPM_TOKEN_RESERVATION_ENABLED comes enabled by default and reserves estimated tokens before the call, which adds a round trip to Redis per request. And there is an asyncio.Lock per instance that serialises the atomic batch checks (:665), documented by the code itself as a 99th-percentile risk under contention.

If Redis goes down, requests do not fail: there is a circuit breaker with a threshold of 5 failures and a recovery of 60 seconds (constants.py:456), and the proxy degrades to local memory. What is lost silently is the globality of the limits.

Postgres

The pool is also per process. The default values are database_connection_pool_limit = 10 and database_connection_pool_timeout = 60 (proxy_cli.py:62), injected as parameters of the Prisma URL. A pod with four workers opens forty connections. Three pods, one hundred and twenty. It is the fast route to exhausting Postgres’s max_connections without having served any traffic, and the reason the official chart offers PgBouncer inside the pod itself with maxDbConnections: 20 and maxClientConn: 1000.

On the request path there is only the authentication, which is cached for 60 seconds, and the rate limiting. Spend is deferred. The SpendLogs queue is capped at 64 MB and on overflow it discards the oldest elements with an error in the log (utils.py:6704). The write interval is proxy_batch_write_at, with a default value of 10 seconds plus a random spread of up to 5 (constants.py:1719, proxy_server.py:9704); the production guide recommends raising it to 60.

With a slow Postgres, the queue grows to those 64 MB and then loses billing data without returning any error to the client. It is the silent failure mode you have to alert on.

The token counter, which is the agentic trap

This is the piece that turns a correct sizing into an insufficient one, and it appears in no guide.

Before admitting a request, the proxy counts tokens. With chat load the cost is negligible. With agentic load it is not, for two reasons that add up.

The first is the prompt length. The project itself measured the admission counting cost at 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens (benchmarks, consulted on 12 September 2026), and published that with a counter written in Rust those figures fall to 4.9, 6.8 and 10.2 ms.

The second is the tool catalogue. The function that prepares the definitions serialises the entire catalogue to a pseudo-TypeScript and tokenises it (litellm_core_utils/token_counter.py:934), plus nine fixed tokens. The cost is linear in the size of the catalogue and it is paid on every request. That is exactly the pattern of an agent connected to an MCP gateway: a long catalogue, resent on every turn. The token bill of that catalogue was already covered in the MCP gateway article; here its other face appears, which is proxy CPU.

And on top of that there is an explicit ceiling. The counting runs in a thread with a capacity limiter whose default value is 4 per worker (token_counter.py:340, constants.py:407). With 100K prompts, four concurrent counts of 100 ms mean a theoretical ceiling of around 40 admissions per second per worker before the limiter starts queueing, and that is without having called the model yet.

What is cached is the tokeniser, with an LRU of 64 entries (utils.py:2206), not the result of the count.

Three levers, in order of return: raise TOKEN_COUNTER_MAX_CONCURRENT_COUNTS if CPU is spare, reduce the tool catalogue exposed per key, and watch the gateway overhead figure in /metrics to know whether this is happening.

Streaming and admission

There is no connection limit by default. --limit_concurrency exists but is optional, and it returns 503. The built-in admission control is only activated if general_settings.max_in_flight_requests_per_worker is defined (middleware/admission_control_middleware.py:285), with max_queued_requests_per_worker and admission_queue_timeout_seconds at 1.0 by default, rejecting with 503 and a retry-after: 1 header.

For agents, that header matters: an agentic client that retries without respecting it multiplies the load exactly when the gateway is saying it cannot take any more. The combination of gateway retries was already calculated by the fourth article and it reached 45 calls per turn in the worst case.

The default request timeout is 6,000 seconds (constants.py:512). The production guide recommends 600. With long agentic tasks, leaving it at 6,000 means a hung connection occupies a slot for an hour and a half.

Resources and autoscaling

The official chart recommends 1 vCPU and 4 GiB per worker, with requests equal to limits (helm/litellm-helm/values.yaml:235), and the production documentation suggests one worker per pod scaling horizontally. The project’s own high-performance benchmark, by contrast, uses four workers per pod with PgBouncer in front.

The HPA comes with targetCPUUtilizationPercentage: 60. And there is a comment in the chart worth reading twice: memory is deliberately left without a target because the resident consumption of Prisma’s query engine is a high-water mark that is never returned, so that an HPA on memory would add replicas and never remove them (values.yaml:260).

The other problem is startup. The startupProbe has failureThreshold: 30 with a period of 10 seconds, that is up to 300 seconds before the pod is considered ready (values.yaml:109). An HPA that reacts at 60 % CPU and takes five minutes to add capacity arrives late to any agentic burst.

The alternative, optional in the chart, is to scale by Prometheus metrics: targetRequestsPerSecond and targetTokensPerSecond over rate(litellm_proxy_total_requests_metric_total[1m]) and rate(litellm_total_tokens_metric_total[1m]). The chart itself warns that the token metric lags with long streaming responses, because tokens are accounted for on completion. For agents, that is almost always.

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

Metric	What for
`litellm_in_flight_requests`	Real saturation per worker. It is the scaling metric
`litellm_overhead_latency_metric`	Proxy cost per request; it rises with token counting
`litellm_overhead_with_guardrails_latency_metric`	The same including guardrails
`litellm_request_queue_time_seconds`	Arrival through to pre-call, includes authentication and ASGI queueing
`litellm_deployment_cooled_down`	Replicas out of the pool
`litellm_service_latency`	Redis and database latency, per service

The /health/backlog endpoint returns in_flight_requests, admitted_requests, queued_requests and rejected_requests (health_endpoints/_health_endpoints.py:1871).

What it does not expose: the depth of the SpendLogs queue or its bytes, the Prisma connections in use against the limit, nor the depth of the logging worker’s queue. All three are real failure modes and have to be instrumented from outside.

At the engine

Names verified in the vLLM 0.29.0 metrics documentation:

vllm:num_requests_running, vllm:num_requests_waiting
vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:inter_token_latency_seconds
vllm:prefix_cache_queries, vllm:prefix_cache_hits, vllm:kv_cache_usage_perc

And the ones that break dashboards on upgrade:

vllm:time_in_queue_requests is duplicated by vllm:request_queue_time_seconds.
The direct prefix cache hit rate indicator disappeared. It is now computed as rate(vllm:prefix_cache_hits) / rate(vllm:prefix_cache_queries).
vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are deprecated along with CPU swapping.
vllm:gpu_cache_usage_perc became vllm:kv_cache_usage_perc, which also breaks any HPA that uses it.

The SLO worth chasing

A 95th percentile of time to first token describes nobody’s experience when the unit of work is 43 turns. Coding tasks frequently exceed ten minutes and research tasks reach hours (2608.15127).

Two indicators serve better:

90th percentile of full-task latency, measured per session, not per request. It comes from grouping LiteLLM_SpendLogs by session identifier.
90th percentile of interactivity, in tokens per second per user. It is the one AgentX uses, with values between 58.3 and 74.2 in its runs.

To autoscale the engine, the official production-stack example uses vllm:num_requests_waiting with a threshold of 5, polling every 15 seconds and a cooldown of 360. With agentic load it is worth adding vllm:kv_cache_usage_perc as a second trigger, because with 26 to 43 GiB of KV per sequence the memory pressure arrives before the queue does.

Part 7. The engine levers, ordered by return

Verified against vLLM 0.29.0, released on 9 September 2026.

--long-prefill-token-threshold 512. The AgentX measurement attributes to it +93 % in tokens per GPU per second, with a 2.3-times improvement at the 90th percentile, by attacking the head-of-line blocking that a long prefill produces in front of short requests. It is the highest-return lever documented for this profile.

KV cache in FP8. It halves the memory budget per sequence, which is the limiting resource. Validate with evals.

--max-num-queued-reqs and --max-num-queued-tokens. New in 0.29. They are admission control in the engine, and they are the right mechanism to protect the SLO instead of letting the queue grow without limit.

max_num_batched_tokens downwards. High values improve time to first token; low values improve inter-token latency. In a regime where 91 % of the time is decode, inter-token latency is what is perceived. Chunked prefill is enabled by default whenever possible in the V1 engine.

Session affinity at the gateway. It is not an engine setting, but it is what holds up everything above. Without it, that 98 % hit rate turns into random distribution across replicas.

Two version warnings: Model Runner V2 becomes the default in 0.29 and V1 is deprecated with removal planned for 0.32; and starting with python -m vllm.entrypoints.openai.api_server is deprecated.

The full calculation, from start to finish

Case: twenty developers, a coding scaffold, a 30B-class model in the fleet, maximum context of 128K, KV in FP8.

1. Steady-state concurrency. $C = 20 \times 0.11 \times 3 \approx 6.6$ sequences.

2. KV budget per sequence. 0.1875 MiB per token in BF16, half in FP8, that is around 0.094 MiB. At 128K of context: around 12.0 GiB per sequence. For 6.6 sequences with margin, rounding to 8: 96 GiB of KV.

3. GPUs. The weights of a 30B in FP8 take up around 30 GB. On an 80 GB H100 with gpu_memory_utilization of 0.90 there are around 72 GB usable per card. With tensor parallelism of 2, the weights are split: around 15 GB per GPU, so around 57 GB of KV per GPU and around 114 GB in the replica. That covers the 96 GiB calculated, and TP=2 also adds memory bandwidth, which is the resource that dominates decode. Two H100s, one replica, TP=2. It is worth seeing the contrast: on a single H100 the full weights leave around 42 GB of KV, that is three and a half sequences, less than half of what is needed. With a second replica for availability and maintenance windows: four GPUs.

4. Requests per second at the gateway. 20 people, a mean session of 40.6 calls in 62.6 minutes, duty cycle included: of the order of 0.2 to 0.3 requests per second on average, with peaks of 2 to 3.

5. Gateway workers. The ceiling is not set by throughput but by token counting: four concurrent counts per worker, at 100 ms with long prompts. With peaks of 3 requests per second and six-figure prompts, two replicas of one worker each cover the case with margin for the rolling update, at 1 vCPU and 4 GiB each. Postgres will see 20 connections.

6. What needs switching on. Session affinity, admission control at the gateway and at the engine, --long-prefill-token-threshold, and Redis so that the limits are global.

The same calculation with the chat profile would have given one GPU and one worker. The distance is not in the safety margin, it is in the inputs.

A reference `config.yaml`

model_list:
 - model_name: agentes-30b
 litellm_params:
 model: hosted_vllm/qwen3-coder-30b
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 rpm: 600
 model_info:
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072

router_settings:
 routing_strategy: simple-shuffle
 # without this, the capacity calculation in this article is worthless
 optional_pre_call_checks: ["session_affinity"]
 num_retries: 2
 allowed_fails: 3
 cooldown_time: 30

general_settings:
 # admission control: below this the queue grows without limit
 max_in_flight_requests_per_worker: 24
 max_queued_requests_per_worker: 48
 admission_queue_timeout_seconds: 1.0
 # the spend queue discards at 64 MB; shorter interval, less backlog
 proxy_batch_write_at: 60
 enable_drain_endpoint: true
 drain_endpoint_token: os.environ/DRAIN_TOKEN
 database_connection_pool_limit: 10

litellm_settings:
 # 6000 s by default: a hung connection holds a slot for an hour and a half
 request_timeout: 600
 callbacks: ["langfuse_otel", "prometheus"]
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

Environment variables that go with it:

NUM_WORKERS=1
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS=8 # default 4; raise only if CPU is spare
LITELLM_TPM_TOKEN_RESERVATION_ENABLED=true
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS=30

And on the engine side:

vllm serve Qwen/Qwen3-Coder-30B \
 --tensor-parallel-size 2 \
 --max-model-len 131072 \
 --kv-cache-dtype fp8 \
 --long-prefill-token-threshold 512 \
 --max-num-queued-reqs 64 \
 --gpu-memory-utilization 0.90

Checklist

Measure three figures of your own over a week: calls per hour of human work and per profile, prompt length distribution, and the time series of prefix cache hit rate.
Calculate the KV per sequence with the model’s real geometry, taken from its config.json, not from a table.
Set --max-num-seqs from that budget, not from a chat heuristic.
Switch on session affinity before accepting the calculated number of GPUs.
Switch on admission control in both layers and check that the client respects retry-after.
Lower request_timeout from 6,000 to 600.
Raise proxy_batch_write_at to 60 and alert on the growth of the spend queue from outside, because there is no metric.
Leave the HPA on CPU at 60 %, never on memory, and assume the 300 seconds of the startupProbe when planning the margin.
Watch litellm_in_flight_requests and litellm_overhead_latency_metric; if the overhead rises with the size of the tool catalogue, the culprit is the token counter.
Go over the dashboards for the vLLM metric renames before upgrading.
Check that no routing policy moves a live session between model groups.
Redo the calculation when a new user profile arrives: the range across profiles is 50 to 1.

Traps and things that are not what they seem

The ratio of 275 to 1 does not mean the system is prefill-bound. The new tokens per turn are between 1.5 and 7.3 times the output. The time is between 91 % and 98.6 % in decode.

The aggregate prefix cache hit rate hides the problem. An aggregate 90 % may be 98 % intra-turn and 55 % at boundaries. You have to look at the series, not the number, and segment by request type.

vLLM’s hit rate counts the reprefill after an eviction as a hit. The counter exists internally and is not exposed, so the published number is optimistic under a memory-pressure regime.

Agents that fail cost more than those that succeed, up to 1.8 times more context. Sizing on the task that goes well leaves the calculation short.

The database pool is per process, not per pod. Four workers are forty connections.

The HPA on memory does not work with this piece, and not because of a configuration mistake: Prisma’s query engine does not return the memory.

The 300-second startupProbe turns any autoscaling into late reaction. For agentic peaks, it is better to over-provision than to trust the scaling.

Raising --max-num-seqs to “make use of the GPU” degrades perceived latency: two long requests in the batch raise time per token by 79.7 %.

The token counter has a limiter of four per worker and it appears in no sizing guide. With large tool catalogues and long contexts it is the first ceiling you hit.

The figures in this article come from three different scaffolds and are not averageable. Fifteen, forty-three and sixty-two turns per session are three systems, not three measurements of the same one.

Closing

The arithmetic of sizing has not changed. What has changed are the inputs, and now there are public measurements at scale instead of intuitions. Three figures sum up the change of regime: 68,000 tokens of median prompt, 87 % of calls started by the agent rather than by the person, and a time split that is decode for more than 90 % as long as the prefix cache hits.

From that comes an order of work that is not the usual one. First, session affinity, because without it the rest of the calculation describes a different system. Second, the KV budget with the model’s real geometry, which is what sets how many sequences fit. Third, the gateway, where the ceiling is not throughput but a token counter with four threads per worker.

And a warning about your own figures. The measured range across user profiles is 50 to 1. Any platform that does not segment its measurement by profile is averaging two different populations and producing a number that describes neither of them.

Sources

Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale, arXiv 2608.00101, 30 de julio de 2026.
Agentic AI Workload Characteristics, arXiv 2605.26297, 25 de mayo de 2026.
From LLM Inference to Agentic Workloads, arXiv 2608.15127, 15 de agosto de 2026.
vLLM x AgentX: serving agentic workloads, 8 de septiembre de 2026.
vLLM metrics design y optimization guide, consultados el 12 de septiembre de 2026.
LiteLLM benchmarks y production setup, consultados el 12 de septiembre de 2026.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Autoescalado con KEDA en vLLM production-stack, consultado el 12 de septiembre de 2026.

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

Fri, 11 Sep 2026 01:45:00 +0200

Sixth post in the operational track on the control layer. The previous five treat the gateway as the piece the tokens pass through: the pair with Langfuse, day 2, virtual keys, humans and agents and prefix routing. This one is about the other door. Verified against LiteLLM 1.102.0, commit of 10 September 2026.

TL;DR

The MCP gateway is not a module, it is a complete second surface. A native MCP endpoint with four route spellings, a proxy mode with three fixed tools, four REST routes, some thirty-five administration routes and a long set of OAuth discovery endpoints. It still lives under _experimental and it is nearly 20,000 lines.

I correct what I said in the previous post: access is NOT open by default. In 1.102.0, a key with no declared permissions and no team resolves zero servers. What does open things up is a per-server flag, allow_all_keys, and that flag overrides the key’s explicit scope unless you enable a setting that is off by default.

Permissions are enforced on execution, not just on the listing. Calling a tool that did not appear in tools/list returns 403. There are three chained checks before the call, including one on the allowed arguments. The same predicate governs listing and calling, by declared design.

The hierarchy is built out of intersections, with two escapes and a fail-open. The key’s access groups are additive on top of the key and team ceiling. The organisation ceiling replaces rather than intersects when there is nothing below it. And on an indeterminate error while resolving tool permissions, the code returns “unrestricted”. It is a deliberate decision and it has to be declared in the risk analysis.

Per-tool cost is worth zero and that is why its metric disappears. There are exactly two Prometheus metrics for MCP, and the spend one is only incremented if the cost is greater than zero. With the default configuration, that series never exists. There is no listing metric, no latency metric, no error metric, no server health metric.

The cost that does matter is not attributed to MCP anywhere. It is the tokens of the tool definitions, resent on every turn inside the prompt_tokens of the model call. Five typical MCP servers add up to 58 tools and some 55,000 tokens. The tool’s spend row, with its zero cost, has nothing to do with that.

And the gateway’s semantic filter does not act on the MCP door. It only applies when it is LiteLLM calling the model. A client connected to /mcp receives the whole catalogue, filter enabled or not. And even if it did act, filtering per request rewrites the cached prefix of the prompt, which is the opposite of what Anthropic and OpenAI do with their own tool searches.

The good observability ships switched off. Without LITELLM_OTEL_V2, a tool call comes out as a generic span with the model set to MCP: name. With it enabled there is a dedicated span and correct attributes, and the arguments and the result stay hidden unless you opt in. And in neither case is trace context propagated towards the upstream MCP server: the trace is cut at the gateway.

Tool arguments are written, and they are written in the clear. They go in full into the metadata column of the spend table, without passing through the switch that governs prompt storage nor through the redaction function, which only knows about messages and response. Whoever enabled redaction for GDPR probably believes they are covered.

The 1.83 branch concentrated four serious vulnerabilities, and one of them belongs to the MCP gateway itself. The endpoint for testing a connection before saving accepted the stdio transport’s command, args and environment in the request body. A low-privilege key got command execution on the proxy host.

And the specification moved underneath. The revision in force since July 2026 removed protocol sessions, the initialisation handshake and the session header. LiteLLM 1.102.0 advertises the June 2025 revision and its version enumeration does not even contemplate the two later ones.

You are here: the door nobody chose

The gateway was chosen for the reasons in the switchboard post: licence, fit inside the cluster, maturity. What was evaluated was the inference traffic.

Later, somebody connects an agent, the agent needs tools, and the same process that is already authenticated and deployed turns out to speak MCP too. The decision to turn it into the tool door is almost never taken: it is inherited. And that door has neither the same controls, nor the same observability, nor the same cost model.

When MCP grows covered the identity of your own MCP servers. MCP from the inside covered the protocol and its instrumentation. This post covers the piece in between.

The analogy: the switchboard that also hands out keys

An office building’s telephone switchboard. Its original job is to route calls: it decides who can call outside, how much each department spends, and it leaves a record of every conference call.

Over time, the switchboard operator is also given the key cabinet. They are already at reception, they already know who everybody is, it looks like the natural place. And from that moment there are two services at the same desk with very different rules.

Calls are priced by the minute and show up on the bill. Keys have no tariff, so on the department’s bill they come out at zero, and somebody might conclude that the key cabinet costs nothing. It does cost: it costs the operator’s time, it costs having to read the whole key catalogue every time somebody asks what is available, and it costs that nobody checks that the label on a key still says what it said the day its loan was authorised.

The analogy ends at the interesting spot. When somebody takes a key, the record says which key it was and what they said they wanted it for. That record is simultaneously the evidence of the action and a piece of data that perhaps should not be stored in the clear.

The real routes

The first thing that surprises you when you look at the code is how many spellings get you to the same place.

The native endpoint

The MCP sub-application is mounted at /mcp and inside it has four mounts: the root as a wildcard, /mcp, /{name}/mcp and /sse.

And there is also a hand-declared route for bare /mcp on the main application, with a comment that explains why: the mount cannot match its own bare prefix, and the resulting 307 redirect breaks MCP clients sitting behind a proxy that terminates TLS. It is the kind of detail you discover in production on a Friday.

/mcp/sse holds a surprise. It does not use the classic SSE transport: it uses the same HTTP session manager in stateless mode. The SseServerTransport object is constructed and is not referenced on any other line of the file. It is dead code, and the /mcp/sse/messages endpoint that this object advertises does not exist as a working route.

The /{name}/mcp route resolves in a documented order: server alias, comma-separated list with a maximum of 16 elements, tool set, access group. If nothing matches, 404.

And /mcp/proxy exposes a fixed surface of three tools. It is the progressive disclosure pattern, and it will come up again further down because it is the most important mitigation in this post.

The REST routes

Four, under the /mcp-rest prefix:

Method	Route	Authentication
GET	`/mcp-rest/tools/list`	Virtual key
POST	`/mcp-rest/tools/call`	Virtual key
POST	`/mcp-rest/test/connection`	Key plus administrator role
POST	`/mcp-rest/test/tools/list`	Key plus administrator role

The variants without -rest that appear in the internal types have no router registering them. The /mcp/* wildcard absorbs them as JSON-RPC.

The last two are the stars of the security section.

Administration and discovery

Some thirty-five routes under /v1/mcp, almost all of them with virtual key authentication. Three details worth knowing:

GET /v1/mcp/registry.json carries no authentication dependency. It returns 404 if the public registry is not enabled, which is the real protection.

POST /v1/mcp/server requires the proxy administrator role. POST /v1/mcp/server/register does not: it is the proposal path, which requires a key with a team, explicitly forbids the stdio transport and leaves the entry pending review.

And on top of that there is a long set of OAuth .well-known endpoints, with dynamic client registration, authorisation, token, revocation and introspection, plus their per-server variants.

Configuration and transport

The mcp_servers schema in config.yaml has fifty-odd fields, all of them optional. A warning for anyone coming from earlier versions: spec_version no longer exists in 1.102.0.

The default transport when omitted is http, that is, streamable HTTP. The other two values are sse and stdio. The string streamable_http is not valid in the configuration file, although it is accepted as a synonym when importing connector files.

Tool prefixing uses the format {server}{separator}{tool} with - as the default separator, configurable through an environment variable read at import time. The prefix chosen is the alias, or the name, or the identifier, in that order. There is an optional short mode that derives three characters from the hash of the server identifier and resolves collisions by rehashing.

Access control, and the correction to the previous post

In the humans and agents post I closed the list of traps with this one: “access to MCP servers is open by default if no level defines a list”. Verified in 1.102.0, that is false.

The function that resolves the allowed servers does the opposite. If the team declares nothing, it inherits from the key; if the key declares nothing either, the base set is empty and the scope is marked as unbounded with zero servers. There is no branch that expands to “the whole registry” other than the administration role one.

What does open things up, and this is the important part, is something else.

`allow_all_keys`

It is a per-server flag, in configuration, in the API and in the database, defaulting to false. When set to true, that server joins the set of open servers and any key can see it.

The detail to know: by default it overrides the key’s explicit scope. A key that declares access only to server A will also see server B if B has the flag. The key’s scope is only respected if you enable general_settings.mcp_allow_all_keys_respects_mcp_scope, which is off by default.

There is a second opening case: anonymous subjects, with no user identifier, role or key, additionally receive the servers configured with authentication delegation towards the upstream and the pass-through ones. It is consistent with the design of those two modes, and it is a surface you have to know about.

`require_key_mcp_access_defined`

It lives in general_settings, defaults to false, and needs no licence. What it inverts is only the empty-key inheritance towards the team: with the flag on, a key with no list of its own gets an empty set instead of inheriting the team’s.

It does not affect keys that do declare servers, which still intersect with the team. And it does not close off the access group grants, which are additive and are the escape route you have to audit separately.

There is a sibling, require_end_user_mcp_access_defined, with the same shape and applied to the end user.

The hierarchy

The function’s docstring is normative and says that all the rules are intersections. In detail:

Key and team: intersection when both declare. Inheritance in one direction when one is empty.
Key access groups: additive union on top of the previous result.
End user: intersection if it declares anything.
Agent: intersection.
Internal user: ceiling that only narrows.
Organisation: ceiling that intersects if there are restrictions below, and that replaces by becoming the ceiling when there are none.

For subjects with no key, admitted by SSO or by the gateway, the model changes to a per-source union instead of an intersection.

The fail-open

This is the point you have to take to the risk analysis and not leave in a technical note.

On an indeterminate failure while resolving the tool permissions of a key or a JWT, the code returns “unrestricted”, that is, all tools allowed. The organisation ceiling is skipped in the same situation. For subjects with no key the behaviour is the opposite, fail-closed.

There is a well-resolved intermediate case: when a permission is named but cannot be read, a specific error is raised and it is denied in both cases.

The doctrine is documented in the class docstring, which is appreciated. And it remains a behaviour that, in a system under the ENS, you have to declare in writing instead of discovering it during an audit.

What is done well

Permissions are enforced on the call, not just on the listing. The execution path chains three checks before contacting the server: allowed or forbidden tools, per key and team tool permission, and validation of the arguments against the allowed ones. All three return 403.

Listing and calling share the same predicate, and the docstring declares it as an explicit invariant. A client that calls a tool directly when it did not appear in its listing receives 403, it does not execute. The same over REST and over the responses API route.

The x-mcp-servers and x-mcp-access-groups headers only narrow, and they fail closed if they do not resolve.

Header forwarding towards the upstream is an explicit per-server allowlist, with a centralised decision about when to strip the caller’s Authorization, and a protection against cross-forwarding: in a multi-server listing, the global header is withheld if more than one server would consume it. That is RFC 9700 applied.

The bill that never shows up

Here is, in my opinion, the most important thing in this post, and it has nothing to do with security.

Every tool the gateway aggregates takes up space in the model’s context. Name, description and parameter schema travel in the request’s tools block, on every turn, and they are paid for as input tokens of the real model.

The figures published by Anthropic in November 2025 are the best public reference there is. Five connected MCP servers add up to 58 tools and some 55,000 tokens. The breakdown: GitHub, 35 tools and some 26,000 tokens. Slack, 11 tools and some 21,000. Sentry, five and some 3,000. Grafana, five and some 3,000. Splunk, two and some 2,000.

With tool search enabled, that set drops from some 72,000 to some 8,700 tokens, and the usable context goes from 122,800 to 191,300.

The second cost of a large catalogue is not economic and is not billed: it is that the model chooses worse. It has its own section further down, because of the two it is the one that decides whether the agent is any good.

How to measure it with what is already there

The gateway stores enough data, even if it does not cross-reference it for you.

In the listing’s spend row, with call type list_mcp_tools, it enriches the metadata with allowed_server_count, tool_count_total, per_server_tool_counts and per_server_list_outcomes.

In the model call rows, the proxy_server_request column keeps the request body with the tools block inside it.

The practical method: pair the listing rows with the LLM rows of the same session by session_id, read the total number of tools from the first, and pass the tools array of the second through litellm.token_counter(model=..., tools=...), which accepts that argument. Multiply by the number of turns, because the block is resent on all of them.

For the relative cost, a quicker comparison: mean prompt_tokens of keys with MCP enabled against those without, same model and same period.

Tool selection

The previous section treats the catalogue as a bill. This one treats it as what determines whether the agent gets it right. This is ground where the talk runs well ahead of the measurement, so it is worth separating what has been measured from what gets repeated.

How many tools a model can take

Anthropic’s official figure is that Claude’s ability to pick the right tool degrades when going from 30 to 50 available tools. The same documentation gives the inverse criterion, which is the more useful one: the full catalogue with nothing else is the right option with fewer than ten tools, when all of them are used on every request, or when the definitions add up to fewer than a hundred tokens.

There is also an administrative ceiling before the semantic one. VS Code with Copilot cuts off at 128 tools per request and returns an error when you exceed it.

The full curve gets cited far more than it has been measured. The soundest primary data I found is from May 2025: varying the number of candidates from 1 to 11,100 over a registry of more than 4,400 servers, below some 30 tools the success rate is above 90 %, between 31 and 70 intermittent failures appear, and past the hundred mark the degradation becomes dominated by retrieval. It is a qualitative heat map, not a numeric table.

The figure that circulates most (78 % with ten tools, 40 % with a hundred, 13.6 % with seven hundred) comes from a conference talk with no paper, no repository and no published methodology. Do not take it to a client presentation.

And there is a warning that affects everything else. An audit of 496 tasks from four tool calling benchmarks found 18.5 % misalignment between the label and reality. In the BFCL v4 subset, 80 % of the failures assigned to the agent came from fragile state comparisons. In one of the benchmarks with an automatic judge, 23 identical re-runs gave a range from 57.9 % to 76.8 %, almost 19 points of spread, enough to reorder the ranking. Any absolute figure in this section deserves a generous margin of error.

How it fails

The work with the cleanest taxonomy evaluates 36 servers, 220 tools and 1,000 tasks against twenty frontier models, with an average of 15.2 tools exposed per task of which 4.1 are relevant.

Failure mode	Share
Not using a tool when one was needed	10.5 %
Choosing the wrong tool	9.0 %
Malformed parameters	6.9 %
Not recovering from an error	10.3 %
Cognitive failures, not tool failures	63.3 %

The interesting part is in the per-model breakdown: the dominant failure mode changes with the model, not with the size of the catalogue. In one of them, 40.1 % of the failures are simply not calling any tool. In others the malformed parameter dominates. In another, the wrong tool. There is no canonical failure to attack.

On tools that do the same thing, which is the real case of a gateway with fifteen servers, there is a good measurement. A study with ten groups of five functionally equivalent tools and a thousand query pairs puts the selection bias between 0.3 and 0.4, that is, you would have to redistribute between 30 and 40 % of the probability mass for equivalent tools to be chosen equally. The same work shows that a biased continued training takes the selection of a specific endpoint from 0.6 % to 12.8 %.

That study contributes another figure that reorders priorities: shuffling a tool’s description moves the selection distribution substantially, whereas changing only the name has minimal and inconsistent effects. Applied to the gateway’s server-tool prefixing: it costs tokens and probably does not change which tool the model goes to. There is no study isolating the effect of name spacing, and it is a real gap in the literature.

If the description is what carries weight, the state of the estate is the problem. An audit of 856 tools from 103 MCP servers found undeclared limitations in 89.8 %, missing usage guidance in 89.3 % and opaque parameters in 84.3 %. Only 2.9 % were free of problems across all their components.

The optimum is not a number

The May 2026 work I cited above proposes a chance-corrected metric and uses it to size the list adaptively. Over 370 functions it reaches 90.3 % coverage while seeing an average of 7.4 tools, against 90.8 % seeing a fixed 50.

But the same paper publishes its own counterexample, and it is honest to cite it: over another benchmark of 3,251 tools, the fixed K of 5 wins in aggregate, 64.7 % against 61.9 %. The adaptive advantage concentrates in the hard queries, where the right tool sits between sixth and twentieth by similarity: there it finds 16.7 % of the cases where the fixed K finds 0 %.

The March 2026 work over 121 tools puts the plateau at K=3, with 97.1 % that does not improve going up to K=5. With the caveat that it measures retrieval against labels, not model accuracy.

And a July 2026 work gives the formulation that seems to me the right one: ranking by score is inconsistent with optimal acquisition when costs are heterogeneous, so the optimal K is a function of cost, not a constant. Trimming from 7 to 4.4 exposed tools while keeping success is possible; fixing a universal number is not.

Filtering can make the result worse

This is the part that appears in no product blog. A June 2026 benchmark with three registry sizes (25, 100 and 250 tools), six filtering methods, seven models and 26,460 runs:

Strategy	Success	Visible tools	Tokens
Expose the whole catalogue	32.1 %	125.00	56,062
Keyword, top-5	22.1 %	4.80	3,200
Keyword, top-10	22.4 %	9.54	5,356
State-aware	24.0 %	25.95	13,522
Full causal path	24.0 %	26.24	13,697

Four of the five filtering strategies perform worse than not filtering. The fifth, which does win comfortably, exposes 0.99 tools per step and requires hand-written precondition and effect contracts for every tool, plus a search of the causal path to the goal state. It is a planner with the goal given, not a retrieval filter, and it is not comparable with anything a gateway offers.

There is more evidence in the same direction. A work over a corpus of 43,000 tools measured that the agent’s completion rate drops when using retrieved sets against hand-annotated sets: the filter introduces its own recall ceiling. And top-K breaks the tasks that need several tools: on a benchmark of 16,464 endpoints, a fine-tuned retriever reaches a Recall@3 of 68.6 % but only 39.7 % of the queries have their complete tool set within the first three. Almost 29 points of gap between “the one I needed is there” and “all the ones I needed are there”.

Filtering is not free on security either. A March 2026 work attacks the retrieval layer by injecting adversarial tools that saturate the top-k: with injection rates of 1.2 % it manages to dominate the top-k between 91 % and 97 % of the time. It is an attack surface that exposing the whole catalogue does not have.

The most elegant counterpoint comes from Anthropic’s own numbers. With the same catalogue of more than fifty tools, the gain from tool search drops from 25 points on one model to 8.6 on the next, because the unfiltered baseline rises from 49 % to 79.5 %. The margin left by filtering shrank to a third in one generation of models. Any architecture decision taken here has a short shelf life.

The prompt cache trap

Here is what makes a homemade top-K filter a bad idea even if it worked.

Tool definitions do not go just anywhere in the prompt: they go at the beginning. Anthropic documents the order of the cacheable prefix as tools, then system, then messages, with a hierarchy where each level rests on the previous one. And the invalidation rule is explicit: modifying tool definitions, be it names, descriptions or parameters, invalidates the entire cache, and a change at one level invalidates that level and all subsequent ones.

OpenAI says the same and adds a detail: the prefix must remain unchanged, and order counts. The same set of tools served in a different order is a cache miss.

With vLLM serving an open model there is no tool block of its own. The tools parameter is serialised by the chat template inside the first system message, behind the system prompt text. For automatic prefix caching the effect is identical, because it is still near the absolute start of the prompt.

The consequence is the one that matters. A filter that returns a different set of tools on the next turn does not just invalidate the tool block: it invalidates everything behind it, which is the system prompt and the whole conversation, which is exactly the part that grows with the turns. At list prices, a prefix token goes from costing 0.1 times when read from cache to 1.25 times when written. An agent with sixty thousand tokens of stable prefix goes from some six thousand equivalent tokens per turn to some seventy-five thousand, plus the prefill that gets paid again in latency.

The two big providers have separately arrived at the same solution, and it is the opposite of filtering in the prefix. Anthropic excludes deferred tools from the prefix and, when the model discovers one through search, injects it inline inside the conversation: “the prefix stays intact, so prompt caching is preserved”. OpenAI: “all tools are loaded at the end of the model’s context window (…) this allows the cache to be preserved from one request to the next”. Anthropic’s operational recommendation is the stable core pattern: leave the three to five most used tools undeferred.

The closest academic work, from January 2026, states the problem without measuring it and recommends exactly that, keeping a fixed set of reusable general-purpose functions and implementing dynamic capability through code generation.

The July 2026 MCP specification picks up both halves of the matter. Servers “should return the tools from tools/list in a deterministic order to allow caching in the client and improve the hit rate of the model’s prompt cache”. And it adds mandatory ttlMs and cacheScope in listing results, so that the client can freeze the catalogue for a known window instead of re-querying it and propagating any variation to the prompt.

Nobody has publicly measured the cost in euros of this effect. The only attempt I found is a self-published audit of ten thousand turns that puts it at 2.4 % to 3.5 % of cache misses, with non-reproducible methodology. It is a gap in the literature, and the order of magnitude in the previous paragraph comes from applying list prices, not from a measurement.

And now, LiteLLM

With all of the above in mind, the gateway’s levers read differently.

The semantic filter does not act on the MCP endpoint. It is registered as a generic LiteLLM callback, and its only dispatcher is the proxy’s precall hook, which additionally discards anything that is not completion, acompletion or aresponses. The tools/list path does not invoke that hook at any point; there is a comment in the code that admits it in passing, when explaining why the listing’s JWT signature is done separately. An external MCP client connected to /mcp receives the whole catalogue, filter enabled or not.

Lever	Serves external MCP clients	Breaks the prompt cache
Per-server `allowed_tools`	Yes	No, it is static
Name and description overrides	Yes	No, it is static
Selection by route or header	Yes	No, if it is stable per agent
`/mcp/proxy` and virtual tools	Yes	No, the surface is fixed
Semantic filter	No	Yes, it rewrites the block per request

When it does act, the filter has details you need to know before enabling it. It embeds only each tool’s description, with the name as a fallback if it is missing; the parameter schema does not go in. Of the query it embeds only the last user message: on turn twelve of an agent conversation, that might be “yes, go ahead”. The tool embeddings are cached and indexed at start-up; the query’s is computed on every request, with a synchronous call inside a coroutine.

Its failure modes are almost all open, which is the reasonable thing: if the embedding model does not respond, if nothing passes the threshold, if the names do not match the catalogue, it returns all the tools. The exception is context overflow, which fails closed with a 400, and which if it happens while building the index during start-up is memoised and makes all subsequent requests fail until you restart. The defaults are text-embedding-3-small, top_k of 10 and a threshold of 0.3, under litellm_settings.mcp_semantic_tool_filter.

/mcp/proxy does serve external clients, and that is where its real value lies. It replaces the catalogue with three tools with opaque identifiers. How search_tools searches deserves a line, because it is not what you would assume: by default it is neither a regular expression nor a semantic match, it is a keyword counter, how many tokens of the query appear as a substring in the concatenated name and description. It only moves to embeddings if you configure a model in mcp_tool_search. The result cap is hard-coded at five in this mode, and the client cannot ask for more. In exchange, call_tool goes through the same permission checks as a normal call plus a validation of the argument schema that the normal route does not do at that point.

There is a related variant: with mcp_tool_search_enabled in the key’s permissions, the catalogue is replaced by four virtual tools, and there the search one does expose top_k to the client.

The catalogue order is not deterministic, and that breaks the client’s cache without anybody filtering anything. The list of allowed servers comes from iterating a Python set of strings, so with the random hash seed the order differs between processes, between uvicorn workers and between restarts. There is no sorting anywhere in the listing path, and there is no tool cache: every tools/list queries the upstreams live, so the order within each server also depends on what the upstream returns that time. The gateway does not offer today the guarantee that the July specification recommends. The fix, sorting the catalogue before serving it, fits on one line.

Nor is there any pagination towards the client. The catalogue is returned whole, always.

And the lever almost nobody uses is the static one. Every MCP server accepts tool_name_to_display_name and tool_name_to_description, and those overrides really do replace what is sent to the client in tools/list, not just what the administrator sees in the interface. The tools/call routing translates from the new name to the real one before any permission check. With that you can trim a four-hundred-token description written by a third party, rename an ambiguous tool so it does not compete with a similar one, and do it without touching the prompt cache, because it is static and holds for all turns.

It is also the rug pull mitigation left pending from the security section: if the description the model sees is the one the operator wrote, whatever the upstream server changes stops reaching the context. Two limits. They are configured through the database or the management API, not through config.yaml. And the reverse name translation is by exact equality and is not disambiguated between servers, so two servers with the same display name route unstably, made worse by the non-deterministic order of the previous paragraph.

Alongside that, per-server allowed_tools remains the blunt and effective instrument: it filters the listing the client sees, reduces tokens, and is static.

The ladder, in order

Trim. Per-server allowed_tools, plus a per-key allowlist. Cheap, static, does not break the cache and reduces the permission surface at the same time.
Rewrite the descriptions with the overrides. That is where the evidence is, because the description carries weight and the name almost none, and along the way it pins the contract against the rug pull.
Split by route. Each agent to /{server}/mcp or to a tool set, never to the aggregated endpoint. Stable per agent, so the cache survives.
Sort the catalogue before serving it if you control the deployment, while the gateway does not do it.
/mcp/proxy when the catalogue goes past thirty or fifty tools. It is progressive disclosure and it works with external clients, which is more than you can say for the semantic filter.
The semantic filter, only when the one calling the model is LiteLLM. With a generous top_k, knowing that it embeds only the last user message and that context overflow is a closed and sticky failure.
What not to do: a home-grown top-K filter that rewrites the tools array on every turn. It costs the whole prefix and, according to four of the five measured strategies, probably makes selection worse.

The declared cost, which is worth zero

The calculation has four precedence steps: cost set by a post-call hook, per-tool cost in tool_name_to_cost_per_query, server default cost, and 0.0.

Without configuring mcp_server_cost_info, all MCP traffic is accounted at zero and consumes neither key nor team budget.

There are exactly two Prometheus metrics for MCP:

litellm_mcp_tool_calls_total
litellm_mcp_tool_call_spend_metric

Both with the same eight labels: tool name, server name, key hash, key alias, team, team alias, user and end user.

And the second is only incremented if the cost is greater than zero. With the default configuration, that time series never appears in Prometheus. It is not that it is worth zero: it is that it does not exist.

Nor is there a tools/list metric, or MCP latency, or MCP errors, or upstream server health. For that you are left with the generic request metrics.

In the reports, /spend/calculate has no MCP branch: it accepts a model and messages, or a completion response, and it always calls the LLM cost calculation. The per-team report does include MCP calls, grouped by the model column, which for these rows is worth MCP: tool_name. That is: every tool appears as if it were a model, with zero input and output tokens. Only the sessions endpoint separates the two call types explicitly.

Observability, and the trace that gets cut

There are two OpenTelemetry stacks in LiteLLM and which one runs depends on an environment variable.

Without LITELLM_OTEL_V2=true, the old logger runs, and it has no MCP handling: the only appearance of MCP in the whole file is the metadata key in a list. The tool call comes out as a generic litellm_request span with the model set to MCP: name.

With v2 enabled there is a dedicated span, with a dedicated role, client kind and a parent in the proxy’s request span. The span name is tools/call tool-name. The attributes are correct and follow the conventions:

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`mcp.method.name`	`tools/call`
`mcp.session.id`	MCP session identifier
`gen_ai.tool.name`	Tool name
`gen_ai.tool.call.arguments`	Only with content capture enabled
`gen_ai.tool.call.result`	Only with content capture enabled
`server.address`, `server.port`	From the upstream URL, redacted
`litellm.cost.total`	The cost that is probably worth zero

Content capture is off by default. Without it you see neither the arguments nor the result in the trace.

A note for anyone working with semantic conventions: mcp.tool.name does not exist. The MCP registry has exactly four attributes (mcp.method.name, mcp.session.id, mcp.protocol.version and mcp.resource.uri), all four in development, and the tool name goes in gen_ai.tool.name. LiteLLM gets it right. And the GenAI conventions were extracted into their own repository which as of today has no release at all, so there is no version number to cite.

In Langfuse, the vendor mapper returns an empty dictionary for any span that is not an LLM call. An MCP call appears as a raw span with its attributes, with no native input, output or model fields.

The trace is cut at the gateway

Incoming context is read. The traceparent the client puts in params._meta, following the specification’s propagation proposal, is extracted and turned into a link on the span, never into its parent. The reason is documented in the code: parenting to the remote trace would leave the span hanging off a trace whose root never reaches the backend. The client’s baggage is discarded on purpose to avoid forging identity attributes.

Outgoing context is not propagated. There is no propagator injection call anywhere in the tree. The headers that go out towards the upstream MCP server are built with authentication and the authorised extra headers, and nothing else.

The operational consequence: there is no end-to-end trace. If the upstream MCP server is instrumented, its spans live in another trace and there is no way to correlate them automatically.

Auditing and the conflict with privacy

A tool call leaves a normal spend row, with call type call_mcp_tool, spend equal to the cost per query, model MCP: name, zero tokens in the three token columns, a dedicated mcp_namespaced_tool_name column, and the usual key, team, user and session fields.

And in the metadata column, in JSON, the complete structure of the call: name, arguments, result, server, namespaced name, MCP session identifier, authentication mode and server resource with the URL redacted.

The problem

That structure is written into the metadata column without passing through the privacy switch. The assignment is direct and is not conditioned by the function that governs whether prompts and responses are stored, which does control the messages and response columns.

And the redaction function does not cover it either: it only touches messages and response. A search for MCP in the redaction file returns nothing.

The practical result: an operator who enables turn_off_message_logging for GDPR keeps sending the full arguments of every tool to every logging destination they have configured. To the SIEM, to Langfuse, to S3.

If an argument carries a national ID number, an address, a case number or a medical record, it has been exported.

There is an aggravating factor that does not depend on LiteLLM. The revision of the specification in force introduces an extension that allows mirroring tool arguments into HTTP headers visible to load balancers, proxies and web application firewalls. The specification recommends not marking sensitive parameters that way, and it is a recommendation, and the upstream server decides.

The fit with the ENS

The conflict is real and has a familiar shape: activity logging (op.exp.8) requires the call to be traced and non-repudiable; protection of personal data (mp.info.1) requires that log not to become a secondary repository of personal data with a different purpose. The tool argument is simultaneously the evidence of the action and the data. Redacting it destroys non-repudiation; storing it in the clear creates the repository.

The defensible way out you can build today: a hash of the argument in the audit log, the argument in the clear with encryption (mp.info.3), separate retention and its own access control. The hash sustains non-repudiation and the clear text is only opened under procedure.

There are two other measures this material touches directly. Change management (op.exp.5) because registering an MCP server and, above all, changing the description of an already approved tool are changes to the security configuration. And maintenance and updates (op.exp.4) because, as you can see in the next section, an old version of the gateway is a non-compliance with a name and a number.

Two honesty warnings. The codes follow the numbering of Annex II of RD 311/2022 and you have to check them against the text in force before citing them in a compliance document. And on NIS2: as of the date of this post the directive is still not transposed in Spain, with the draft bill on cybersecurity coordination and governance going through the process and a reasoned opinion from the Commission over the delay. What is enforceable today in the public sector and its suppliers is the ENS.

What is not logged

An administrative gap: registering, modifying and deleting MCP servers generates no audit log. There are three TODO comments in the management code that say so, one of them with an if is_audit_logging_enabled(): pass. In the same deletion block there are two more TODOs about not cleaning up orphaned permissions on keys and teams.

Security

The vulnerability you need to know about

CVE-2026-42271, published on 8 May 2026. Affects LiteLLM from 1.74.2 up to versions prior to 1.83.7. CVSS 3.1 of 8.8.

The POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list endpoints, which exist to preview an MCP server before saving it, accepted the stdio transport’s command, args and env in the request body. A low-privilege key executed arbitrary commands on the proxy host. It also affects Red Hat OpenShift AI on several branches.

It is the story of this post condensed into one line: the gateway you put in to contain MCP was the hole.

It does not come alone. The 1.83 branch concentrates three more serious ones:

CVE	Component	CVSS	Fixed in
`CVE-2026-42208`	SQL injection in key validation, with no prior authentication	9.8	1.83.7
`CVE-2026-35030`	OIDC userinfo cache keyed on `token[:20]`, collision and impersonation	9.1	1.83.0
`CVE-2026-40217`	Remote execution through bytecode rewriting in the guardrails test endpoint	8.8	no version recorded

Plus two privilege escalations fixed in 1.83.10 and 1.83.14.

The operational conclusion is short: a deployment below 1.83.14 is exposed to remote execution or to an unauthenticated SQL injection. The current branch is 1.102.0.

And a note for context: the whole MCP ecosystem has the same pattern. The unauthenticated remote execution in MCP Inspector, the command injection in mcp-remote when connecting to an untrusted server, DNS rebinding disabled by default in the Python and TypeScript SDKs until late 2025, the directory escapes in the reference servers. Connecting to an untrusted MCP server compromises the client, not just the other way round.

The rug pull nobody covers

Documented since April 2025: a server can change a tool’s description after the client has approved it. The description is what the model reads to decide when and how to use it, so changing it is changing the behaviour without asking for permission again.

The specification in force does not address it. It defines list change notification and cache freshness fields, which are recency mechanisms, not integrity ones. The closest normative warning is that clients must treat tool annotations as untrusted unless they come from trusted servers.

LiteLLM does not cover it either. The only “pinning” in the code is of the configuration’s server identifier, so that it does not change when you edit the YAML. There is no hash or comparison of the description or the input schema between successive listings. A search for related terms in the MCP module: nothing functional.

The control exists outside the gateway. mcp-scan, from the same researchers who documented the attack, implements description hash pinning precisely to detect this.

In ENS terms it is an unauthorised change to the security configuration that today goes undetected. And the home-made fix fits in a cron job: store the hash of the description and schema of every approved tool, compare against tools/list periodically, alert on the difference.

Three more holes in the code

No URL validation towards the upstream. A new server’s url field has no validator. The SSRF protection utilities exist in the tree and are used in OAuth discovery and in downloading OpenAPI specifications, but not in the egress of tools/call. A server pointing at a private range or at the cloud metadata IP is accepted as is.

The stdio transport is local execution on the proxy host. There is a command allowlist: npx, uvx, python, python3, node, docker, deno, extendable through an environment variable. The code’s own comment admits the residual risk: the permitted runtimes can execute code through arguments. With docker and npx in there, it is de facto arbitrary execution for any proxy administrator.

Two server modes skip the gateway’s authentication. Pass-through and authentication delegation to the upstream completely omit virtual key validation. They are well hardened (fail-closed if the target is mixed or unresolvable, strict comparison against truthy values, explicit blocking of the client credentials flow so as not to lend out the proxy’s service account), and the result is still that those servers have no gateway access control.

The limit that truncates in silence

The tools/list pagination stops and returns what it has accumulated in three cases: repeated cursor, one thousand pages, or expiry of a global deadline. The docstring declares it: a slow or faulty upstream produces a partial catalogue instead of an error.

The first two log a warning. The third, the deadline, exits without logging anything at all. A slow server makes part of its tools disappear from the catalogue with no trace left behind.

Correction on the slot leak

In the previous post I took issue 34534 at face value, according to which every MCP tool call acquires a max_parallel_requests slot and does not release it. Tracing the 1.102.0 code, that diagnosis no longer holds: the release path exists, on both success and failure, and the limiter is registered as a litellm callback and not just as a proxy hook.

With one honest caveat: I found no regression test covering the full cycle for MCP, and the definitive confirmation is a real proxy with max_parallel_requests: 2 and five MCP calls in a row. If anybody reproduces it today, I would like to know.

The specification moved underneath

The MCP revision in force is 2026-07-28, published on 28 July 2026, and in between there was another one, 2025-11-25. The changes in the latest one are not cosmetic:

Protocol sessions and the session header are removed. Cross-request state becomes an explicit identifier in the tool arguments.
The initialisation handshake is removed. Every request carries version and capabilities in _meta. MCP becomes stateless.
server/discover appears as a mandatory call.
Elicitation changes shape: out go server-initiated requests, in comes a multi-round pattern where the server returns an “input required” result and the client retries with the answers.
ping, logging/setLevel and the roots change notification are removed. SSE stream resumption is removed.
Standard HTTP headers are mandatory on POST, which lets a gateway or a web application firewall decide without parsing the JSON.
On authorisation: mandatory issuer validation in the client, and dynamic client registration is deprecated in favour of client identifier metadata documents.
Roots, sampling and logging are deprecated, with a minimum window of twelve months.

LiteLLM 1.102.0 advertises 2025-06-18. Its version enumeration has three values and none of them is later than that date. The literal is sent in the initialize against the upstream.

One nuance that is not entirely bad: the version the gateway advertises to its clients is not hard-coded in LiteLLM, the SDK negotiates it. And there is partial implementation of the intermediate revision, because the elicitation and sampling handlers reference it in their docstrings. That is: partial implementation of one revision while advertising an earlier one.

In practical interoperability it does not hurt yet, because the client ecosystem is still mostly on the 2025 revisions. It hurts in planning: July’s stateless redesign invalidates the gateway’s session model, and that is work ahead.

Two ecosystem notes for anyone choosing. There are clients that speak remote streamable HTTP natively (Claude and Claude Code, ChatGPT with OAuth and dynamic registration, VS Code with Copilot, Cursor, Zed, Kiro) and others that still demand stdio or a local bridge with mcp-remote, among them Windsurf, Cline and Continue.

Tool execution from the proxy itself

Besides serving MCP outwards, the proxy can consume it itself. It is enabled with an mcp block in the body, with server_url equal to the litellm_proxy sentinel, and it works both on /v1/responses and on /chat/completions.

The flow: it separates the gateway’s tools, lists them, filters by allowed, deduplicates, translates them into OpenAI function format and concatenates them. It calls the model. If there are tool calls and auto-execution is permitted, it runs them against the MCP servers. It builds the follow-up and makes one second call.

Two limits you need to know before designing on top of this.

The loop is a single hop. There is no multi-step iteration. The calls from the first response are executed and the follow-up is returned. A plan that chains tools requires the client to call again.

Auto-execution fails closed and as a block. It requires all MCP references to have approval set to “never”; a single one requiring approval disables auto-execution for the whole request.

And a third one, which is the one that bites: the multi-turn state of /v1/responses is rehydrated from the spend table. The query is literal, it looks up the session of the previous request_id and retrieves all the rows of that session ordered by time.

With disable_spend_logs: true nothing is written, the query comes out empty and the history is lost in silence, with no 4xx error. The code itself acknowledges it: it does not expect retries because deployments that do not write spend logs have nothing to wait for. It is a coupling between a performance option and a conversation feature that nobody expects to find.

Health and day-to-day operation

Health checks exist, they are on demand and they are not periodic. GET /v1/mcp/server/health accepts repeated identifiers and returns per-server status. The implementation opens a client and runs an empty operation inside the session, with a ten-second timeout.

It skips the check entirely if the server requires per-user authentication, and those always return unknown status.

The surprising part: the status, last check and error columns have existed in the database since a May 2025 migration, and the check function returns an object without writing to the database. Status is recalculated on every request. There is no periodic MCP health job.

When an upstream server is down, the behaviour is to degrade and not block. The listing fans out in parallel and each failing server contributes an empty list plus a result classified into six categories. Those results reach the client in the response’s _meta under a vendor key, without filtering the upstream’s text or URL. The reason for the design is in the code: an empty contribution with no signal makes a broken upstream indistinguishable from a healthy server with no tools.

There is no cache of the previous list. A server that is down disappears from that turn’s tools.

On registering and deregistering without restarting, there are two worlds. Database servers can be created, edited and deleted through the API, and every operation reloads the registry with an atomic swap. Propagation between replicas goes through Redis publication on the configuration change channel, if there is Redis, and through polling every 30 seconds if there is not.

Servers declared in config.yaml are loaded once, at start-up. There is no re-read. Changing them requires a restart.

And there is a precedence trap between the two worlds, with two different warnings in the code. A database row with the same identifier as a configuration entry hides the configuration one entirely. And the other way round, a configuration identifier that matches the name or alias of a database server captures its permissions, because identifiers are resolved before names. Both cases emit a log warning, once.

A hardened `config.yaml`

general_settings:
 # Access is not open by default, but this closes the empty-key
 # inheritance towards the team, which is the one that surprises people.
 require_key_mcp_access_defined: true
 require_end_user_mcp_access_defined: true
 # Without this, allow_all_keys overrides the key's explicit scope.
 mcp_allow_all_keys_respects_mcp_scope: true

litellm_settings:
 # Does not cover MCP arguments. See the auditing section.
 turn_off_message_logging: true
 callbacks: ["langfuse_otel"]

mcp_servers:
 inventario:
 url: https://mcp-inventario.interna.svc/mcp
 transport: http
 auth_type: bearer_token
 auth_value: os.environ/MCP_INVENTARIO_TOKEN
 # Explicit allowlist. Also enforced on tools/call.
 allowed_tools:
 - buscar_articulo
 - consultar_stock
 # Never true on a server with data.
 allow_all_keys: false
 access_groups: ["operaciones"]
 timeout: 20
 max_concurrent_requests: 8
 mcp_info:
 description: "Inventario de almacen, solo lectura"
 # Without this, MCP spend is zero and its metric does not exist.
 mcp_server_cost_info:
 default_cost_per_query: 0.0005
 tool_name_to_cost_per_query:
 consultar_stock: 0.0002

And in the process environment:

# Without this there is no MCP span, only a generic litellm_request.
LITELLM_OTEL_V2=true
# Full block hashes, not truncated, if correlating
# with engine events.
LITELLM_MCP_CLIENT_TIMEOUT=45
LITELLM_MCP_TOOL_LISTING_TIMEOUT=20

For clients, the rule is not to use the aggregated endpoint. Each agent points at /{server}/mcp or at a tool set, or at the three-tool proxy mode if the catalogue is large.

Checklist

Check the version. Below 1.83.14 there is remote execution and an unauthenticated SQL injection with an assigned CVE.
Inventory which servers have allow_all_keys and enable the setting that makes it respect the key’s scope.
Audit the access groups separately, because they are additive on top of the key and team ceiling.
Declare in writing the fail-open behaviour on tool permission resolution errors.
Measure the real cost: number of tools per server from the listing row, and tools block tokens from the stored request body.
Choose a catalogue strategy before connecting the third server: three-tool proxy mode, semantic filter, or selection by route. Not the aggregated endpoint.
Declare mcp_server_cost_info even with a token value, or the spend metric will not exist.
Enable the OTel v2 logger, and decide deliberately whether to enable content capture.
Assume the trace is cut at the gateway and plan correlation with the upstream by other means.
Treat the spend table’s metadata column as a personal data repository: encryption, its own retention and separate access control.
Set up the rug pull check, because nobody does it: hash of the description and schema of every approved tool, periodic comparison against the listing.
Restrict egress towards MCP servers at the network level, because the gateway does not validate the destination URL.
Do not enable the stdio transport unless you need it, and if you do, review what is on the command allowlist.
If you use /v1/responses with multi-turn state, do not enable disable_spend_logs.

Traps and things that are not what they seem

MCP access is not open by default in 1.102.0. I got that wrong in the previous post. What opens things up is per-server allow_all_keys.
allow_all_keys overrides the key’s explicit scope unless you enable a setting that ships disabled.
require_key_mcp_access_defined only closes key-to-team inheritance. Access group grants remain additive.
On an indeterminate failure resolving tool permissions, the code fails open for keys and JWTs.
The default cost of an MCP call is 0.0, and the spend metric is only emitted if the cost is greater than zero, so the series does not exist.
MCP calls appear in the per-team report as if they were models, named MCP: tool and with zero tokens.
The OTel logger with MCP support is off by default. Without the environment variable there is no MCP span.
Trace context is not propagated towards the upstream MCP server. There is no end-to-end trace.
mcp.tool.name does not exist in the semantic conventions. The name goes in gen_ai.tool.name.
Message redaction does not cover tool arguments. They are written in the clear into the spend table.
Registering, modifying and deleting MCP servers generates no audit log. There are three TODOs in the code.
The semantic tool filter is not applied on tools/list. Only on /chat/completions and /v1/responses.
The catalogue order is not deterministic, because it comes from iterating a set of strings. It breaks the client’s prompt cache even though nobody filters anything.
search_tools in proxy mode counts keywords by default, it uses neither regular expressions nor embeddings, and its cap of five results is hard-coded.
Description overrides really do replace what the client sees, and they are the only catalogue lever that does not break the cache. They are only configurable through the database or the API.
spec_version no longer exists in the configuration schema.
/mcp/sse does not use the SSE transport, and the messages endpoint its object advertises is dead code.
Listing pagination truncates in silence when the deadline expires, without leaving a single log line.
A database row hides a configuration entry with the same identifier, and the other way round a configuration identifier can capture the permissions of a database server.
Servers declared in config.yaml require a restart. Only the database ones reload hot.
The health check does not write its result to the database even though the columns exist, and there is no periodic job.
The gateway does not validate the upstream MCP server’s URL. The anti-SSRF utilities exist and are not used on that path.
docker and npx are on the stdio command allowlist.
The /v1/responses tool loop is a single hop, and disable_spend_logs: true breaks its multi-turn behaviour with no error.
LiteLLM advertises the June 2025 revision of the protocol, two behind the one in force.

Closing

There is a thread running through the six posts in this track that is more visible here than anywhere else. The gateway is chosen for one job and ends up doing five, and each new job arrives with its own defaults, which are almost never the ones you would have chosen.

In the case of MCP the asymmetry is twofold. On one side, the part that looks dangerous is better built than you would assume: access control is enforced on execution and not just on the listing, the hierarchy is built out of intersections, header forwarding is an allowlist, and default access is closed, contrary to what I myself wrote two days ago. On the other, the part nobody looks at is worse: tool arguments come out in the clear bypassing the privacy switch, spend is worth zero and its metric never gets to exist, the trace is cut, and a change to the description of an approved tool goes undetected by anybody in the whole chain, neither the specification nor the gateway.

The architecture decision that comes out of this is that the MCP door has to be treated as what it is, an interconnection of systems with third parties, and not as one more function of the LLM proxy. That means its own inventory, formal registration, change control over every tool’s description, network isolation towards the upstream and a log designed knowing that it is going to contain personal data.

And it means accepting that the cost you really pay for connecting fifteen MCP servers is in no column of the spend table. It is spread across the prompt_tokens of every model call, on every turn, and across the accuracy you lose when the catalogue is too large for the model to choose well. That number can be computed with what the gateway already stores. The first step is wanting to compute it.

Sources

LiteLLM, MCP Gateway: https://docs.litellm.ai/docs/mcp.
LiteLLM, código: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, auth/user_api_key_auth_mcp.py, cost_calculator.py, faults/list_outcomes.py, litellm/proxy/management_endpoints/mcp_management_endpoints.py, litellm/proxy/spend_tracking/spend_tracking_utils.py, litellm/litellm_core_utils/redact_messages.py, litellm/integrations/otel/, litellm/responses/litellm_completion_transformation/session_handler.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en llamadas MCP: https://github.com/BerriAI/litellm/issues/34534.
NVD, CVE-2026-42271, ejecución de comandos por los endpoints de prueba de MCP: https://nvd.nist.gov/vuln/detail/CVE-2026-42271.
NVD, CVE-2026-42208, inyección SQL sin autenticación en la validación de clave: https://nvd.nist.gov/vuln/detail/CVE-2026-42208.
NVD, CVE-2026-35030, colisión de clave de caché de userinfo OIDC: https://nvd.nist.gov/vuln/detail/CVE-2026-35030.
NVD, CVE-2025-49596, ejecución remota en MCP Inspector: https://nvd.nist.gov/vuln/detail/CVE-2025-49596.
NVD, CVE-2025-6514, inyección de comandos en mcp-remote: https://nvd.nist.gov/vuln/detail/CVE-2025-6514.
NVD, CVE-2025-66416, protección de DNS rebinding desactivada por defecto en el SDK de Python: https://nvd.nist.gov/vuln/detail/CVE-2025-66416.
Invariant Labs, MCP Security Notification: Tool Poisoning Attacks (tool poisoning, rug pull, shadowing): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks.
Invariant Labs, Introducing mcp-scan (fijado de herramientas por hash): https://invariantlabs.ai/blog/introducing-mcp-scan.
Model Context Protocol, changelog de la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/changelog.
Model Context Protocol, changelog de la revisión 2025-11-25: https://modelcontextprotocol.io/specification/2025-11-25/changelog.
Model Context Protocol, Security Best Practices (confused deputy, indicadores de recurso, validación de audiencia): https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.
NSA, Model Context Protocol: Security Design Considerations for AI-Driven Automation, mayo de 2026: https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDF.
CISA y agencias aliadas, Careful Adoption of Agentic AI Services, mayo de 2026: https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services.
Anthropic, Advanced tool use (coste en tokens de las definiciones, búsqueda de herramientas, precisión de selección): https://www.anthropic.com/engineering/advanced-tool-use.
Anthropic, Code execution with MCP: https://www.anthropic.com/engineering/code-execution-with-mcp.
Anthropic, Prompt caching (jerarquía tools → system → messages e invalidación): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.
Anthropic, Tool search tool (herramientas diferidas fuera del prefijo, umbral de 30 a 50 herramientas): https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, Prompt caching (el orden de las herramientas cuenta): https://developers.openai.com/api/docs/guides/prompt-caching.
OpenAI, Tool search (las herramientas descubiertas se cargan al final del contexto): https://developers.openai.com/api/docs/guides/tools-tool-search.
vLLM, Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html.
Microsoft, Use tools in chat (límite de 128 herramientas por petición en VS Code): https://code.visualstudio.com/docs/copilot/agents/agent-tools.
RAG-MCP, prueba de estrés de 1 a 11.100 servidores candidatos, mayo de 2025: https://arxiv.org/abs/2505.03275.
Benchmarking the Benchmarks, desalineación del 18,5 % entre etiqueta y realidad, junio de 2026: https://arxiv.org/html/2607.02577v1.
MCP-Atlas, taxonomía de fallos sobre 220 herramientas y veinte modelos, febrero de 2026: https://arxiv.org/html/2602.00933v3.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307.
MCP Tool Descriptions Are Smelly, auditoría de 856 herramientas de 103 servidores, febrero de 2026: https://arxiv.org/html/2602.14878v1.
ToolMenuBench, seis estrategias de filtrado y 26.460 ejecuciones, junio de 2026: https://arxiv.org/html/2606.15508.
Retrieval Models Aren’t Tool-Savvy (ToolRet), corpus de 43.000 herramientas, marzo de 2025: https://arxiv.org/abs/2503.01763.
Tools Are Not Islands, brecha entre Recall@3 y conjunto completo, julio de 2026: https://arxiv.org/html/2607.25718.
ToolFlood, saturación adversaria de la capa de recuperación, marzo de 2026: https://arxiv.org/html/2603.13950.
Scores Are Not Decisions, el K óptimo como función del coste, julio de 2026: https://arxiv.org/html/2607.27083v1.
Don’t Break the Cache, caché de prompts en tareas agénticas de horizonte largo, enero de 2026: https://arxiv.org/html/2601.06007v2.
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer, mayo de 2026: https://arxiv.org/html/2605.24660v1.
Semantic Tool Discovery for LLMs, marzo de 2026: https://arxiv.org/abs/2603.20313.
OpenTelemetry, convenciones semánticas de GenAI y MCP: https://github.com/open-telemetry/semantic-conventions-genai.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
IBM ContextForge, gateway y registro MCP: https://github.com/IBM/mcp-context-forge.
Docker MCP Gateway: https://github.com/docker/mcp-gateway.
agentgateway: https://github.com/agentgateway/agentgateway.

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

Fri, 11 Sep 2026 00:30:00 +0200

Fifth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability, virtual keys covered governance and humans and agents covered the coexistence of two classes of traffic. Here a claim of mine from June gets taken apart. Everything that follows is verified against LiteLLM 1.102.0 (commit of 10 September 2026), vLLM 0.29.0 and the main branches of production-stack, llm-d and Dynamo from the second week of September 2026.

TL;DR

LiteLLM does not do prefix-aware routing, neither in the open version nor in the paid one. It is not a licensing question. A search across the whole tree for kv_aware, cache_aware, radix, prefix_cache and block_hash returns zero matches in the router. Nor is there any call from the router to /metrics or to any vLLM endpoint. In June I wrote the opposite in the inference router article and I was wrong.

What LiteLLM has is affinity, which is a different thing. Four pre-call checks pin a request to a replica by session identifier, by key hash, by previous response identifier or by exact hash of a marked prefix. None is active by default. Affinity distributes conversations; prefix routing distributes prefixes shared between different conversations, which is the case that multiplies throughput.

The only one that looks at the prompt is fragile in three ways. If the client does not send cache_control, the key comes out null and the check does nothing. If the cut point goes in the last turn, which is what the automatic injection does by default, the key changes on every turn and it never hits. And the pin’s TTL is 300 seconds hardcoded in two lines, while the provider offers one-hour caches.

The engine does not answer questions, it publishes events. There is no vLLM endpoint you can ask whether it has a prefix. What there is, is a ZMQ socket that emits stored and removed blocks in msgpack, switched off by default, with silent discarding when the consumer does not keep up. A router that wants to be exact has to replicate the engine’s hash in full: chained SHA256, deterministic seed, block granularity and the extra keys in their order.

And by default it does not publish the hits, only the additions. The index of a router that only listens drifts away from the engine’s real LRU without noticing. To see the hits you have to ask for it request by request with kv_cache_report_mode: full.

The three routers that do do it are very different from each other. The vLLM production-stack one indexes chunks of 128 raw characters in a trie that is never purged, models no eviction and dies on every restart. The llm-d one offers an approximate mode and an exact event-driven one, all its plugins are in alpha or beta, and its default tokeniser does not produce real tokens. NVIDIA Dynamo’s keeps an event-driven radix tree with a weighted cost function, and ships with the parameter that avoids starving a freshly scaled replica switched off.

The big improvements only appear in the best possible case. The two- and three-digit numbers are published by the projects themselves, with synthetic shared prefixes and a round-robin balancer as the reference. The most complete independent measurement, from Meta, obtains 2.3 times more throughput and also identifies load regimes where affinity leaves capacity at between half and two thirds of the reference. Another, on a multi-region deployment with 89 % reuse, measures single-digit improvements.

The correct architecture has two layers, not one. LiteLLM on top for identity, budget and audit, pointing at a single destination per model group. The KV-aware router underneath, inside the engine’s domain. Mixing it into a single piece is not an option that exists today.

You are here: between the gateway and the engine

This article occupies the gap between two layers that already have an article of their own. Above, the L7 switchboard and the gateway that has been operated since the first article in this track. Below, the fundamentals of the KV cache and hit rate engineering.

The gap is the piece that decides which replica each request goes to, knowing what each replica has in memory. It is the only function in the stack that cannot be resolved in the gateway or in the engine separately.

The analogy: the library with four counters

A library with four counters. Each librarian has in front of them a pile of the books they have consulted lately. When a reader comes in asking for the third volume of an encyclopaedia, if that volume is in the pile at counter two, serving them there takes seconds. At any other counter someone has to go down to the store room.

A normal balancer sends the reader to the counter with the shortest queue. It is the right decision if all the counters are interchangeable, and it is the wrong decision here, because they are not: what distinguishes them is not the queue, it is the pile.

Session affinity resolves one specific case of this. The returning reader is always sent to the same counter, because they will probably ask for the fourth volume of the same encyclopaedia. It works, and it does not resolve the general case: when a hundred different readers ask for the same third volume, session affinity spreads them across the four counters and four librarians go down to the store room for the same book.

Prefix routing is looking at what the reader asks for before assigning them a counter. It has two costs that the analogy makes visible straight away. The first is that somebody has to keep track of what is in each pile, and that bookkeeping takes work. The second is that if everybody asks for the third volume, counter two becomes the library’s only desk while the other three look on.

The correction: what I wrote in June

In the inference router article, on 2 June 2026, I closed the prefix routing section with this sentence: “vLLM Production Stack router implements it natively. NVIDIA Dynamo too. LiteLLM in its enterprise version has a beta.”

The first two hold up. The third is false and was no less false in June. While preparing the humans and agents article the first hint appeared, and full verification against 1.102.0 confirms it.

Case-insensitive search over litellm/, enterprise/ and gateway/, for the terms kv_aware, kv-aware, cache_aware, cache-aware, radix, prefix_cache, prefix-cache, kv_cache, block_hash, prefix_hash, prefix_score and longest_prefix: zero matches. Search for vllm inside litellm/router.py, litellm/router_strategy/ and litellm/router_utils/: zero. Search for num_requests_waiting, gpu_cache_usage or any scraping of the backends’ /metrics from the router: zero. LiteLLM’s router never asks the engine anything.

Search for premium_user in the files where those checks would live: a single result, and it is about something else. There is no paid version with this feature. The only routing piece that is behind a licence is per-tag budgets, in litellm/router_strategy/budget_limiter.py.

Where the error came from. LiteLLM’s documentation has an entry on dynamic TPM and RPM allocation marked as beta and enterprise, and a pre-call check called prompt_caching. Skim-read and added together, they give the impression of prefix routing in beta. Read in the code, prompt_caching is something else, and it deserves a section of its own because it is still recommended for this problem.

What LiteLLM does have: four affinities

The optional_pre_call_checks field accepts eight values. None is enabled by default: apply_default_settings passes an empty list. One of the eight, forward_client_headers_by_model_group, is declared in the type and in the interface schema and is consumed by no branch of the code. It is a dead value.

The four that pin a request to a replica are these.

Check	What it pins	Where it stores it	TTL
`session_affinity`	Session identifier	Redis with a Lua script, mirrored in memory	`deployment_affinity_ttl_seconds`, 3600 by default, refreshed
`deployment_affinity`	API key hash	Same	Same
`responses_api_deployment_check`	`previous_response_id`	Same	Same
`prompt_caching`	Exact hash of a marked prefix	`DualCache`	300 s, hardcoded

The other four are router_budget_limiting, encrypted_content_affinity, enforce_model_rate_limits and the dead value.

`prompt_caching`: why it is no use for this

The key is a SHA256 over the serialisation of the cacheable prefix, and “cacheable prefix” means everything up to and including the last block marked with cache_control: {"type": "ephemeral"}. Three consequences, checked by running the functions in isolation over the repository’s code.

If the request carries no cache_control, the extraction returns an empty list, the key comes out null and the check pins nothing. Automatic injection of cut points is only enabled with litellm.enable_anthropic_prompt_caching, which defaults to false, and only for Anthropic and Bedrock providers.

If the cut point is in the system prompt, the key does not change when turns are added and the pin works. This is the good case and it requires the client to mark it deliberately.

If the cut point is in the last message, the key changes on every turn and the pin never hits. And this is exactly the default case of the automatic injection, which places one point on the system and another at index -1. Since the second is the last, the cacheable prefix becomes the whole conversation.

On top of that there is the TTL. Two lines with ttl=300, one of them with the comment # store for 5 minutes. There is no parameter to change it. Issue 28427 has had it open for months with no response from the maintainers, and it points to the mismatch with the one-hour ephemeral caches that LiteLLM itself knows how to request via cache_control.ttl: "1h".

There is a nuance the issue does not capture and that makes diagnosis worse: in the default case the pin does not even last five minutes, because the key changes first.

`session_affinity`: the one to use

For conversations, this is the right piece and it is reasonably built. The session identifier is looked for in this exact order of headers:

x-litellm-trace-id
x-litellm-session-id
Any x-<vendor>-session-id, with the regular expression ^x-.+-session-id$ and a value of at least eight alphanumeric characters
Only if the User-Agent is Codex’s: session-id, session_id, thread-id, conversation_id
Plain x-session-id, for opencode

And if nothing matches, two fallbacks: a metadata.user_id in Anthropic format, and the W3C baggage with session.id=. The third point is the one that captures Claude Code with nothing configured.

Storage is a Lua script in Redis that does GET, SET NX EX and EXPIRE, so the first writer wins and the TTL is refreshed on every request that confirms the pin. It is an inactivity TTL, not a total session duration. Without Redis it degrades to a pod-local reservation, which is atomic but not shared.

A detail of the pipeline order that already came up in the previous article and still holds: the affinity checks run inside async_get_healthy_deployments before tag filtering. If the pin returns a single deployment and that deployment does not satisfy the request’s tag, the later filter removes it and the request fails. Affinity gets no vote on the tags.

The routers that do read the prompt, and why they are no good

LiteLLM has four pieces that read message content: AutoRouter with semantic matching, RequestComplexityRouter, AdaptiveRouter and QualityRouter. They all live in async_pre_routing_hook, which runs before looking for healthy deployments.

The reason they do not solve this problem is in what they choose. All four choose a model group, not a replica. They decide whether an easy question goes to the thirty-billion-parameter model or to the small one. None has access to the replica list or knows what is in each one’s memory.

What did land: admission control

A minor correction to the previous article, which placed this in the 1.101 branch as an upcoming feature. In 1.102.0 it is already there, in litellm/proxy/middleware/admission_control_middleware.py, with the three settings in general_settings: max_in_flight_requests_per_worker, max_queued_requests_per_worker and admission_queue_timeout_seconds, the last with a default value of 1.0 seconds.

It stays inactive as long as the first one is not declared. It exposes an authenticated /health/backlog and the three metrics litellm_admission_admitted_requests, litellm_admission_queued_requests and litellm_admission_rejected_requests_total, the last with a reason label. It is a per-process semaphore, not coordinated across pods.

What the engine exposes upwards

Before looking at the routers that do do this, you have to understand what raw material they work with. The natural question would be whether vLLM has an endpoint you can ask about a prefix. It does not.

The only cache router in the entry points is vllm/entrypoints/serve/dev/cache/api_router.py, and it only exposes destructive writes: POST /reset_prefix_cache, /reset_mm_cache and /reset_encoder_cache. All three are behind VLLM_SERVER_DEV_MODE=1 and do not exist in a normal deployment. There is no read.

What there is, is an event stream.

The ZMQ stream

It is enabled with --kv-events-config, and it is off by default. The configuration accepts a publisher (null or zmq), an endpoint (by default tcp://*:5557, which binds), an optional replay endpoint, the number of batches retained for that replay (10,000), and a high water mark limit of 100,000.

That limit is the one that matters operationally: above it, ZMQ discards events if the consumer does not keep up. Silently.

The transport is a PUB socket with msgpack serialisation and three-part frames: topic, eight-byte sequence number and payload. The sequence number lets a subscriber ask for replay from where it left off through the optional ROUTER socket. With data parallelism the port shifts by rank, so you have to subscribe to 5557+rank.

Three event types. BlockStored carries the hashes, the parent hash, the token identifiers, the block size, the LoRA adapter name, the extra keys, the medium (GPU, CPU or storage) and a session identifier. BlockRemoved carries only hashes. AllBlocksCleared carries no payload and is emitted when the prefix cache is reset.

They are published once per scheduler step, only if there are events. With the feature switched off the cost is zero. With it switched on, the full token identifiers of each block travel over the wire, so bandwidth scales with tokens per second and not with requests per second.

An example in vLLM’s own repository declares itself experimental on its eighth line.

The hash that has to be replicated

An exact router does not receive the engine’s state, it reconstructs it. For that it has to compute the same hashes, and the engine computes them like this:

# vllm/v1/core/kv_cache_utils.py
def hash_block_tokens(hash_function, parent_block_hash, curr_block_token_ids, extra_keys):
 if not parent_block_hash:
 parent_block_hash = NONE_HASH
 return hash_function((parent_block_hash, tuple(curr_block_token_ids), extra_keys))

It is chained: each hash identifies the complete prefix up to that boundary, not the isolated block. The default algorithm is SHA256, configurable with --prefix-caching-hash-algo among four values.

Five things a router has to get right for its hashes to match the engine’s:

The seed. Since 0.29, NONE_HASH is deterministic from a fixed seed, so two different processes produce the same hashes with nothing configured. It is a silent change of semantics from earlier versions, which required setting PYTHONHASHSEED. With the xxhash algorithms it is still needed.

The block size. By default 16 tokens, but the attention backend can override it if the user did not pass --block-size: 256 with sliding-window sparse attention, 64 with ROCm’s AITER, 64 or more with FlashAttention on XPU. A router must not assume 16, it has to read it from the event.

The hashing granularity, which is not necessarily the physical block size. There is --prefix-match-unit, which with several cache groups resolves to the greatest common divisor of the cacheable groups’ sizes.

The extra keys, in their order: LoRA adapter name (by name, not by identifier), multimodal identifiers with their relative offset inside the block, cache_salt only in block zero, and the hash of the prompt embeddings. Sampling parameters do not go in, which is correct: the prefix KV does not depend on the temperature.

And truncation. By default the hashes travel over the wire as 64-bit integers truncated from a SHA256, not as the full bytes. It can be disabled with VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=0. With the default value, a router has a theoretical collision risk that the engine does not have.

The hits are invisible by default

This is the detail that takes the most effort to discover and has the most consequences. The events are incremental: only what gets newly cached is announced. A reused block generates no event.

A router that only listens sees the additions and the removals, but does not see the hits. Its notion of what is recent and what is cold drifts away from the engine’s real LRU with nothing to warn it.

A full mode exists, and it is per request. sampling_params.extra_args["kv_cache_report_mode"] = "full" makes BlockStored events be emitted for blocks that were hits too. The function’s docstring says so plainly: it generates events so that external consumers, “for example a gateway”, know which blocks were reused. That is to say: for this to work, the router has to inject a parameter into every request it forwards.

The metrics, and what broke

For aggregate measurement two counters remain: vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, both in tokens and not in requests, with model and engine labels. There are two equivalents for the external connector, which allow a hit in HBM to be distinguished from one brought in from CPU or NVMe.

And there are two removals that break existing configurations. vllm:gpu_prefix_cache_hit_rate no longer exists: zero appearances in the repository. It was deprecated in 0.8, hidden in 0.9 and removed in 0.10. And vllm:gpu_cache_usage_perc was renamed to vllm:kv_cache_usage_perc, which breaks Grafana panels and autoscalers that depended on the old name.

The hit rate has to be derived in PromQL with rate() over the two counters. The one that appears in the text log is something else: it uses a window of the last thousand requests.

And it has a bias you need to know about before taking decisions with it. When a request is evicted, its blocks return to the free queue keeping the hash, so the later reprefill finds them again and counts as a hit. A high hit rate can mean useful reuse between users or it can mean the engine is evicting and recomputing in a loop. The internal statistics separate the two things with a preempted field, which is not exposed as a Prometheus metric.

The last token

A small detail with an effect on long conversations: the maximum hit length is the request’s token count minus one. There is always something to recompute in order to get logits. Since the reservation requires alignment to the block size, in practice that can force recomputing the whole final block. With blocks of 16 it is noise. With a backend that prefers 256, it is up to 256 tokens recomputed per turn.

The ones that do do it

vLLM production-stack

Eight routing logics in a single --routing-logic selector: roundrobin, session, kvaware, loadaware, prefixaware, disaggregated_prefill, disaggregated_prefill_orchestrated and priority. Two of them, loadaware and priority, are not in the Helm chart’s enum and are only enabled through extraArgs.

The naming trap is that prefixaware and kvaware are radically different things.

prefixaware is a hash trie in the router’s process. It chunks the prompt string every 128 characters, not tokens, and hashes each chunk with 64-bit xxhash. Its own docstring says so: the chunk size is “in number of characters”. That 128 is hardcoded, there is no flag or Helm value to change it.

Four consequences come out of that. --prefix-min-match-length is measured in characters and is quantised to multiples of 128. A shared prefix that differs by one character inside the first chunk breaks the whole match, because there is no partial match inside a chunk. For chat, it concatenates the content of all the messages with newlines and without applying the chat template, so the indexed string is not what the engine tokenises. And the tie-break between tied replicas is random.choice, without looking at load.

The trie has no notion of saturation. A popular prefix concentrates all the traffic on the same replica indefinitely. The docstring assumes the simplified model and declares it: “we assume there is no prefix cache eviction”.

It has no pruning either. HashTrie only has insertion and longest prefix lookup; there is no deletion method. When a replica disappears, its URLs stay forever in every node’s sets, and filtering happens at query time. It works, and the trie grows monotonically: each distinct prompt adds its length divided by 128 permanent nodes, with no bound, no TTL and no LRU. The router’s memory limit in the chart is 1,000 MiB. By contrast, the round-robin router in the same file does bound its caches to 1,024 entries.

And it dies on every restart: it is in-process memory state, with no snapshot and no initial load.

kvaware and loadaware are better and have a different price. They do not index themselves: they ask the LMCache controller, which knows about real tokens. That requires lmcache==0.3.11 and vllm==0.13.0 pinned exactly, and the official router image drags in the whole of vLLM. The limitation to read before planning anything: they only look at request_json.get("prompt", ""), with an open TODO for chat completions. With /v1/chat/completions, both tokenise an empty string.

loadaware is the only one of the three that breaks affinity on saturation, with the formula score = relative_match - beta * relative_load and beta = 1.0 by default. The authors’ own reading: a replica at twice the average load loses the equivalent of a full cache hit. The guide suggests 0.25 to favour cache and 2.0 to favour balancing.

On published results, the documentation’s benchmarking file literally says that the measurement platform is coming soon. The project blog’s two posts claim between three and ten times lower response latency, and only one declares a partial configuration: Llama 3.1 70B on four nodes with tensor parallelism 2 over 80 GB A100s, inputs of 9K tokens and outputs of 10, against AIBrix 0.2.0 and a flat deployment. Charts with no labelled points, no absolute figures and no declaration of which routing logic was used in each.

llm-d

There has been a reorganisation here that invalidates a good part of the third-party documentation. The Endpoint Picker code left the Gateway API Inference Extension repository and was merged into llm-d’s, which has been renamed to llm-d Router. The Kubernetes project keeps the InferencePool API and a lightweight reference EPP that does nothing about prefixes. If you look for KV-aware routing in the extension’s repository, today it is not there.

The InferencePool is at v1 and is GA. The other two APIs moved to the llm-d.ai group at v1alpha2, and when they coexist with the old ones the EPP prefers the new ones and ignores the old.

Scoring is a weighted sum. The default weights, when no configuration is passed: queue 2, KV utilisation 2, prefix 3. Any scorer referenced without a weight counts as 1.0. There is no configuration through environment variables: the old ENABLE_*_SCORER no longer exist, everything goes in an EndpointPickerConfig in YAML.

The prefix scorer consumes an attribute published by a producer, and there are two producers.

The approximate one computes the hashes in the EPP itself without talking to the engine, and keeps a local LRU per pod and block pair that is populated after the decision. It is an approximation by construction. It has a hard floor: a configured block size below 64 tokens is raised to 64 at request time, because the index stores one entry per pod and block and going down to 16 would multiply memory by four.

The exact one builds the real index from the engine’s ZMQ events, with one subscriber per pod that is installed and torn down with the endpoint’s registration and deregistration. It has adapters for vLLM and SGLang, deduplicates evictions by reference counting, and the replay buffer that allows the index to be rebuilt after an EPP restart requires vLLM 0.26.0 or above.

The trap is in the tokeniser. The EPP tokenises through a plugin, and the default backend is estimate: byte packing with no tokeniser, roughly 4 bytes per token, with identifiers that do not correspond to real engine tokens. The exact scorer needs real tokens, and for that you have to explicitly configure the vllm backend, which does HTTP against the engine’s /v1/completions/render endpoints. If it is omitted, the auto-created estimate satisfies the dependency and the system degrades silently.

On maturity, the file itself declares it: all in-tree plugins are in alpha or beta, and they will be promoted to stable when the project approaches 1.0. An alpha plugin configured without the corresponding flag makes the EPP fail to start.

And on replicas, the warning is explicit: active-active mode should be avoided with approximate prefix routing, because the EPP replicas do not share state and each one only sees the traffic it has served itself, which significantly degrades the hit rate. The exact index is safe in high availability, and even so the reference guides fix a single replica because the accounting of in-flight requests is local to the process.

The cost is the best documented in the whole field, and it deserves a table. Simulator with Qwen3 8B, 100 pods, 100K input tokens:

Requests/s	Cap on tokens to match	Peak CPU	Peak memory
5.0	4,096	1.19 cores	0.26 GiB
5.0	100,000	3.82 cores	0.65 GiB
98.7	4,096	35.17 cores	2.46 GiB
98.8	100,000	46.50 cores	3.41 GiB

With outputs of 10K tokens at 50 requests per second, 32.53 cores and 12.54 GiB. The maintainers’ sizing rule is half a core to one core per request per second on large agentic loads. And idle consumption scales with the number of pods because of continuous scraping: with 100 pods, about 7.5 cores with no traffic.

NVIDIA Dynamo

A precise event-driven index, with a global radix tree in each frontend’s process. Since all the replicas consume the same event plane, there is no need to synchronise routers with each other for the prefix state.

The cost function is explicit and weighted. It discounts from the prefill the blocks that overlap in GPU, host and disk with different weights (1.0 for the overlap credit, 0.75 for host, 0.25 for disk), and adds the potential decode blocks and the active requests.

There is a parameter to look at before deploying: --router-kv-overlap-score-credit-decay is at 0 by default, that is, disabled. Its description says what it is for: preventing loaded, cache-rich replicas from winning again and again while the freshly autoscaled ones receive too little traffic. With the default value, a new pod has no cache, therefore never wins the scoring, therefore never generates cache.

The approximate mode exists (--no-router-kv-events, with a TTL of 120 seconds) and the documentation says it is not the recommended route in production.

Dynamo is the most honest project in its measurement methodology: its comparison guide recommends contrasting --router-mode random against kv over the public Mooncake trace from FAST'25, with a declared cache ratio of 59 %, and it generates the table locally instead of publishing a number.

The rest of the field

SGLang has renamed its router to sgl-model-gateway, version 0.3.2 within SGLang 0.5.19. Its cache_aware policy is an approximate radix tree over raw characters, with switching to shortest queue when the system is unbalanced according to two combined thresholds. One detail that can cost an afternoon: the Rust defaults and the Python CLI defaults do not match, and on the maximum tree size they differ by four orders of magnitude (10,000 nodes against 2^26).

AIBrix 0.7.0 offers a standard mode with a local hash table and a KV event synchronisation mode behind a flag and a build tag, with vLLM 0.7.0 or above.

Ray Serve LLM has PrefixCacheAffinityRouter with an explicit alpha API warning, a character-based prefix tree in a decoupled actor, and its imbalance threshold at infinity by default, that is, out of the box it never breaks affinity on load.

Envoy AI Gateway 1.1.0 implements none of this and delegates to InferencePool and the EPP. KServe the same, with its LLMInferenceService at v1alpha1. Mooncake Conductor is still an unimplemented proposal.

Comparison table

Project	Version and date	Index	Where it lives	Kubernetes	Declared maturity
production-stack `prefixaware`	main branch, 09-Sep-2026	Approximate, character trie, no eviction	Router process	Optional	WIP in the README
production-stack `kvaware`/`loadaware`	same	Exact via LMCache controller	LMCache controller	Optional	No chat completions
llm-d Router approximate	v0.10.0, 17-Aug-2026	Approximate, LRU per pod	EPP process	Yes	Beta, active-active discouraged
llm-d Router exact	same	Exact through ZMQ events	EPP process	Yes	Beta
NVIDIA Dynamo	v1.4.2, 27-Aug-2026	Exact, event-driven radix tree	Frontend process	No	GA with experimental subsystems
SGLang `sgl-model-gateway`	0.3.2, 03-Sep-2026	Approximate, character radix	Gateway process	No	Default policy
AIBrix	v0.7.0, 16-Jun-2026	Both	Gateway plugin	Yes	Synchronisation behind a flag
Ray Serve LLM	Ray 2.58	Approximate, characters	Ray actor	No	Declared alpha
Envoy AI Gateway	v1.1.0, 21-Aug-2026	None of its own	Delegates to EPP	Yes	Stable 1.x API

How much it really improves

Here two types of source have to be separated.

The projects’ own numbers. Red Hat published the llm-d case in May: Qwen3 32B, eight vLLM pods over 16 H100s with tensor parallelism 2, synthetic shared-prefix load with 150 groups of five prompts, 6,000 tokens of system prompt, 1,200 of question and 1,000 of output. Result: up to 109 % more throughput and up to 99 % less TTFT, around 200 concurrent users within SLO against around 20. The repository’s result files give more detail: peak throughput from 6,986 to 14,892 output tokens per second, p90 TTFT from 135.5 seconds to 0.26.

That same file publishes the regression, and that is why it is worth reading: p50 inter-token latency rises by 22.4 % with vLLM and by 45.4 % with SGLang. The trade-off is declared: routing by affinity concentrates more concurrent work on the pods with a warm cache.

It is the best possible case. A synthetic shared prefix and a reference that is a flat Kubernetes Service.

The independent measurements. There are three and they say different things.

CacheRoute, from Meta, August 2026, is the most complete. Llama 3.3 70B in fp8, 30 targets with tensor parallelism 2 over 60 H100s, a semi-synthetic trace from production telemetry with 128,824 business keys, and all the alternatives reimplemented in the same harness instead of comparing against a dumb balancer. It obtains 176 requests per second with a p99 SLO of 3.5 seconds against 76 for the best competitor, and a hit rate of 93.2 % against 72.0 %.

And it publishes two negative findings that nobody else publishes. There are load regimes where affinity reduces capacity to between half and two thirds of the reference, to the point that the authors make a mirrored-traffic trial mandatory before deploying. And growing the fleet can cool a prefix even though total cache capacity grows, because the time between revisits stretches beyond the eviction window. Scaling horizontally can make the hit rate worse.

GORGO, June 2026, measures on a multi-region deployment with a dataset where intra-user prefix reuse is 89.4 % and the average prompt is almost 18,000 tokens. An improvement of 6.9 to 15.5 % in p95 TTFT. With extremely high reuse, the improvement is single-digit, because network latency and queuing dominate.

LAAR, from IBM Research Tokyo, April 2026, is the uncomfortable counterpoint: in its comparison, session affinity routing was the worst of the baselines, and at 64K contexts a load-based routing gives lower absolute latency at the cost of fewer hits.

The joint reading fits in three conditions. The big improvements require high, concentrated prefix reuse, long prompts, and a weak reference. Take away any of the three and the numbers deflate.

The five failure modes

Hot spot from affinity. The busiest bucket inherits the load’s skew, and tail latency starts following the hottest target instead of the fleet average. Mitigations exist in the four main products and in three of the four they ship switched off: Dynamo’s credit decay at 0, Ray’s imbalance threshold at infinity, and production-stack’s prefixaware with none at all.

Starvation of the freshly scaled pod. A new pod has no cache, so it never wins the scoring, so it never warms up. Dynamo documents it as the reason for the decay parameter. I found no primary documentation from any of the projects about the specific interaction with KEDA.

Cost of the index. The llm-d numbers in the previous table are the best public reference that exists. Raising the cap on tokens to match from 16,384 to 400,000 can more than double the EPP’s CPU with little traffic.

Traffic with unique prefixes. Each project degrades to something different. SGLang routes to the smallest tree, which is not the same as the least loaded. Ray falls back to power of two choices. Dynamo has an explicit preset, --load-aware, to say “I want the load model without reuse”. And production-stack’s prefixaware keeps inserting into the trie, that is, keeps paying the cost while receiving nothing.

Sibling bursts. Parallel sampling, best of N, or an agent that opens five branches at once. With event-driven routing, no engine has yet emitted the block stored event, all the siblings score zero overlap, and the prefix is precomputed on every replica. Dynamo has a prediction TTL for this; the others do not.

The architecture that comes out of all of the above

Two layers, with the responsibilities separated.

On top, LiteLLM, with what it does do well: non-forgeable identity through the virtual key, budgets, spend logging, concurrency limits, fallback between providers and API unification. For the gateway, each model group has a single destination: the URL of the router below.

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://kv-router.inferencia.svc:8000/v1
 api_key: os.environ/VLLM_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 1
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

Below, the KV-aware router, inside the engine’s domain, talking to the vLLM replicas and consuming their events.

Four consequences of this separation that have to be accepted before building it.

Per-replica visibility in LiteLLM is lost. For the gateway there is a single deployment, so its cooldowns, its per-deployment counters and its strategies stop meaning anything. Observability of which replica served what goes downstairs.

Retries stack up. The client SDK retries, LiteLLM retries, and the router below may have its own failover. In the previous article 45 calls per turn came out with three layers; with four it is worse. Lowering num_retries in the gateway to 1, or to 0, is the sensible decision when there is a router below that already retries.

LiteLLM’s session affinity stops making sense for the KV, because it no longer chooses a replica. It is kept if it is used for something else, such as pinning a conversation to a specific provider.

And the choice of the router below conditions the rest of the cluster far more than the gateway’s does. llm-d requires Kubernetes 1.32 or above, Gateway API, the inference extension and a conformant Gateway. Dynamo does not require Kubernetes. production-stack has a Helm chart and works outside it. That decision is the expensive one, not the gateway’s.

When not to build this

Three cases where the correct answer is not to do it, at least not yet.

A single replica per model. With one replica, the engine’s prefix cache already does all the work and there is nothing to route. That is the case for a good part of the four H100 deployments with one large model and tensor parallelism.

Low or scattered prefix reuse. Before building anything, the measurement is rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) per replica, with a mental correction for the bias from reprefills after eviction. If it is already high with blind distribution, reuse is concentrated inside each conversation, and there LiteLLM’s session affinity costs one line of configuration and resolves the case.

Traffic where the network dominates. The GORGO result is the warning: with enormous prompts and a distributed deployment, the improvement is eaten up by queuing and network latency.

The case where it does pay off is specific: several replicas of the same model, a large system prompt shared between different users, or a RAG with a corpus of documents that repeat across requests. There the prefill saved is real and it is most of the work.

Checklist

Measure the current hit rate per replica with the two vLLM counters before touching anything, and check whether the engine is evicting with vllm:num_preemptions_total.
Check that prefix caching is switched on. It is no longer --enable-prefix-caching that switches it on: the default value is derived from the model and the log of the decision is at DEBUG level. What is actionable today is --no-enable-prefix-caching to switch it off.
Audit the prompts before routing: a timestamp or a session identifier at the start of the system prompt breaks the match from block zero and no routing layer fixes it.
If the KV event stream is enabled, size the bandwidth by tokens per second and watch discarding at the high water mark. With data parallelism, subscribe to one port per rank.
If an exact router is chosen, verify the vLLM version against the one the replay buffer requires.
If llm-d is chosen, explicitly configure the vllm tokenisation backend. The default estimate does not fail, it degrades.
If production-stack is chosen with kvaware or loadaware, check first whether the traffic goes through /v1/chat/completions, because today it does not work there.
Review the load-based affinity-breaking parameter, because in three of the four main projects it ships disabled.
Lower num_retries in LiteLLM when there is a router below that already does failover.
Trial with mirrored traffic before changing production. It is the explicit recommendation of the only independent measurement that went looking for bad regimes, and found them.

Traps and things that are not what they seem

LiteLLM has no prefix routing in any version. Neither the open one nor the paid one. I got it wrong in June.
prompt_caching does nothing if the client does not send cache_control, and with the default automatic injection it fails on every turn because it marks the last message.
That pin’s TTL is 300 hardcoded seconds, incompatible with the one-hour caches that LiteLLM itself knows how to request.
production-stack’s prefixaware indexes characters, not tokens, in non-configurable chunks of 128, and its trie is never purged and models no eviction.
production-stack’s kvaware and loadaware do not support chat completions. They tokenise an empty string.
The default tokeniser in llm-d’s EPP is a byte estimate, and with it the exact scorer degrades silently.
llm-d’s active-active mode is discouraged with approximate prefixes, and the reference guides fix one replica even with the exact one.
Dynamo’s overlap credit decay ships at zero, which can leave a freshly autoscaled replica with no traffic.
The SGLang router’s default values differ between the Rust and the Python CLI, by as much as four orders of magnitude.
vllm:gpu_prefix_cache_hit_rate no longer exists and vllm:gpu_cache_usage_perc is now called vllm:kv_cache_usage_perc. Old panels and autoscalers are measuring nothing.
The hit rate counts the reprefill after eviction as a hit. A high number can be reuse or it can be thrashing.
KV events do not publish the hits by default, only the additions, unless full mode is requested request by request.
The hashes travel truncated to 64 bits unless it is disabled with an environment variable.
The default block size is 16, and the attention backend can change it without anyone asking.
The Endpoint Picker left the Gateway API Inference Extension repository. Looking for it there today leads nowhere.

Closing

June’s error has a reading that goes beyond the erratum. It came from reading a product’s documentation and not its code, and from the fact that LiteLLM’s documentation uses the word “caching” in a pre-call check that does something else. The distance between what a name suggests and what a function does is, at this point in the track, the most repeated pattern of the whole series.

The architectural conclusion is that KV-aware routing is not a gateway function. It is not one today in LiteLLM and probably should not be: it requires keeping an index synchronised with each engine’s memory, replicating its hash function, consuming an event stream and sizing CPU per request per second. That belongs to the engine’s domain, not to that of identity and budget.

What remains is a two-part decision. The first is whether the traffic deserves it, and that is answered with two Prometheus counters and without installing anything. The second is which of the three routers fits the cluster you already have, and there the deciding variable is not the hit rate but whether you are willing to bring in Gateway API and the inference extension, or to pin vLLM 0.13.0, or to operate a Dynamo frontend.

And the honesty the material demands: the only independent measurement that specifically went looking for regimes where this makes things worse found them, and they are not marginal. Half to two thirds of reference capacity is a serious regression. That no project publishes that side of the curve does not mean it does not exist in theirs.

Sources

LiteLLM, Router - Load Balancing (estrategias, comprobaciones previas, afinidad): https://docs.litellm.ai/docs/routing.
LiteLLM, código: litellm/router.py, litellm/router_utils/prompt_caching_cache.py, litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/litellm_pre_call_utils.py, litellm/proxy/middleware/admission_control_middleware.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
vLLM, código: vllm/v1/core/kv_cache_utils.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_manager.py, vllm/distributed/kv_events.py, vllm/config/kv_events.py, vllm/v1/metrics/loggers.py, vllm/v1/metrics/stats.py: https://github.com/vllm-project/vllm.
vLLM, KV offloading usage (niveles CPU, filesystem y objeto; compartición entre procesos): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html.
vLLM, notas de la versión 0.29.0 (semilla determinista, seguimiento de propiedad en eventos, flags de admisión): https://github.com/vllm-project/vllm/releases/tag/v0.29.0.
vLLM, RFC 16669, publicación de bloques KV y métricas: https://github.com/vllm-project/vllm/issues/16669.
vLLM production-stack, código: src/vllm_router/routers/routing_logic.py, src/vllm_router/prefix/hashtrie.py, src/vllm_router/parsers/parser.py: https://github.com/vllm-project/production-stack.
LMCache Lab, resultados de production-stack frente a AIBrix (parte interesada): https://blog.lmcache.ai/en/2025/03/06/open-source-llm-inference-cluster-performing-10x-faster-than-sota-oss-solution/.
llm-d Router, código y documentación de operaciones (pesos por defecto, dimensionamiento del EPP, aviso de activo-activo): https://github.com/llm-d/llm-d-router.
llm-d, guía de enrutado preciso por prefijo y resultados publicados (parte interesada): https://github.com/llm-d/llm-d.
llm-d Router, incidencia 1290, estado de prefijo no compartido entre réplicas del EPP: https://github.com/llm-d/llm-d-router/issues/1290.
Red Hat, Same 16 GPUs, twice the users (parte interesada): https://www.redhat.com/en/blog/same-16-gpus-twice-users-inference-aware-routing-llm-clusters.
Kubernetes SIG Network, Gateway API Inference Extension (traslado del EPP y del BBR): https://github.com/kubernetes-sigs/gateway-api-inference-extension.
NVIDIA Dynamo, diseño y ajuste del router KV: https://github.com/ai-dynamo/dynamo.
SGLang, sgl-model-gateway: https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway.
AIBrix, enrutado por caché de prefijo: https://github.com/vllm-project/aibrix/blob/main/pkg/plugins/gateway/algorithms/prefix_cache_readme.md.
Ray Serve LLM, enrutado consciente de prefijo (API en alfa): https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html.
Huang Cheng (Meta), CacheRoute, agosto de 2026: https://arxiv.org/html/2608.19677.
Yuan et al., DualMap, febrero de 2026: https://arxiv.org/abs/2602.06502.
Ricci Toniolo et al., GORGO, junio de 2026: https://arxiv.org/html/2602.11688.
Yoshimura, Chiba y van de Beek, Accuracy Is Speed (EuroMLSys ‘26), abril de 2026: https://arxiv.org/html/2604.15732v1.

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

Thu, 10 Sep 2026 08:00:00 +0200

Fourth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability and virtual keys covered governance. Here comes the new tenant: agents, which speak the same protocol as humans and behave in a different way. Everything that follows is verified against LiteLLM 1.100.1 and vLLM 0.29.0, both from the first ten days of September 2026.

TL;DR

The day a team connects an agent to the corporate gateway, the platform changes regime without anyone touching a YAML. These are the seven conclusions.

LiteLLM’s priority parameter prioritises nothing under normal conditions. It is still marked as beta and its queue is only consulted when all the deployments in the group are in cooldown. With free capacity, the only thing it produces is a response header. On top of that, the success branch does not pop the item off the queue, so the list grows indefinitely; and on /chat/completions the value 0, which is the one used by all the documentation examples, is falsy in Python and disables queuing.

Separating traffic is done with two pools, not with priorities. The non-forgeable way is model_group_alias at key or team level: the client always asks for qwen-30b and the credential decides whether that resolves to qwen-30b-interactivo or to qwen-30b-agentes. The way that does not touch clients is tag_regex against the User-Agent, with a pattern like ^User-Agent: claude-code\/, which serves to classify load and not to control access, because that header is set by the client.

The useful admission control arrives in the 1.101 branch, with max_in_flight_requests_per_worker and a per-process queue that returns 503 with retry-after. It rejects before authenticating, so what it discards does not appear attributed to any key.

max_parallel_requests is only applied at virtual key level. The team, user, end customer and organisation endpoints accept the field and store it in the database, and nobody reads it. global_max_parallel_requests is a no-op unless the old limiter is reactivated, and general_settings.max_parallel_requests is read by no code path.

vLLM with --scheduling-policy priority does not push ahead anyone who is already generating. The only thing that changes is who it evicts when KV cache runs short. If the agents have filled max_num_seqs, a chat with the highest priority waits just the same. And the two knobs that in 2025 allowed short prompts to get ahead of long ones, max_num_partial_prefills and max_long_partial_prefills, disappeared from the code in 0.27.

The retry product reaches 45 calls per agent turn. Three from the client SDK times three from the router times up to five fallback groups. The proxy sets max_retries=0 on the SDK it uses itself, but it cannot touch the client’s. The timeout is per attempt, not per request, and LiteLLM’s Timeout comes out as a 408, which is retryable.

A single 429 takes a deployment out of the pool for five seconds, without going through the allowed_fails threshold. With an agentic burst against a fleet of four replicas, the whole pool enters cooldown and the gateway returns a 429 of its own to everybody, humans included.

You are here: the gateway layer with two classes of traffic

In the seven-layer stack this still lives in the gateway layer, with one difference compared to the previous three articles. There the variable was load: more requests, more teams, more budget to share out. Here the variable is the shape of the load.

A gateway sized for three hundred engineers using a chat works. The same gateway, with thirty of those engineers running coding agents, serves a fraction of the human traffic with the previous latency and the rest degrades. The GPUs stay just as busy, the utilisation metrics stay good, and the support team gets complaints from people waiting nine seconds for a paragraph.

The analogy: the registry desk and the courier with two hundred files

A registry office serves people in order of arrival. Citizens come in one at a time, present a piece of paper, and leave. The system is fair and the queue moves.

One Tuesday a courier turns up with a trolley of two hundred files. He joins the queue like anyone else, because rules are rules, and when his turn comes he occupies the desk for forty minutes. Behind him are twelve people with one piece of paper each. Nobody has broken any rule. The courier has the same right as everyone else and uses the same procedure. What fails is that the office was designed assuming a certain size of transaction and another one three orders of magnitude bigger has turned up.

The real solutions for an office are three, and all three have an exact equivalent in the platform. Open a desk for bulk deliveries, which is separating pools. Put up a capacity sign and send people back later, which is admission control with its 503. And say that files are handed in during the afternoon, which is the part that on an inference platform can almost never be applied, because the agent does not negotiate schedules.

What does not work in the office is giving the citizen a priority number when the courier is already at the desk. That number does not interrupt the transaction in progress. It is, literally, what priority does in the two layers of the platform, and it is worth keeping in mind while reading the next section.

The two kinds of traffic, measured

Before configuring anything, you have to be able to state the difference in numbers, because every later adjustment comes out of it.

Trait	Human chat	Agentic loop
Input per request	hundreds of tokens	tens of thousands, and it grows with each turn
Output per request	hundreds of tokens	from two tokens (a tool call) to several thousand
Concurrency per user	1	3 to 20 requests in flight
The metric that hurts	time to first token	total turn throughput
Tolerance for waiting	seconds	minutes
Cancelled requests	rare	routine, by the agent’s own decision
Retries	the browser’s	the SDK’s, plus the loop’s
Witness when it fails	the user	nobody

Two rows deserve comment because they contradict intuition.

The first is the metric row. An agent does not perceive time to first token, so on a shared engine it is the natural candidate to give up that metric. The operational conclusion is that the agent pool can be configured for aggressive throughput, with large batches and deep queues, while the interactive pool is configured for the opposite.

The second is the cancellations row. An agent that launches three speculative continuations and discards two generates two aborted streams per turn. At the gateway, that touches spend accounting, the release of concurrency counters and the open connection to the engine. LiteLLM 1.100 handles it reasonably well, and it did not six months ago.

What does not work: `priority` in LiteLLM

The first reaction of any operator is to look for a priority field. LiteLLM has one, documents it on its Request Prioritization page, and that page carries the beta label and the sentence saying it is for testing. The label is correct and has to be taken literally, because behind it there are four things.

The queue is only consulted when the whole group is in cooldown. The scheduler’s poll() method returns true if there is any healthy deployment, without touching the queue. With available capacity, which is the normal situation, the request goes straight through and priority orders nothing. The only thing that changes is that the response carries the x-litellm-request-prioritization-used header.

The queue does not empty. In that success branch there is no heappop and no call to the cleanup, which only happens in the expiry branch. The list under the key scheduler:queue:{model} grows without bound, is serialised whole into Redis on every insertion, and there is no maximum size and no discard policy. When the group does enter cooldown, the head of the heap is occupied by fossil entries from requests that finished hours ago, and new ones compare their identifier against that head, never match, and wait until the timeout runs out.

On /chat/completions, the highest priority disables the feature. The code reads the value with a kwargs.get("priority") or self.default_priority, and in Python 0 or None is None. Since in this scheduler the lowest number is the highest priority, priority: 0, which is the value in the documentation’s own examples, goes down the path without queuing. On /completions the check is done with is not None and it works; the two routes do not behave the same.

And default_priority in router_settings breaks the request. If the client does not send a priority but the global setting is in place, the value is truthy, the scheduling branch is entered and a function is invoked that requires the positional argument nobody has injected. The setting has no associated test and does not appear in the router_settings reference.

There is also a /queue/chat/completions, marked as experimental and hidden from the OpenAPI schema, which reads data["priority"] with no default value: a request without that field raises an exception that comes out as a 400 authentication error, with a message that has no relation to the cause.

There is a priority feature that is solid, and it is a different thing. The v3 dynamic limiter shares out token and request per minute quota between classes with weights, through litellm.priority_reservation with values like {"premium": 0.75, "standard": 0.25}. It shares out capacity, it does not order a queue: below the saturation threshold it lends free capacity and above it applies the weights. It never makes anyone wait, either it passes or it returns a 429. It requires Postgres and an enterprise licence, and it is still marked as beta.

What does work: two pools

Effective separation is done earlier, by deciding which set of replicas each class of traffic goes to. LiteLLM offers two mechanisms with very different security properties.

By credential: `model_group_alias` at key or team level

Router settings are resolved in the order key, team, global. Among those that accept that override is model_group_alias, and there lies the piece:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 model_info:
 id: chat-01
 max_input_tokens: 32768
 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 model_info:
 id: agentes-01
 max_input_tokens: 131072

And on the agent team’s key:

curl -X POST "$PROXY/key/generate" \
 -H "Authorization: Bearer $MASTER_KEY" \
 -d '{
 "team_id": "plataforma-agentes",
 "models": ["qwen-30b"],
 "router_settings": {
 "model_group_alias": {"qwen-30b": "qwen-30b-agentes"}
 }
 }'

The client asks for qwen-30b and knows nothing about any of it. The resolution is tied to the credential, so the caller cannot get around it by changing a header. The same list allows routing_strategy, fallbacks, context_window_fallbacks, retry_policy, cooldown_time and allowed_fails to be set per key or team, which gives different reliability policies for each class of traffic on the same proxy instance.

It should not be confused with access groups (model_info.access_groups), which are permissions and not routing: they decide whether a key can invoke a model name, not which replica the request goes to.

By `User-Agent`: `tag_regex`

When separate credentials cannot be issued, because the agentic client uses the same personal key as that person’s chat, LiteLLM classifies by regular expression against the request headers:

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 tag_regex: ["^User-Agent: claude-code\\/", "^User-Agent: .*codex"]
 model_info: {id: agentes-01}
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 tags: ["default"]
 model_info: {id: chat-01}

router_settings:
 enable_tag_filtering: true
 tag_filtering_match_any: true

The proxy composes strings of the form User-Agent: <value> and tests them with re.search. Exact tag matching takes precedence over the regex. The documentation itself carries a warning that has to be respected: the User-Agent is written by the client, so this is traffic classification and not a security boundary. As load sharing it is enough, because an agent that lies about its identity to sneak into the interactive pool is a governance problem, not a routing one.

The behaviour when no deployment carries the requested tag has a nuance that comes as a surprise in production. If the tag is unknown to the group, the request falls through to the pool marked with tags: ["default"]. If the tag exists in the group but no healthy deployment carries it at that moment, the request fails with the tag configuration error, even though a default pool is available. To get deterministic fallthrough to the default pool you have to set model_info.allow_fail_open: true.

A different strategy per pool with `routing_groups`

router_settings:
 routing_strategy: simple-shuffle
 routing_groups:
 - group_name: interactivo
 models: [qwen-30b-interactivo]
 routing_strategy: simple-shuffle
 - group_name: agentes
 models: [qwen-30b-agentes]
 routing_strategy: simple-shuffle

The recommendation to leave simple-shuffle on both is not laziness, and sustaining it requires looking at what the alternatives do on a homogeneous vLLM fleet.

least-busy keeps a complete dictionary under a single key and updates it with non-atomic reads and writes, without using INCR, so with several proxy replicas increments are lost. It has no TTL, so a request that never completes leaves the counter raised forever and that replica is permanently excluded. Counters can go negative. And when the minimum corresponds to an identifier that is no longer in the healthy list, it falls back to random.choice without warning. The pattern of requests that do not complete is exactly that of an agent that cancels.

latency-based-routing measures seconds per output token, not wall-clock latency. A chat turn of twenty tokens looks extremely slow by that metric and an agent generation of two thousand tokens looks extremely fast, so mixing both kinds of traffic in the same group poisons the signal. It also seeds new deployments with zero latency, and with the buffer defaulting to zero, a freshly started replica takes all the traffic until its first success callback.

usage-based-routing-v2 uses minute windows that reset all at once, and it accounts for tokens when the request completes, so long generations in flight are invisible to the router. With long outputs, it systematically underestimates the real load.

cost-based-routing sorts by the sum of the input and output unit prices and takes the first, with no random tie-break. With replicas of the same model at the same price, all the traffic always goes to the same place. And a model that is not in the price map gets a default cost of 5.0 per token, a figure designed to deprioritise it.

Admission control: the new piece in the 1.101 branch

When the agent pool saturates, what you want is to reject fast and with a code the client understands, not to accumulate requests inside the proxy until memory runs out. That arrives with per-worker admission control, added in the 1.101 branch, in release candidate as this is written.

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

It is a semaphore with a per-uvicorn-process queue, without Redis. When the queue fills or the wait expires it returns 503 with retry-after: 1 and a body with "type": "overloaded_error". It is observed through /health/backlog, which exposes requests in flight, admitted, queued and rejected, and through the Prometheus litellm_admission_* metrics, whose rejection counter distinguishes queue_full from queue_timeout.

Two properties have to go into the runbook. The middleware rejects before authenticating, so what is discarded is not attributed to any key in the spend records, and the real number of requests rejected per team cannot be reconstructed from the spend table. And the limits are read on the first request: changing them requires a restart.

Concurrency per key, and the three settings that do nothing

max_parallel_requests is the right brake for a specific agent, and its configuration surface is full of dead ends.

The only level where it is applied is the virtual key. The v3 limiter, active by default since 1.94, has descriptors for user, team, team member, end customer, organisation, model per key, tag and agent, and all of them carry only request and token counters per window. The only reference to max_parallel_requests is the one in the key descriptor. The old limiter says so in a comment: supporting it for model, user and team is pending.

At the same time, /team/new, /user/new, /customer/new and the organisation endpoints accept the field, validate it and write it to Postgres, and the documentation shows it in their examples. Nobody reads it afterwards. A team with max_parallel_requests: 20 in its row has no concurrency limit at all.

global_max_parallel_requests is injected into the request metadata and is only consulted by the old limiter, which is disabled unless LEGACY_MULTI_INSTANCE_RATE_LIMITING=true is set. The server settings documentation still describes it as a global limit coordinated through Redis. And general_settings.max_parallel_requests does not appear in any execution path: its only two appearances are the copy from the database and the type schema of the interface.

What does work, in detail:

curl -X POST "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" \
 -d '{"team_id":"plataforma-agentes","max_parallel_requests":8,"rpm_limit":600}'

Counting is done with a sorted set in Redis scored by Redis’s own clock, with Lua scripts to acquire and release. Each slot expires after 3,600 seconds, which is at once the maximum request duration the meter can track and the time a leaked slot takes to heal by itself. Release is hooked to success, to failure and to client disconnection during streaming, which was the classic leak with clients that cancel.

One live leak remains and it affects exactly the case in this article: every tool call through the MCP gateway acquires a slot and does not release it. The MCP route invokes the pre-call hook, which acquires, and none of the release functions is called from that module. With an agent that makes twenty tool calls per turn and a key with eight slots, the 429 arrives on the first turn and lasts an hour.

The order of evaluation matters for interpreting 429s. First the window counters, requests and tokens per minute, are checked, and only if they pass is the concurrency slot acquired. The code comment explains why: the other way round, every rate limit rejection would leave an orphan slot. In practice, with a low rpm you always see the requests type 429, and the max_parallel_requests one only appears when the relationship between concurrency and duration makes it binding first. The message distinguishes the two cases with the Limit type: field and the rate_limit_type header.

A minor detail with consequences on the client: the response to a concurrency rejection carries retry-after and reset_at computed from the window size, sixty seconds by default, when a concurrency meter has no window. A well-behaved client that respects that header will wait a minute when it might have had a free slot in two seconds.

Session affinity: the KV cache that gets thrown away

An agent turn resends the whole conversation context, system prompt and tools included. If that request lands on a different replica from the previous turn’s, vLLM’s prefix cache does not have the prefix and the prefill is recomputed in full. With twenty turns and four replicas balanced blindly, the long prefill is paid for fifteen times over.

LiteLLM has two mechanisms for this and they are not equally good.

The prompt_caching check computes a SHA-256 of the cacheable prefix, defined as everything up to the last block with cache_control, and pins the deployment with that hash. It has two problems. The TTL is hardcoded to 300 seconds in two places in the code, with no way to configure it, something which is reported and open; with one-hour caches, affinity is lost after five minutes of inactivity. And the key is an exact hash, so a client that moves the cut point forward on every turn, which is exactly what agents do to extend the cached prefix, generates a new key every time.

The session_affinity check is the one to use:

router_settings:
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

It pins the deployment with an atomic Lua script in Redis, with a configurable TTL that defaults to one hour and that is refreshed on every request, so it bounds the idle time between turns and not the duration of the conversation. If the pinned deployment enters cooldown, the request goes out through the normal strategy and keeps the pin so it can come back when it heals.

The session identifier comes from a precedence chain that starts with x-litellm-trace-id, continues with x-litellm-session-id and then accepts any header of the form x-<something>-session-id whose value looks like an identifier. That includes the session header Claude Code already sends, with nothing configured on the client. Whatever matches populates both the session identifier and the trace identifier, which links this to the correlation from the first article in the track.

One dangerous interaction when combining it with tags: affinity is evaluated before tag-based routing. If the pin reduces the list to a deployment that does not satisfy the request’s tag, the request fails instead of falling through to the default pool, unless allow_fail_open has been enabled. Retagging a pool with live sessions breaks them.

And a clarification that saves a pointless search: LiteLLM does not do prefix routing. There is no prefix tree and nothing that directs the request to the vLLM replica that already has that prefix warm. What there is, is affinity by session or by key. For the other thing you need a specific router underneath the gateway, and that architecture has its own article pending.

The engine: what vLLM can and cannot do

Everything above shares out requests between engines. Inside one engine, the question is what happens when a twenty-token request and a sixty-thousand-token one live side by side.

Priority exists and does less than it looks like. --scheduling-policy priority turns the waiting queue into a heap ordered by priority, arrival time and identifier, with the lowest number first. The request can carry the priority field in the body, and there is also the X-Vllm-Priority header, which takes precedence over the body and is not documented anywhere.

What priority does not do is push ahead of whoever is already generating. In the running queue loop, when KV cache runs short, the policy only changes who is chosen as the victim of eviction: with priority the one with the worst priority and arrival time is evicted, and with fcfs the last in the queue. In the waiting queue loop, if there are no blocks available the iteration is cut short, without evicting anyone. And the max_num_seqs cap is a hard cut. If the agents have filled that cap, a chat with maximum priority waits just the same. There has been an open issue since April and an unmerged pull request about this.

That nuance about the victim is not negligible, because with fcfs the victims are the most recently admitted, that is, the chats that have just come in. Switching the priority policy on already improves the situation even if it pushes nobody ahead.

Eviction is always by recomputation. The swap-to-host-memory mode and the enumeration that controlled it disappeared from the code; all that is left is freeing blocks and returning the request to the queue with the computed token counter at zero. An evicted request loses all its prefill, and the prefix cache recovers part of it, but the perceived time to first token restarts.

Two admission settings help and do not appear in the narrative documentation. watermark reserves a fraction of free KV blocks on admission, with the declared purpose of avoiding repeated eviction, and it ships disabled. And scheduler_reserve_full_isl, active by default, checks that the full input length fits in the cache before admitting, instead of looking only at the first chunk.

Chunked prefill is active by default and prioritises decode. The scheduler walks the whole running request loop before touching the waiting queue, so a long agent prefill does not freeze a chat’s generation in progress. That protects the inter-token latency of what has already been admitted, and it does not protect the time to first token of what is waiting.

Here there is a breaking change that forces a review of the manifests. max_num_partial_prefills and max_long_partial_prefills were removed in 0.27. They were precisely the mechanism that allowed short prompts to get ahead of long ones. A deployment with those flags in its args does not start. long_prefill_token_threshold still exists, and its meaning changed: today it is a per-request chunk size cap, not a classification of long requests.

The partial replacement arrives in 0.29 with max_num_queued_tokens and max_num_queued_reqs, admission valves that return 503 when the pending prefill work exceeds the target. The code itself frames it as a quality of service mechanism for time to first token, with the formula of setting it to the latency target multiplied by prefill throughput. They are blind to the class of traffic: they reject a chat just the same as an agent.

And the default value of --max-num-batched-tokens depends on the hardware and on the start-up mode. With vllm serve on H100 or H200 it is 8,192, on cards of 160 GiB or more it is 16,384, and on the rest 2,048. The 2,048 that appears in the scheduler configuration only applies to tests.

The metrics for diagnosing coexistence, with the exact names from 0.29:

Metric	What for
`vllm:num_requests_waiting`	queue depth, the autoscaling signal
`vllm:num_requests_waiting_by_reason`	separates waiting for capacity from waiting in a blocked state
`vllm:num_preemptions_total`	evictions, the signal that the two classes are treading on each other
`vllm:kv_cache_usage_perc`	cache pressure, the cause of the evictions
`vllm:request_queue_time_seconds`	what a chat waits with the agents inside
`vllm:prefix_cache_hits_total` and `_queries_total`	measured in tokens, not in requests

A warning for anyone with old alerts. The log warning about evicted sequences that appears in the optimisation documentation belongs to the previous version of the engine and no longer exists in the code. Preemptions are only visible as a counter. An alert based on searching for that string in the logs has not fired for months, and not because there are no evictions.

On isolation between classes inside one engine, the short answer is that it does not exist. There are two queues and neither is per class of traffic. The conclusion, which is the one that holds up this article’s architecture: effective isolation means separate fleets, and on top of them the gateway deciding who goes to which.

Retries: the product nobody calculates

A human who gets an error sees it and decides. An agentic loop retries, and beneath it there are two more layers that also retry.

Layer	Default
Client SDK	2 retries, 3 attempts
LiteLLM router	2 retries, 3 attempts
Fallbacks	up to 5 groups

The product is about 45 calls to the engine for one agent turn, before counting the loop’s own retry. LiteLLM does what it can: when the call comes from the router, it forces max_retries=0 on the SDK it itself uses against the engine. What it cannot touch is the client’s SDK. The only layer that breaks the product is that one, and it is broken by setting max_retries=0 on the client and delegating the retry to the gateway.

The retryable errors are 408, 409, 429 and anything 500 or above. And there lies the problem with long reasoning: litellm.Timeout is a subclass of OpenAI’s timeout exception and comes out with code 408, which is retryable, so the most expensive request of all, the one that expired after ten minutes of generating, is launched again in full. Twice by the router, and as many times again by the client.

On timeouts you have to be precise because the code comment is: the configured timeout is per attempt, not per request. A request_timeout: 600 with two retries and two fallback groups gives a worst case of almost two hours before the client receives anything. The default value is still 6,000 seconds, as was already covered in the day 2 article.

The retry storm has two specific causes and both are default settings.

The first: a single 429 takes the deployment out of the pool, without going through the allowed_fails threshold. The cooldown logic has an explicit branch which, on a 429 in a group with more than one deployment, returns immediate cooldown. With an agentic burst against four replicas, all four leave the pool at once and the gateway answers everybody with its own 429 of “no deployments available”. The default cooldown is five seconds, which in a sustained burst gets renewed.

The second: backoff is zero while the pool looks healthy. The function that computes the wait returns 0 if the healthy deployment list is not empty. Exponential backoff with jitter, capped at eight seconds, only kicks in when there is nothing healthy left. Under partial degradation, the proxy hammers away without waiting and only slows down once everything has already fallen over.

On top of that, the engine’s Retry-After header is only respected if its value is between 1 and 60 seconds. A longer limit window is ignored and replaced by the proxy’s own backoff.

The settings that cut the problem off:

router_settings:
 num_retries: 1
 retry_after: 2 # real floor; the default 0 is hammering
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3 # stops a 429 taking the deployment out
 retry_policy:
 TimeoutErrorRetries: 0 # do not relaunch the expensive request
 AuthenticationErrorRetries: 0
litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false

That last setting deserves an explanation, because it is a silent way of breaking an agent. With drop_params: true, LiteLLM removes from the body any parameter the destination provider does not support, and in that branch it writes no warning at all. The candidates for disappearing are tools, tool_choice, response_format and parallel_tool_calls. An agent that has tools taken away receives free text where it expected a function call, and interprets it as the model having decided not to use tools.

Worse still is prompt-based emulation. For a non-OpenAI-compatible provider that does not support tools, LiteLLM does not fail: it turns the tools into text inside the prompt, forces JSON format, discards tool_choice, and on the way switches on a process-global variable that makes any other tool-less provider in the same process go to emulation instead of erroring.

With that in mind, the warning about fallbacks is clear: LiteLLM does not check capabilities before falling back. The function that knows whether a model supports function calling exists and is only used in an informational endpoint. The pre-call checks filter by context window, request limit and region, and nothing else. A fallback list that goes from a model with tools to one without them degrades the agent silently. There are guards for provider-scoped resources, files and batches, so the absence of the capability guard is an omission and not a design oversight.

A month’s budget in twenty minutes

A runaway agent consumes in minutes what a team consumes in weeks, and the operational question is how long the gateway takes to find out.

The answer has changed, and the old mental model, the one where spend consolidates every minute and until then there is no brake, is no longer correct. Since 1.84 there are two new pieces. An optimistic reservation before the call, which estimates the maximum cost of the request and reserves it atomically in the counter before heading out to the engine, reconciling it with the real cost on completion. And a counter increment after the call that is explicitly awaited, with the comment that this way the counter is up to date before the next request goes through authentication. Budget checks read that counter, with Redis first and seeding from the database when cold.

Batch writing to Postgres still exists, every 10 to 15 seconds, and it affects persistence and what is shown in the interface, not the cut-off. The production documentation’s recommendation to raise that interval to 60 does not relax the control, and it does delay by a minute what /key/info shows, something which frequently generates operational panic. And the budget rescheduler’s ten minutes do not consolidate spend: they fire the job that resets expired budgets.

The overspend window then comes down to the cost of the requests in flight, provided there is Redis. Without Redis, the counters are per pod, and with N replicas you can spend up to N times the budget. That is the configuration failure that matters and it is the same Redis that the day 2 article declared non-optional for other reasons.

Two settings that do not appear in the public documentation, only in the data model descriptions, and that decide behaviour at the worst moment:

general_settings:
 fail_closed_budget_enforcement: true # false by default: if the counter fails, it lets things through
 # disable_budget_reservation: do not set it to true

With the default value, if the counter backend does not respond, the proxy fails open and carries on serving. For an agent in a loop that is exactly the opposite of what you want.

Time for the honest warning: this route is still unstable. There are open issues in both directions, one where the key and user budget limit stops being applied while spend is recorded, and another where a 429 is returned for stale spend in the reservation counters while the key query shows spend below the limit. Project budgets are not on the atomic reservation route and concurrency overshoots them. The practical conclusion is not to trust the hard cut-off to the budget alone.

On the manual cut-off, there is an asymmetry to know about before you need it. /key/block invalidates the local cache, the Redis one, and notifies all workers by publish, so with Redis the effect is almost immediate. /customer/block does the same. /team/block only writes to the database, without invalidating or notifying, so it takes however long the management object cache TTL takes, sixty seconds by default. To cut a team off dead you have to block its keys one by one, or lower user_api_key_cache_ttl. And /user/block no longer exists, even though the docstring of the function that replaced it still shows that route in its example.

On alerts, two numbers. The default thresholds warn at 85 % and 95 % of the budget. And deduplication of those alerts is 24 hours per event and identifier, so crossing 85 % generates one warning and then silence for a day. For agentic traffic, where 85 % and 100 % can be twenty minutes apart, the second warning arrives when it is no longer any use.

Attribution: who the user is when the caller is an agent

With agents, the question of who the spend is charged to has two answers and both are needed: the agent that executes and the human on whose behalf it acts.

The chain that extracts the end customer identifier tries the standard headers first, then the configurable header mapping, and only then the user field of the body, the litellm_metadata.user of the Anthropic format, the metadata.user_id and the safety_identifier. The last four are body fields and are therefore written by the client. The code itself warns about it: for untrusted callers the identifier has to be set from headers or from a middleware on the server, to avoid impersonation.

Out of that comes the correct pattern, which is the inverse of the one people put together out of inertia: the virtual key identifies the agent and is the non-forgeable part, and the header or the user field identify the human and serve for attribution, not for control. An agent that lies about the human behind it is still bounded by the budget and the limits of its key.

The spend table stores user, end_user, team_id, organization_id, request_tags, session_id and agent_id, with daily aggregation per agent, so the detail is available once it is sent. The x-litellm-agent-id header is what populates that last field.

A note about the MCP gateway, which on a platform with agents ends up being the other entry door. Tool calls do appear in the spend records, with call types of their own and a metadata block that stores tool name, arguments, result and server. What is missing is token counting: the cost is 0.0 by default and is only populated if the administrator configures a price per query, per server or per tool. It is the same pattern of silent zero cost that already came up with unpriced models in the first article in the track.

And the permission setting to review on day one: access control for MCP servers is open by default. If no level of the hierarchy defines a list, the request reaches every configured server. general_settings.require_key_mcp_access_defined: true inverts that inheritance, and it is not the default value.

A reference `config.yaml`

All of the above, together, on a fleet split into two pools:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018 # without this, silent zero cost
 output_cost_per_token: 0.00000072
 model_info:
 id: chat-01
 max_input_tokens: 32768 # without this there is no context filter

 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072
 tag_regex: ["^User-Agent: claude-code\\/"]
 model_info:
 id: agentes-01
 max_input_tokens: 131072
 allow_fail_open: true

router_settings:
 routing_strategy: simple-shuffle
 enable_pre_call_checks: true
 enable_tag_filtering: true
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600
 num_retries: 1
 retry_after: 2
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3
 retry_policy:
 TimeoutErrorRetries: 0
 AuthenticationErrorRetries: 0

general_settings:
 fail_closed_budget_enforcement: true
 require_key_mcp_access_defined: true
 user_api_key_cache_ttl: 10
 proxy_batch_write_at: 10
 # 1.101 branch onwards:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false
 cache: true # Redis, not optional

And the start-up of the two engines, with the deliberate asymmetry:

# interactive pool: protect the first token
vllm serve Qwen/Qwen3-30B \
 --scheduling-policy priority \
 --max-num-seqs 64 \
 --max-model-len 32768 \
 --max-num-batched-tokens 4096

# agent pool: throughput and deep queues
vllm serve Qwen/Qwen3-30B \
 --max-num-seqs 256 \
 --max-model-len 131072 \
 --max-num-batched-tokens 8192

Checklist

Issue separate keys for agents, with model_group_alias at key or team level. It is the only separation the client cannot evade.
Add tag_regex over the User-Agent as a safety net for agents that use personal credentials, with a default pool and allow_fail_open: true.
Leave simple-shuffle on both pools. The other four strategies have verifiable defects on homogeneous replicas.
Enable session_affinity with a one-hour TTL. Do not use prompt_caching for this while its TTL stays hardcoded.
Declare max_input_tokens on every deployment. Without that value, the context window filter is not applied and it fails silently for models that are not in the price map.
Declare per-token prices on every self-hosted model, or the agents’ spend will be zero and their budget will never run out.
max_parallel_requests on the agent’s key, never on its team. On the team it is stored and not applied.
Lower num_retries to 1, put a floor on retry_after, raise cooldown_time and disable retry on timeout. On the client, max_retries=0.
fail_closed_budget_enforcement: true and mandatory Redis. Without Redis, N replicas are N times the budget.
Start the interactive engine with --scheduling-policy priority even though it pushes nobody ahead: it changes who it evicts.
Alert on vllm:num_preemptions_total and vllm:num_requests_waiting_by_reason, not on log strings that are no longer emitted.
Prepare the cut-off: block by key, never by team, and verify the real propagation time in your own deployment.

Traps and things that are not what they seem

priority: 0 on /chat/completions disables prioritisation. It is the value in the documentation examples and it is falsy in Python.
max_parallel_requests on a team is stored and not applied. The field exists in the API, in the database and in the documentation; the limiter only reads it from the key descriptor.
global_max_parallel_requests is a no-op with the default limiter, and the server settings documentation still describes it as active.
Every MCP call leaks a concurrency slot, which only heals when its hour expires. With agents calling tools in a loop, the 429 appears early and lasts a long time.
The TTL of prompt cache affinity is 300 hardcoded seconds. For agent conversations, session affinity is the right option.
Affinity is evaluated before tag-based routing. A pin to a deployment that stops satisfying the tag makes the request fail instead of falling through to the default pool.
LiteLLM’s timeout is per attempt. The worst case is the timeout multiplied by attempts and by fallback groups.
A 429 takes the deployment out of the pool without going through allowed_fails, and backoff is zero while anything healthy remains.
The engine’s Retry-After is ignored if it asks for more than 60 seconds.
drop_params: true removes tools and response_format without leaving a warning, and the fallback does not check whether the destination model supports function calling.
/team/block does not invalidate the cache. It takes the full TTL; /key/block does propagate by publish in Redis.
Access to MCP servers is open by default if no level defines a list.
max_num_partial_prefills and max_long_partial_prefills no longer exist in vLLM. A manifest with those flags does not start from 0.27 onwards.
vLLM’s priority does not push ahead of running requests, and with max_num_seqs full it pushes nobody ahead.
The eviction log warning that appears in the vLLM documentation is no longer emitted. Only the Prometheus counter is left.

Closing

The coexistence of humans and agents on the same platform is not solved with a priority field, and that is the conclusion that takes the most effort to accept because the field exists and looks made for this. Neither the gateway nor the engine has a usable notion of class of service today: one orders a queue that is almost never consulted, the other picks who to evict. What they do have is the ability to direct each request to a different place, and that is where the win is.

The architecture that comes out of all of the above fits in one sentence. Two vLLM fleets with opposite parameters, a credential that decides which one each request goes to, session affinity so as not to throw away the prefill work, concurrency limits on the agent’s key and not on its team, retries trimmed back on the three layers where they multiply, and budgets that fail closed.

The rest is measurement. If vllm:num_preemptions_total rises in the interactive pool, the two classes are treading on each other despite the separation. If chat queue time grows while total throughput holds, the partition is badly sized. And if the agents’ spend comes out as zero, the problem is not coexistence but undeclared prices, which is where this track started.

Sources

LiteLLM, [BETA] Request Prioritization: https://docs.litellm.ai/docs/scheduler.
LiteLLM, Router - Load Balancing (estrategias, afinidad de sesión, comprobaciones previas): https://docs.litellm.ai/docs/routing.
LiteLLM, Tag Based Routing (incluye tag_regex y el ejemplo con clientes agénticos): https://docs.litellm.ai/docs/proxy/tag_routing.
LiteLLM, Router Settings for Keys and Teams (resolución clave, equipo, global): https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.
LiteLLM, Server Tuning (control de admisión por worker): https://docs.litellm.ai/docs/proxy/server_tuning.
LiteLLM, Dynamic TPM/RPM Allocation (reserva de prioridad, beta y enterprise): https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.
LiteLLM, Budgets, Rate Limits: https://docs.litellm.ai/docs/proxy/users.
LiteLLM, Production Best Practices: https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Reliability (fallbacks, ventana de contexto, enfriamientos): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, código: litellm/scheduler.py, litellm/router_strategy/tag_based_routing.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/hooks/parallel_request_limiter_v3.py, litellm/proxy/middleware/admission_control_middleware.py, litellm/router_utils/cooldown_handlers.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de la afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en cada llamada MCP: https://github.com/BerriAI/litellm/issues/34534.
LiteLLM, incidencia 26672, presupuesto de clave y usuario no aplicado: https://github.com/BerriAI/litellm/issues/26672.
LiteLLM, incidencia 872, comprobación previa de soporte de llamada a funciones, cerrada sin implementar: https://github.com/BerriAI/litellm/issues/872.
vLLM, Optimization and Tuning (prefill troceado, prioridad de decodificación): https://docs.vllm.ai/en/v0.29.0/configuration/optimization.html.
vLLM, Metrics design (nombres, convención _total, autoescalado como problema abierto): https://docs.vllm.ai/en/v0.29.0/design/metrics.html.
vLLM, Data Parallel Deployment (caché KV independiente por réplica): https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment.html.
vLLM, código: vllm/v1/core/sched/scheduler.py, vllm/v1/core/sched/request_queue.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/metrics/loggers.py: https://github.com/vllm-project/vllm.
vLLM, incidencia 40004, la prioridad no desaloja peticiones en ejecución: https://github.com/vllm-project/vllm/issues/40004.

Virtual keys, budgets and limits in LiteLLM: the layer that decides who gets the GPU, and the four things the documentation gets wrong

Tue, 08 Sep 2026 07:30:00 +0200

Third article in the operational track of the control layer. The pair with Langfuse covered observability, and day 2 of the proxy covered availability. Here it is governance: who can call, which model, how much and on what budget. The economic model behind those figures is in FinOps and multi-tenancy with LiteLLM.

TL;DR

On a platform running on your own GPUs there is no provider invoice to put consumption in order. Capacity is fixed, known and shared, so someone has to impose the allocation, and that someone is the gateway.

The hierarchy has six levels: organisation, team, team member, internal user, key and end customer, plus the project and tag scopes. Each level’s budgets are evaluated independently and all of them have to pass. They neither add up nor cancel each other out, with one exception.

The exception is in the code and it is the most important governance trap in the system. A key that belongs to a team suppresses the check of its owner’s personal budget, unless apply_user_budget_to_team_keys is enabled. A user with a hundred-euro cap who works with their team’s key spends whatever the team allows.

Rotating a key preserves its spend. /key/regenerate does an UPDATE on the same row changing the hash, so spend, max_budget and the budget window remain intact. It accepts a grace period, stored in a table of its own, and if the period’s format is wrong it is ignored without blocking anything.

The v3 limiter is already the default in 1.100.0, and you go back to the previous one with an environment variable. Its window is sliding, sixty seconds, anchored to the first request, and the counters are incremented with Lua scripts inside Redis using Redis’s own timestamp to close the race between replicas.

And four points where the official documentation does not match the code, verified against tag v1.100.0: an exhausted budget returns 429, not 400; the audited actions are six, not three; /key/info and /key/list are GET, not POST; and the v3 limiter carries a docstring saying it is not production-ready while it is the one loaded by default.

You are here: the gateway as an administrative boundary

Without this layer, an in-house inference platform has a single mode of operation: whoever knows the URL consumes. It works while there is one team. It stops working the day there are three, or the day someone connects an agent that fires twenty calls per interaction and wrecks everyone else’s time to first token.

The two questions this layer answers are different and have to be kept apart. The capacity one is instantaneous: how many requests and how many tokens per minute each consumer can push, so the fleet does not saturate and latency stays within the service agreement. The cost one is cumulative: how much each consumer has spent this month against its budget. They are configured in the same place, applied at different moments and fail in different ways.

The analogy: the building’s key ring

An office building hands out keys at three levels. The master opens everything and the building manager has it. The floor keys open a whole floor and each tenant company has one. And the office keys open one office.

Three properties of that system are the ones that matter here. A key is replaced without changing the lock of the whole building, and the tenant keeps their parking space and their electricity meter: that is rotation preserving the spend. A lost key is cancelled from the central panel without collecting anything physical: that is blocking from the interface. And the record of who took out each key and when lives in a separate book, which is what you show an inspector: that is the audit table.

The analogy breaks at one point, and that point is where this system’s governance flaw lives. In the building, having a floor key does not cancel the office’s consumption limit. In LiteLLM, it does.

The hierarchy, and the line that breaks it

The levels and their management endpoints:

Level	Endpoints	Budget
Organisation	`/organization/new`, `/organization/member_add`, …	`max_budget`
Team	`/team/new`, `/team/member_add`, `/team/block`, …	`max_budget`
Team member	`/team/member_add` with `max_budget_in_team`	Its own row in the budgets table
Internal user	`/user/new`, `/user/update`, …	`max_budget`
Key	`/key/generate`, `/key/update`, …	`max_budget`, `soft_budget`
End customer	`/customer/new`, and the parallel `/end_user/*` set	`max_budget`

The default behaviour is the right one for a multi-tenant platform: each scope carries its own counter and the check of each one is independent of the rest. A request passes if all of them pass.

And then there is this condition, which lives in the authorisation checks:

is_team_key = team_object is not None and team_object.team_id is not None
if is_team_key and general_settings.get("apply_user_budget_to_team_keys") is not True:
 return

Translated: when the key belongs to a team, the check of the budget of the user who owns that key returns without evaluating anything. The personal cap exists in the database, is visible in the interface, and is not applied.

For a deployment where personal budgets are decorative, it makes no difference. For one where the per-person cap is the control that was shown to someone as a guarantee that a user cannot overrun the platform, it is a hole. The line that closes it is one:

general_settings:
 apply_user_budget_to_team_keys: true

Virtual keys: the complete life cycle

The endpoints all exist and some of them are not where you expect:

Route	Method	What for
`/key/generate`	POST	Create
`/key/update`	POST	Modify
`/key/info`	GET	Query one
`/key/list`	GET	List
`/key/delete`	POST	Delete
`/key/regenerate`	POST	Rotate
`/key/block`, `/key/unblock`	POST	Suspend and reactivate
`/key/health`	POST	Check that key’s callbacks

The two marked in bold are GET, and several examples in the documentation suggest otherwise. /key/health is not what its name suggests either: it does not validate the key or its budget, it checks the logging callbacks associated with it.

There are many creation parameters, and these are the ones that govern:

curl -X POST 'https://gateway.interno/key/generate' \
 -H 'Authorization: Bearer sk-...' \
 -d '{
 "key_alias": "equipo-datos-notebooks",
 "team_id": "t-datos",
 "models": ["llama-70b", "qwen-30b"],
 "max_budget": 250,
 "budget_duration": "30d",
 "tpm_limit": 400000,
 "rpm_limit": 600,
 "max_parallel_requests": 8,
 "duration": "90d",
 "tags": ["produccion", "notebooks"]
 }'

Alongside those there are controls that are rarely used and solve specific problems: model_max_budget to set a per-model cap within the same key, model_tpm_limit and model_rpm_limit for the same thing with capacity, enforced_params to require certain keys to always arrive with given fields, allowed_routes so an application key cannot touch the management endpoints, and blocked to be born suspended.

How they are stored

A key’s hash is a single-pass hexadecimal SHA-256, with no salt and no derivation function:

hashed_token = hashlib.sha256(token.encode()).hexdigest()

The row lives in LiteLLM_VerificationToken, with the hash as primary key, and it contains the accumulated spend, the limits, the expiry, the allowed models and the rest of the configuration.

Two consequences to be clear about before an audit. The first is that this hash is not designed to resist a dictionary attack against the database, and the code itself acknowledges it in a comment about encryption key derivation, where it admits that a single-pass unsalted SHA-256 is not a derivation function and that moving to HKDF would be more defensible in an audit. The real mitigation is that virtual keys are generated with enough entropy, not the strength of the hash. The second is that LITELLM_SALT_KEY plays no part here: that variable encrypts the stored provider credentials, not the virtual keys.

There is a trap around that variable that takes down a whole deployment. If it is not defined, it falls back to the master key. Rotating LITELLM_MASTER_KEY without having first set your own LITELLM_SALT_KEY leaves every stored provider credential unreadable, and the failure does not block: it logs an error and returns null, so the symptom shows up as models that stop authenticating with no explanation. Setting LITELLM_SALT_KEY on day one, and never touching it, is one of the things that saves the most grief.

Rotation

/key/regenerate does not create a new key and delete the old one. It updates the same row changing the hash, so the accumulated spend, the maximum budget and the reset window are preserved. For a monthly chargeback this is exactly what you want: rotating mid-month does not zero the counter.

It also accepts a grace period, so the old key remains valid while consumers update:

curl -X POST 'https://gateway.interno/key/sk-vieja/regenerate' \
 -H 'Authorization: Bearer sk-...' \
 -d '{"grace_period": "24h"}'

The old hash is stored in a separate table with its revocation date. Without that parameter, revocation is immediate, and the default value can be set with LITELLM_KEY_ROTATION_GRACE_PERIOD.

A detail with teeth: if the period’s format is wrong, the system logs a warning and continues with no grace period. The rotation completes, the old key dies on the spot, and all that is left is a log line. In an automated rotation procedure, that combination of “carry on” and “warn in the log” is what produces the outage at three in the morning.

What exactly happens when a budget runs out

The code raises BudgetExceededError with status 429. The documentation says 400 in one place, and there the code wins. Its being a 429 has a practical implication: OpenAI clients retry automatically on a 429, so an exhausted budget turns into a retry storm against a gateway that is going to keep rejecting. Internal clients should distinguish the reason, which travels in the message.

The messages are templates and state the scope, the spend and the cap:

Budget has been exceeded! Key=<clave> Current cost: <gasto>, Max budget: <tope>
Budget has been exceeded! Team=<equipo> Current cost: <gasto>, Max budget: <tope>
ExceededBudget: User=<usuario> over budget. Spend=<gasto>, Budget=<tope>
ExceededBudget: End User=<cliente> over budget. Spend=<gasto>, Budget=<tope>
Budget has been exceeded! Organization=<org> ...
Budget has been exceeded! Tag=<etiqueta> ...

Two clarifications about the comparison. It is spend >= cap, with the equals sign included, so landing exactly on the cap already blocks. And the soft budget, soft_budget, blocks nothing: it only writes a log line when it is crossed. It serves to warn, not to contain, and building an alert on top of it is the right way to use it.

The rest of the rejections, with their codes:

Situation	Code	Behaviour
Budget exhausted	429	Message with scope, spend and cap
Expired key	401	It also deletes the cache entry, so the expiry takes effect across the whole fleet without waiting for the TTL
Model not allowed	403	Different error type depending on the scope: key, team, user, organisation or project
Blocked key	Generic exception	`Key is blocked. Update via /key/unblock if you're an admin.`

The last row is an inconsistency in the project worth knowing about: blocking raises an exception with no explicit type or code, while expiry and model access do carry them. Any alert that discriminates by error type will not see blocks the way it sees the rest.

A useful note about allowed models: the check accepts both the alias and the underlying model, so a key authorised for llama-70b can call it by its alias or by its real name with no additional configuration.

Capacity limits: the v3 limiter

This is the most significant silent change of the year in this layer. In 1.100.0, the limiter loaded by default is the third version. Going back to the previous one has to be asked for explicitly:

LEGACY_MULTI_INSTANCE_RATE_LIMITING=true

There is no flag to turn it on, only to turn it off. And it carries a contradiction worth knowing before trusting it: the file itself still opens with a comment saying it is under development and not production-ready, while it is the one running. The design is avowedly inspired by Envoy’s.

The window is sliding and lasts sixty seconds, configurable with LITELLM_RATE_LIMIT_WINDOW_SIZE. It is not an exact rolling log nor a clock-aligned window: it anchors to the first request and resets once the window size has elapsed from that moment. For one-minute limits, the difference from a calendar window shows up at the edges and rarely matters.

What does matter is Redis. The counters are incremented with Lua scripts executed inside Redis, using Redis’s own timestamp and not the pod’s, precisely so window resets are deterministic across replicas and to close the race between checking and counting. Without Redis, each pod counts on its own and the effective limit is multiplied by the number of pods. A limit of 600 requests per minute with five replicas is a limit of 3,000.

There is an inherent imprecision in the token limits that Redis does not solve. The number of output tokens is not known until the response finishes, so the limiter counts with an estimate before the call and reconciles afterwards. The possible excess is, per in-flight request, the difference between the estimate and the real figure. With high concurrency and long responses, that allows the cap to be exceeded within a window. To protect the fleet, the concurrent request limit (max_parallel_requests) is a more direct instrument than tokens per minute.

What the client sees

When limited, a 429 with the message giving the scope, the current limit, what is left and when it resets. And three headers: retry-after with the window size, that is 60 by default, plus rate_limit_type and reset_at.

On the success path, the informational headers use a nested format of their own:

x-ratelimit-api_key-remaining-requests
x-ratelimit-api_key-limit-tokens
x-ratelimit-team-remaining-requests

They are not the standard names x-ratelimit-limit-requests that OpenAI and compatibles emit, so a client expecting the usual format will find nothing. There is also an open issue about these headers being lost on streaming responses, which is most of an assistant’s traffic.

Identity: beyond the static key

Handing out keys by hand works with three teams and stops working with thirty. The two routes for connecting the gateway to corporate identity:

JWT. It is enabled with enable_jwt_auth and configured with the litellm_jwtauth block, which maps token fields to LiteLLM entities: the team comes from team_id_jwt_field, the user from user_id_jwt_field, the organisation from org_id_jwt_field. It supports dot notation for nested claims, automatic creation of users and teams with user_id_upsert and team_id_upsert, role-based access control with enforce_rbac, per-team model restriction with enforce_team_based_model_access, and several issuers at once. The issuer’s public key is cached with a TTL of 600 seconds by default.

general_settings:
 enable_jwt_auth: true
 litellm_jwtauth:
 team_id_jwt_field: "groups"
 user_id_jwt_field: "sub"
 user_email_jwt_field: "email"
 enforce_rbac: true
 enforce_team_based_model_access: true
 team_id_upsert: true

OIDC for the admin interface. With Keycloak you use the generic provider, and the variables are PROXY_BASE_URL, GENERIC_CLIENT_ID, GENERIC_CLIENT_SECRET, GENERIC_AUTHORIZATION_ENDPOINT, GENERIC_TOKEN_ENDPOINT, GENERIC_USERINFO_ENDPOINT, plus a set of attributes to map the identifier, the email, the name and the role, and GENERIC_ROLE_MAPPINGS_* to translate Keycloak groups into LiteLLM roles. Login is rate-limited and sessions have a TTL of their own.

For automatic provisioning there is SCIM in the enterprise version, with base /scim/v2, and deprovisioning a user blocks their keys and takes them out of the authentication cache. That part is not in the open tree, so it cannot be verified against the code.

This fits with the identity work already covered in hardening and secrets of the sovereign stack and in the article on MCP authentication with Keycloak, which uses the same issuer.

Audit: what gets recorded

The table is LiteLLM_AuditLog and its columns are what you would expect from a change log:

id, updated_at, changed_by, changed_by_api_key,
action, table_name, object_id, before_value, updated_values

It stores the previous value and the new one, and attributes the change to an actor and to the key it was made with.

Three clarifications the documentation does not capture well.

The actions are six, in the past tense: created, updated, deleted, blocked, unblocked and rotated. The documentation mentions three. That blocking, unblocking and rotation are actions in their own right is exactly what is needed to reconstruct a credential’s life cycle for an auditor.

The scope is wider than advertised. Besides teams, users, keys and models, changes to the proxy configuration and to the SSO configuration are audited. A change of identity issuer is recorded.

The default depends on the licence. The resolution order is litellm_settings.store_audit_logs, then the variable LITELLM_STORE_AUDIT_LOGS, and if neither is set, it stays enabled in enterprise and disabled in the open version. An OSS deployment that assumes there is an audit trail does not have one:

litellm_settings:
 store_audit_logs: true

Two more pieces for a serious chain of evidence. The actor attribution can be passed with the LiteLLM-Changed-By header, and its use is restricted by the key’s or the team’s configuration, so any caller cannot forge it. And the events can be shipped out with audit_log_callbacks and its S3 parameters, which is what allows depositing them in an immutable store with object lock.

That last part is what turns the record into evidence. The Postgres table is mutable by anyone with database access, and as of September 2026 it has no indexes, so querying it by date range over a long history will not be comfortable either.

What this demonstrates to an auditor, and what it does not

This is the part that decides whether the setup is good enough for a regulated customer. With everything above configured, the platform sustains without difficulty:

Access control via corporate identity, with onboarding and offboarding in the identity provider and not in the gateway, and with the role determining which models are reachable. Segregation between tenants, with independent counters per organisation, team and person. Traceability of administrative changes, with actor, time, previous value and new value, exportable to an immutable store. And demonstrable consumption limits, with the evidence of the rejection in the spend log.

What it does not cover, and has to be solved elsewhere:

The record of the interactions is not this. The audit table records configuration changes, not model calls. The calls are in the spend log and in the traces, and traces are best-effort, which is the discussion in the first article in the series.

The integrity of the record is not something Postgres provides. Without export to immutable storage, the evidence is only as strong as the database permissions.

The content of the prompts is a separate problem, with its own personal-data treatment and its own decision about whether it is stored, masked or not logged at all.

And a warning about the control that tends to be shown first: while apply_user_budget_to_team_keys is not enabled, the per-person cap is not being applied to team keys. Showing that field in the interface as proof of a control that is not executed is the kind of finding a competent auditor discovers. The complete control mapping is in ENS, ISO 42001 and the EU AI Act.

Checklist

apply_user_budget_to_team_keys: true if personal budgets have to be enforced.
LITELLM_SALT_KEY set on day one, different from the master key and never rotated lightly.
store_audit_logs: true explicitly, without trusting the default.
audit_log_callbacks towards an immutable store if an audit has to be sustained.
Redis mandatory if there is more than one replica, or the limits get multiplied by the number of pods.
max_parallel_requests per key, on top of the per-minute limits, to protect the fleet from a runaway agent.
allowed_routes on application keys, so they cannot reach the management endpoints.
soft_budget with an alert on top, understanding that it does not block.
Rotation with a verified grace_period, because an invalid format revokes on the spot.
JWT against Keycloak as soon as there are more than a handful of teams, with enforce_rbac and enforce_team_based_model_access.

Traps and things that are not what they look like

A user’s cap is not enforced. They are using a team key and apply_user_budget_to_team_keys is missing.

Clients retry in a loop against an exhausted budget. The code is 429 and the OpenAI libraries retry 429s automatically.

The soft budget has stopped nothing. It does not block, it only logs.

The per-minute limits are enforced at double or triple. There is no Redis, and each replica keeps its own count.

The client does not see the limit headers. The names are nested and proprietary, and in streaming there is an open issue where they get lost.

There is no audit log. It is an OSS deployment and the default there is disabled.

The models stop authenticating after rotating the master key. There was no fixed LITELLM_SALT_KEY, and the encrypted provider credentials can no longer be decrypted.

The rotation cut the service on the spot. The grace_period carried a format the system did not understand and it carried on without it.

Blocking a key does not fire the alert. It is raised as a generic exception, without the type that expiry and model access do carry.

Closing

The keys and budgets layer is the one that turns a shared URL into a platform with tenants. It is well solved in LiteLLM and has more depth than the documentation suggests, with per-model budgets inside a key, multiple budget windows, per-tag limits and rotation that preserves the history.

Out of the whole article, two things take the attention. The apply_user_budget_to_team_keys line, because it is the difference between having a control and believing you have one. And setting LITELLM_SALT_KEY, because its default turns a routine rotation of the master key into an outage that is hard to diagnose. Both are one line of configuration and both are discovered late.

Sources

LiteLLM, Virtual Keys (parámetros de creación, ciclo de vida, rotación): https://docs.litellm.ai/docs/proxy/virtual_keys.
LiteLLM, Budgets, Rate Limits (jerarquía, presupuestos por nivel, clientes finales): https://docs.litellm.ai/docs/proxy/users.
LiteLLM, JWT Token Auth (bloque litellm_jwtauth y todos sus campos): https://docs.litellm.ai/docs/proxy/token_auth.
LiteLLM, Audit Logs y administración múltiple (la cabecera LiteLLM-Changed-By): https://docs.litellm.ai/docs/proxy/multiple_admins.
LiteLLM, SCIM (aprovisionamiento automático, versión enterprise): https://docs.litellm.ai/docs/tutorials/scim_litellm.
LiteLLM, litellm/proxy/auth/auth_checks.py en la etiqueta v1.100.0 (la condición de apply_user_budget_to_team_keys y los mensajes de presupuesto agotado): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/auth/auth_checks.py.
LiteLLM, litellm/proxy/management_endpoints/key_management_endpoints.py en v1.100.0 (rutas, rotación sobre la misma fila, periodo de gracia): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/management_endpoints/key_management_endpoints.py.
LiteLLM, litellm/proxy/hooks/parallel_request_limiter_v3.py y litellm/proxy/hooks/__init__.py en v1.100.0 (el limitador v3 como valor por defecto, la ventana, los scripts Lua): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/hooks/parallel_request_limiter_v3.py.
LiteLLM, litellm/exceptions.py en v1.100.0 (BudgetExceededError con código 429): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/exceptions.py.
LiteLLM, litellm/proxy/common_utils/encrypt_decrypt_utils.py en v1.100.0 (LITELLM_SALT_KEY cayendo en la clave maestra, y el comentario sobre la derivación de claves): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/common_utils/encrypt_decrypt_utils.py.
LiteLLM, schema.prisma en v1.100.0 (LiteLLM_VerificationToken, LiteLLM_AuditLog, la tabla de claves deprecadas): https://github.com/BerriAI/litellm/blob/v1.100.0/schema.prisma.
LiteLLM, issue #27748 — cabeceras x-ratelimit-* perdidas en respuestas en streaming con el limitador v3: https://github.com/BerriAI/litellm/issues/27748.

LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

Tue, 08 Sep 2026 07:00:00 +0200

Second article in the operational track of the control layer. The operational pair with Langfuse covered the seam between the gateway and observability; here everything stays inside the gateway. The prior decisions, which gateway and why, are in choosing the OSS gateway and in the L7 inference router.

TL;DR

LiteLLM Proxy starts from a twenty-line config.yaml and serves traffic that same afternoon. The operational work consists of correcting seven defaults designed for a scenario other than an in-house inference factory.

One uvicorn worker per pod. The project’s production guide asks for --num_workers 1 on Kubernetes and horizontal scaling, with 1 vCPU and 4 GiB per worker set at the same time as requests and as limits. The 4 GiB are a floor, not a target: the Prisma query engine marks a resident memory high-water mark that grows up to the largest statement it has executed and that glibc does not return to the system.

Autoscale on CPU, never on memory. A direct consequence of the above. Memory goes up and never down, so an HPA on memory scales and never scales back. The recommended target is targetCPUUtilizationPercentage: 60.

The Postgres connection arithmetic. database_connection_pool_limit is 10, and the number of connections is instances times workers times the pool. The Helm charts ship maxReplicas: 100, on the order of a thousand connections, far above what a stock Postgres accepts.

Redis stops being optional from around a thousand requests per second or ten instances. Without it, each instance applies its limits on its own, caches are not shared, and spend updates against the same rows produce locks and exhaust the connections with FATAL: sorry, too many clients already.

Three health endpoints and only two are usable as probes. /health/liveliness and /health/readiness require no authentication and answer fast. Bare /health demands a key and fires a real request against every model in the catalogue, with its cost in tokens.

The resilience numbers are 2, 3 and 5: two retries, three allowed failures and five seconds of cooldown, the last one despite the function’s own documentation saying the default is 1.

request_timeout is 6,000 seconds. A hundred minutes holding a connection against an engine that does not answer.

And a warning about the performance figures: the project publishes 8 ms of p95 overhead on one benchmark and 257.7 ms of p99 on another. Both are theirs, and the difference is in the shape of the load.

You are here: the gateway layer, the day after

In the seven-layer stack, this article lives entirely in the gateway layer. The difference with the selection article is the one that separates buying from maintaining: there the question was which piece to put in front of the fleet, and here it is what happens when that piece has been serving for six months, the team has tripled the traffic and someone has added an agent that fires twenty calls per interaction.

The gateway has an uncomfortable property worth keeping in mind: it sits on the critical path of every request and contributes not a single token. All it does is coordination, and any coordination that fails turns into an outage of the whole inference service, even with the GPUs perfectly healthy.

The analogy: the switchboard and its notebook

A hospital switchboard does three jobs at once. It puts calls through, which is the urgent part. It writes down in a notebook who called, to which extension and for how long, which is what sustains the billing. And it consults a list of extensions to know where to put each call through.

The three jobs compete. If the operator stops to write each line in the notebook before putting the next call through, the waiting queue grows. If there are three operators and each keeps her own notebook, the totals do not add up. And if the list of extensions is consulted in a filing cabinet on another floor, every call costs the trip.

The three tensions have an exact equivalent in LiteLLM. The notebook is the SpendLogs in Postgres, which is why the transaction buffer in Redis exists. The three operators with three notebooks are the pod replicas applying limits independently. And the filing cabinet on another floor is usage-based routing, which adds a Redis lookup inside the request path.

The process model: one worker per pod

The production documentation is explicit: on Kubernetes, --num_workers 1 and scale with replicas. On a virtual machine with no orchestrator, NUM_WORKERS equal to the number of vCPUs.

The reason is not ideological. An additional worker inside the same pod shares the container’s memory limit with the others, multiplies the database connections by the same factor, and registers a copy of every periodic job. With separate replicas, each one has its own resource budget, its own connection quota and a place where the scheduler can put it.

The recommended sizing is 1 vCPU and 4 GiB per worker, and the part that gets overlooked is that these go as requests and as limits at the same time. Setting less than 4 GiB produces an OOM restart loop that shows up late, once some large request has grown the query engine’s footprint.

resources:
 requests: { cpu: "1", memory: "4Gi" }
 limits: { cpu: "1", memory: "4Gi" }

That footprint needs to be described precisely, because it determines the autoscaling policy. The Prisma query engine keeps a resident memory that behaves as a high-water mark: it grows up to the largest statement the process has executed and does not come back down, because glibc does not return that freed memory to the system. A pod that served a spike yesterday still shows the footprint of that spike today.

Two rules come out of this. The first: autoscale on CPU and never on memory. An HPA on memory sees a metric that only goes up, scales at the peak and never scales back. The recommended target is targetCPUUtilizationPercentage: 60, lower than the 80 the charts ship by default, because the startup probe allows up to 300 seconds before passing the first readiness check and there needs to be headroom for the new pod to be ready before the saturated one goes down.

The second: bound the process lifetime. --max_requests_before_restart 10000 recycles the worker before the high-water mark matters.

Postgres: the arithmetic that blows up when you scale

database_connection_pool_limit is 10 by default. The total number of connections the deployment opens is replicas times workers times pool, and it has to be compared against the Postgres max_connections.

The formula the documentation gives is the inverse, and it is the one to apply when sizing:

database_connection_pool_limit = MAX_DB_CONNECTIONS / (instances × workers)

The detail that hurts in production is in the Helm charts, which ship autoscaling.maxReplicas and keda.maxReplicas at 100. With the default pool, that is on the order of a thousand connections at full scale-out, far above what an untouched Postgres accepts. And the moment the HPA reaches those replicas is exactly the moment of highest load, so connection exhaustion arrives when it can least be afforded.

There are two more valves on the same path. Provider errors are written to the database by default, and under a sustained engine failure that inflates the spend table without contributing anything that is not already in the metrics. They are switched off like this:

general_settings:
 disable_error_logs: true # stops writing provider errors
 proxy_batch_write_at: 60 # batches the spend writes

disable_spend_logs: true is the radical version, which removes the per-request detail from the interface and leaves the cost in Prometheus and in the tracing backend. Before going that far you have to decide whether that table is the system’s audit record, which is the discussion in the previous article: if it is, it cannot be switched off, and the way out is the Redis buffer.

Redis: when it stops being optional

Without Redis, LiteLLM works. Each instance keeps its own in-memory cache and applies its own counters. The consequences are three, and all of them show up as you grow.

Requests-per-minute limits are applied per instance, so a limit of 100 with five replicas is in practice a limit of 500. Cache hits are local, so the same repeated question hits once in five. And there is no leader election, so periodic jobs run in every process at once.

The threshold the documentation gives for enabling the transaction buffer is a thousand requests per second or ten instances. Below that, each instance updates the key, user and team rows directly; above it, they all write to the same rows, locks appear and Postgres starts rejecting with FATAL: sorry, too many clients already.

general_settings:
 use_redis_transaction_buffer: true

The metrics to watch when that is enabled are litellm_in_memory_spend_update_queue_size, litellm_redis_spend_update_queue_size and litellm_pod_lock_manager_size. If the first grows without the second coming down, the flush to Postgres is not keeping up.

A note about the routing strategy, which interacts with this. simple-shuffle consults nothing external. Usage-based routing does, and it adds a round trip to Redis inside the request path. On a homogeneous vLLM fleet behind the same model, the improvement it brings rarely pays for that latency; on a heterogeneous fleet, the conversation is different and the router article covered it.

Background jobs register per worker

This detail appears in no getting-started guide and produces considerable bewilderment when it is discovered. The proxy’s periodic jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every job, all of them running with no coordination if there is no Redis for leader election.

The variable that separates them is LITELLM_JOB_ROLE: value serving on the pods that serve traffic and a separate replica with value worker for the jobs. With the one-worker-per-pod recommendation the problem shrinks, but it does not go away while there are several replicas.

Probes: three endpoints and only two are usable

The three endpoints exist and do different things:

Endpoint	Authentication	What it does	What it is for
`/health/liveliness`	No	Returns `I'm alive!`, or 503 during shutdown	`livenessProbe`
`/health/readiness`	No	State of the process and the database. 503 if the configured Postgres does not answer	`readinessProbe`
`/health`	Yes	Fires a real request against every model in the catalogue	Manual diagnosis, never a probe

The last row deserves insisting on. /health spends tokens on every call, one per configured model. Set as a probe, with the Kubernetes default interval and a fleet of eight models, that is thousands of inference requests a day that serve nobody.

There are more endpoints useful for diagnosis: /health/readiness/details, /health/services?service=langfuse to check one specific callback, /health/history and /health/latest for the history, and /health/backlog, which returns the in-flight requests and shows up in the latency section. /health/drain exists but returns 404 unless it is enabled with enable_drain_endpoint: true, and it is protected with X-Drain-Token.

The periodic model checks are configured separately and not in the probe:

general_settings:
 background_health_checks: true
 health_check_interval: 300 # seconds
 background_health_check_model_groups: ["llama-70b", "qwen-30b"]

And one setting with consequences in deployments with no Internet egress: allow_requests_on_db_unavailable: true allows serving traffic with the database down, but leaves /health/readiness always returning 200. With that set, the readiness probe stops detecting the very failure it was designed to detect, and budget and model errors keep blocking just the same.

Retries, cooldowns and fallbacks: the real numbers

The defaults, read from the project’s code and not from the documentation:

Constant	Value	What it governs
`DEFAULT_MAX_RETRIES`	2	Retries before moving on to fallbacks
`DEFAULT_ALLOWED_FAILS`	3	Failures before cooling down a deployment
`DEFAULT_COOLDOWN_TIME_SECONDS`	5	Seconds a deployment stays out
`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD`	1000	Minimum requests before applying the cooldown logic with a single deployment
`DEFAULT_REQUEST_TIMEOUT_SECONDS`	6000.0	Request timeout

Besides the failure counter, there is a proportional criterion: a deployment enters cooldown if 50% of its requests fail in any given minute.

On the cooldown there is a discrepancy worth knowing about before debugging blind. The docstring of the router function says cooldown_time defaults to 1, and the code uses the constant of 5. The code wins.

The fallbacks are three separate lists and are applied after the retries are exhausted:

router_settings:
 num_retries: 2
 timeout: 120
 fallbacks: [{"llama-70b": ["qwen-30b"]}]
 context_window_fallbacks: [{"llama-70b": ["llama-70b-128k"]}]
 content_policy_fallbacks: [{"llama-70b": ["llama-70b-sin-guardrail"]}]

A note on the behaviour when everything is in cooldown: if no deployment is left available in the group, LiteLLM falls back onto a specific model_info.id, skipping the cooldown check. It is a deliberate degradation, serve something rather than return 503, and it is worth knowing it exists because it masks the real state of the fleet.

And a capability lost along the way: since version 1.85.0, the mock_testing_fallbacks parameter and its two siblings are stripped from requests arriving through the proxy and have no effect. Fallbacks can no longer be tested against the proxy with a doctored request; it has to be done against litellm.Router directly, in a test. Anyone who had a smoke test built on that has it broken without warning.

The six thousand second timeout

It deserves its own section because it is the setting with the worst damage-to-effort ratio in the whole system. request_timeout is 6,000 seconds. A hundred minutes. An inference engine that stops answering without closing the connection holds a worker for that long, and with one worker per pod that is an entire pod out of service per trapped request.

general_settings:
 request_timeout: 600

Ten minutes is still generous for a long generation, and it bounds the damage.

Overhead: the project’s two figures

LiteLLM publishes two measurements of its own overhead that differ by a factor of thirty, and both are legitimate because they measure different loads.

The benchmarks page, with Locust, a thousand users with think time and a fake endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms at 1,170 requests per second. With two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures correspond to around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, from July 2026, measures the Python proxy at 257.7 ms of added p99 and 329.5 MB of peak memory, with no callbacks, no spend tracking and no persistence. The Rust variant, in beta, gives 0.7 ms.

The reading for an in-house platform: the first figure is the best case with human clients, and the second is the order of magnitude to plan for agentic load, which arrives in a closed loop against fast responses. There is one reported case that resembles this scenario closely: an in-house OpenAI-compatible engine served around 16 requests per second directly and around 9 through LiteLLM, with the degradation growing with concurrency, on a 4 vCPU, 8 GB pod.

The latency gap the metrics cannot see

LiteLLM’s timers start when its handler starts. The time the request spends queued in the uvicorn event loop, before getting there, appears in none of its metrics. The example in its own troubleshooting guide is a case in which LiteLLM records 10 seconds and the user experiences 20.

It is detected by comparing two sources: GET /health/backlog or the litellm_in_flight_requests gauge against the response time measured by the load balancer in front. If they diverge, the gap is in the input queue and the answer is more replicas, not more tuning.

Two headers are always on and serve to watch it without instrumenting anything: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms. For the second, the documentation sets the suspicion threshold at 100 ms, above which the diagnosis is that the payloads are too large.

And the first cause of latency that guide mentions is none of the above: LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and with payloads over 2 MB that alone can cost between 2 and 5 seconds per request. A log level set to debug a problem and forgotten there produces exactly the problem it was meant to debug.

The failure mode to put in the runbook

There is an open issue describing a cascade with a characteristic shape. Under sustained 429s from the engine upstream, the pods stopped answering readiness probes during startup, Kubernetes killed them on failed probes, and the restart put more pressure on an upstream that was already saturated. Peaks of 500 requests per second, average of 60, between two and five replicas with 1.3 CPU and 4 GB.

The shape of the failure is what matters more than the specific case: the engine’s saturation turns into a restart loop of the gateway, and from the outside it looks like the problem is the gateway. The defences are the ones already in this article, applied together: startupProbe with a wide margin so readiness does not compete with startup, request_timeout bounded so trapped requests get released, cooldowns that take the deployment returning 429 out of rotation, and CPU autoscaling at 60% so the capacity exists before it is needed. It fits with what was covered in incident response runbooks.

Upgrading without cutting traffic

The deployment is a normal Deployment with RollingUpdate, maxUnavailable: 0 and maxSurge: 1, and there is no further mystery to it except on one point: schema migrations. LiteLLM uses Prisma and applies migrations at startup. With several replicas coming up at once after a version change, the migration must be run by a single process, which is another reason for the dedicated replica with LITELLM_JOB_ROLE: worker.

Before a version bump, two checks that pay for themselves: read the release notes looking for changes in the metadata emitted to the callbacks, because a change like that breaks saved dashboards without touching the service, and verify that the new configuration starts with --detailed_debug on a separate pod before applying it to the fleet.

Go-live checklist

--num_workers 1 and scaling by replicas.
requests equal to limits, 1 vCPU and 4 GiB per worker.
HPA on CPU at 60%. Never on memory.
database_connection_pool_limit computed from max_connections, and the chart’s maxReplicas lowered to something the database can take.
Redis as soon as there is more than one replica, and use_redis_transaction_buffer from ten instances or a thousand requests per second.
livenessProbe on /health/liveliness, readinessProbe on /health/readiness, and /health out of the probes.
startupProbe with a margin of up to 300 seconds.
request_timeout: 600.
LITELLM_LOG out of DEBUG, and disable_error_logs: true.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, litellm_in_flight_requests and the response time of the load balancer in front.

Traps and things that are not what they look like

The HPA scales up and never back down. It is measuring memory, which in this process only goes up.

The pods die in a loop right after deploying. Less than 4 GiB of limit. The footprint appears with the first large request, not at startup.

Postgres starts rejecting connections at the peak. The chart has maxReplicas: 100 and the pool defaults to 10.

The per-minute limits are enforced badly. There is no Redis, and each replica keeps its own count.

The health check consumes tokens. Someone set /health as the readiness probe.

The readiness probe returns 200 with the database down. allow_requests_on_db_unavailable is set.

A deployment keeps receiving traffic after failing. The cooldown is 5 seconds, not minutes, and with a single deployment in the group you have to pass a thousand requests for the logic to apply.

The fallback test no longer tests anything. The simulation parameters are stripped from requests arriving through the proxy since 1.85.0.

The p99 is ten times worse than the published benchmark. The benchmark has think time and the real load is agentic.

Closing

None of these settings is complicated and all of them are one line. What unites them is that their defaults describe a small deployment, with one instance, a human client on the other side and a commercial provider taking care of capacity. An in-house inference factory is the opposite on all four axes: several replicas, automated clients, and an engine whose capacity is finite and known.

Of the ten points in the checklist, three decide almost everything in an on-premise deployment: the process model with its sizing, the database connection arithmetic, and the request timeout. The other seven are the ones that avoid the Saturday phone call.

Sources

LiteLLM, Production best practices (topología, workers, memoria, pool de conexiones, LITELLM_JOB_ROLE, buffer de transacciones en Redis, request_timeout): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Health checks (los tres endpoints, comprobaciones de fondo, enable_drain_endpoint): https://docs.litellm.ai/docs/proxy/health.
LiteLLM, Reliability (fallbacks, reintentos, deprecación de los parámetros de simulación en la 1.85.0): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, Benchmarks y Load test advanced (las cifras de sobrecarga por número de instancias y la advertencia de la ley de Little): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/load_test_advanced.
LiteLLM, Rust AI Gateway benchmarks (los 257,7 ms de p99 del proxy en Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, Latency overhead (el hueco de la cola de uvicorn, las cabeceras de diagnóstico, el coste de LITELLM_LOG=DEBUG): https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, DB info (disable_spend_logs, disable_error_logs y qué se pierde con cada uno): https://docs.litellm.ai/docs/proxy/db_info.
LiteLLM, litellm/constants.py (los valores por defecto de reintentos, fallos permitidos, enfriamiento y timeout): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
LiteLLM, litellm/router.py (la discrepancia entre el docstring de cooldown_time y la constante que usa el código): https://github.com/BerriAI/litellm/blob/main/litellm/router.py.
LiteLLM, issue #15526 — cascada de 429 sostenidos, sondas de readiness fallidas y bucle de reinicios: https://github.com/BerriAI/litellm/issues/15526.
LiteLLM, issue #21046 — degradación de throughput contra un motor propio compatible con OpenAI: https://github.com/BerriAI/litellm/issues/21046.

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

Mon, 07 Sep 2026 18:00:00 +0200

This post opens the operational track of the control layer. The previous articles left the pieces chosen and assembled: choosing the OSS gateway and the L7 inference router explain why LiteLLM ends up in front of the fleet, and Langfuse from the inside takes apart the backend that receives the traces. Here we deal with the seam between the two, which is where the work of the next three years lives.

TL;DR

The LiteLLM plus Langfuse pair is configured with a list of callbacks and two API keys, and from there four matters decide whether the dashboard is good for operating or only for showing in a demo.

There are three integration routes and they are not equivalent. The native langfuse callback is tied to the Python SDK v2 (langfuse>=2.59.7,<3.0 in LiteLLM’s pyproject.toml), which writes against the legacy ingestion that Langfuse Cloud retires on 16 November 2026. The langfuse_otel preset exports over OTLP and is the path recommended in the current documentation of both projects. The OTel v2 route, behind the LITELLM_OTEL_V2=true flag, changes span names, leaves the success status at UNSET and, above all, its Langfuse mapper does not read the request metadata: session_id, trace_user_id, tags and trace_id stop arriving.

The cost of a self-hosted model comes out as zero unless a price is registered. get_model_info() raises an exception for a model that is not in the pricing map, completion_cost() propagates it, the logger catches it, stores it in a debugging field and returns None, and the payload turns it into 0.0. There is no warning at the default log level. A team’s budget is never exhausted because its spend is always zero.

Client-gateway-engine correlation does not come out of the box. LiteLLM does continue the client’s trace if a traceparent arrives, and it also extracts from it the 32-hex trace-id to use as litellm_trace_id, which makes the Langfuse identifier equal to the W3C one. Upstream the story changes: forward_traceparent_to_llm_provider is False by default, the function that applies it returns early if no otel logger is loaded, and when it does act it forwards the client’s original header, so the vLLM span ends up a sibling of LiteLLM’s rather than a child. Add to that the fact that vLLM exports OTLP over gRPC by default and Langfuse does not accept gRPC.

Between the request and the dashboard there are four bounded queues that drop events. LiteLLM’s logging thread pool (100 threads, 10,000 pending tasks, dropping once full), the asynchronous LoggingWorker (50,000, and once full it aggressively flushes half), OTel’s BatchSpanProcessor (2,048 spans by default) and Langfuse’s ClickhouseWriter, which drops rows after exhausting the retries with no dead-letter queue. None of the four applies back pressure on the request, which is exactly what you want, and none of them guarantees delivery, which is exactly what an auditor asks about.

The rest of the article develops the four points with configuration applicable to a generic 4×H100 cluster with vLLM, the overhead figures the project itself publishes, and the honest limit of what this data sustains before ENS or ISO/IEC 42001.

You are here: the seam between DEPLOY and OBSERVE

In the seven-layer stack, LiteLLM lives in the gateway layer and Langfuse in the observability one. The separation is clean on the diagram and false in operation: the gateway is the only point in the system that sees the complete request before chopping it up and the complete response after reassembling it, so it is also the only reasonable place from which to emit the trace. The inference engine sees tokens and latencies; it does not see who is asking or how much that team has spent this month.

That turns LiteLLM into a telemetry producer as well as a traffic distributor, and moves onto the gateway a responsibility that was not in the original design: if the callback fails, the request is served all the same and nobody notices that the dashboard has been incomplete for two days.

The analogy: the toll booth that issues the ticket

A motorway toll booth does two independent jobs. It raises the barrier, which is its visible function, and it issues a ticket with number plate, time, section and amount, which is the function the concession lives off. The two share a booth and do not share a destination: the barrier responds in a hundred milliseconds because there is a car waiting, and the ticket travels to a processing centre over a slower channel that batches thousands of records before sending them.

The design is well thought out as long as its consequence is understood. If the processing centre goes down, cars keep going through. If the ticket queue fills up, tickets are thrown away, traffic is not stopped. The concession accepts losing records rather than causing a jam, and charges on a statistical sample it knows to be imperfect.

LiteLLM and Langfuse work the same way. The four queues in the corresponding section are the slow channel, and all of them prefer to drop rather than block. Whoever builds the pair inheriting the complete analogy gets the operation right. Whoever thinks the ticket is an accounting entry ends up explaining to an auditor why traces from a Tuesday afternoon are missing.

Three routes, and one with an expiry date

As of September 2026 three paths coexist for getting a request from LiteLLM to Langfuse. LiteLLM’s documentation already labels the first as legacy.

Route	How it is enabled	Transport	Status
Native `langfuse` callback	`success_callback: ["langfuse"]`	Langfuse Python SDK v2, legacy ingestion	Maintained for compatibility. Tied to `langfuse<3.0`
`langfuse_otel` preset (v1)	`callbacks: ["langfuse_otel"]`	OTLP/HTTP against `/api/public/otel`	Recommended by both projects
OTel v2 route	`callbacks: ["langfuse_otel"]` + `LITELLM_OTEL_V2=true`	OTLP/HTTP, new mapper	Opt-in. Changes semantics and loses metadata

The date that orders the decision is 16 November 2026. Langfuse’s compatibility matrix marks the legacy batch ingestion of the Python SDK v2 as retired on that date for Cloud, at which point trace ingestion through that route stops working. On self-hosted the endpoint survives, but a v4 server in events_only mode returns 400 for every event type except score-create and sdk-log.

Since LiteLLM’s native callback is pinned to langfuse>=2.59.7,<3.0 in its own pyproject.toml, and its code calls APIs that only exist in SDK v2 (self.Langfuse.trace(...), trace.generation(...)), there is no way to move up an SDK without changing route. Anyone who today has success_callback: ["langfuse"] against Langfuse Cloud has a migration with a date on it. Anyone who has it against a self-hosted Langfuse has one with some slack, because v3 receives security patches until the end of January 2027, but they have one all the same.

One detail of the OTLP route worth checking in older deployments: the x-langfuse-ingestion-version: 4 header is the one that enables real-time ingestion. Without it the traces arrive, but with a delay that Langfuse’s documentation puts at around ten minutes (the compatibility page says fifteen, and that discrepancy remains unresolved in their own documents). LiteLLM started sending it in version 1.95.0, of August 2026. A proxy older than that version against a Langfuse v4 produces the classic symptom: the dashboard works, but it is always behind and nobody knows why.

The configuration that works today

For a generic 4×H100 cluster with vLLM behind it and Langfuse self-hosted in the same Kubernetes, the OTLP route in its v1 variant is the one that gives the most functionality with the fewest surprises.

model_list:
 - model_name: llama-70b
 litellm_params:
 model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
 api_base: http://vllm-llama70b.inference.svc.cluster.local:8000/v1
 api_key: os.environ/VLLM_API_KEY
 # Price derived from the cluster's €/GPU-hour. Without this, spend = 0.0
 input_cost_per_token: 0.00000031
 output_cost_per_token: 0.00000089

litellm_settings:
 callbacks: ["langfuse_otel"]
 turn_off_message_logging: false # a conscious decision, see the PII section
 langfuse_default_tags: ["model_group", "user_api_key_team_alias", "cache_hit"]

general_settings:
 disable_error_logs: true # provider errors inflate SpendLogs
 proxy_batch_write_at: 60

The credentials travel through the environment. LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST are enough: the preset builds the Authorization: Basic base64(public:secret) header and adds the ingestion version one, so there is no need to touch OTEL_EXPORTER_OTLP_HEADERS by hand. The preset’s default host points to the US region of Langfuse Cloud, a relevant detail for anyone building a platform with data sovereignty requirements: if the variable is not defined, the telemetry leaves Spain without anyone having decided it.

For real multi-tenancy, LiteLLM supports per-team Langfuse credentials (POST /team/<id>/callback with callback_vars) and per-key ones (metadata.logging[] when generating the key), and the key’s wins over the team’s without mixing. There is a limitation that breaks the architecture of anyone who planned to separate tenants across different Langfuse instances: under OTel v2, a per-key or per-team langfuse_host does not move that tenant to another host, because the exporter fixes the destination at startup and the only thing that varies per tenant are the headers.

And one operational warning with years of history behind it: MAX_LANGFUSE_INITIALIZED_CLIENTS is 50 by default because each Langfuse client starts a thread. The comment in LiteLLM’s code is explicit about the incident that motivated it, with the CPU at 100% from repeated initialisations. A deployment with per-team credentials and many teams hits that ceiling.

What is lost when moving to OTel v2

The LITELLM_OTEL_V2=true flag brings a step up in quality in the proxy’s general instrumentation, and at the same time a concrete regression on the path to Langfuse that the documentation acknowledges without underlining it.

The Langfuse mapper of the v2 route emits langfuse.observation.type, langfuse.observation.model.name, the observation identifier, the model parameters, input and output, usage_details and cost_details, plus langfuse.trace.metadata.team_id and team_alias. It does not emit session.id, nor user.id, nor langfuse.trace.tags, nor langfuse.trace.id, nor the generation name, nor parent_observation_id. LiteLLM’s documents put it this way: those attributes are set by the preset from the request and the response, not from metadata supplied by the client.

The practical consequence is that an application that today groups conversations by sending metadata: {"session_id": "...", "trace_user_id": "..."} in the request body stops grouping them when the flag is enabled, with no error and no warning. The traces keep arriving, loose.

The other v2 changes are semantic and affect alerts already written. The root span goes from being called Received Proxy Server Request to being named after the route (POST /v1/chat/completions), the inference span goes from litellm_request to {operation} {model}, the guardrail spans stop hanging off the inference span and hang off the root instead, and the success status stays at UNSET instead of OK. Any rule that fires on status == OK stops firing. The way back is to remove the flag and restart.

Cost: why the trace says zero

This is the part that takes the most time to diagnose and the least to fix.

LiteLLM computes the cost of a request from a per-model pricing map. For commercial provider models the map comes filled in. For hosted_vllm/whatever, which is all there is in an in-house inference factory, the map is empty, and the failure path is as follows:

litellm.get_model_info() raises an exception with the message This model isn't mapped yet.
completion_cost() and response_cost_calculator() propagate the exception.
Logging._response_cost_calculator() catches it, writes a StandardLoggingModelCostFailureDebugInformation into response_cost_failure_debug_information, logs it at debug level and returns None.
The standard logging payload does llm_response_cost = raw_response_cost or 0.0.

The system delivers a spend of zero, with no visible error at the default log level, and writes it into the spend column of LiteLLM_SpendLogs. The per-key, per-team and per-organisation budgets are evaluated against that zero and are never exhausted. The Langfuse dashboard shows a total cost of zero for everything self-hosted, which is precisely the part you wanted to measure.

Registering the price

The pricing keys are accepted both in litellm_params and in model_info, and those in litellm_params win when they are in both places. The complete set has 91 fields, but for a deployment with vLLM two are enough, and there are three things to know about them.

The first is that the price is per individual token, in USD. There are no per-thousand variants: searching for per_1k or per_1000 in the pricing model returns nothing. A three-orders-of-magnitude error here is easy to make and hard to see, because it produces plausible figures.

The second is that input_cost_per_request exists in the schema and is not applied in chat calls: a model registered with that field returns a cost of 0.0. To charge per request it has to be converted into a cost per token.

The third is that input_cost_per_second needs the response time and is computed as price times response_time_ms / 1000, so outside the proxy’s request path it silently returns zero.

The number that goes into those two keys comes from the article on FinOps and multi-tenancy with LiteLLM and the one on cost per token and per request: amortisation plus energy divided by tokens served. The example arithmetic, to set the order of magnitude: a cluster whose fully loaded cost is 9.60 USD per hour sustaining 3,000 output tokens per second produces 10.8 million tokens an hour, that is 0.00000089 USD per output token. That is the value that goes into output_cost_per_token. If the cluster’s average utilisation is 30 % and not 100 %, the divisor changes and the price triples, which is the same lever that article develops.

Before sending traffic, the /spend/calculate endpoint accepts the same parameters as completion_cost and returns the computed cost. It is the cheap way to check that the price resolves.

How that number reaches Langfuse, and who wins

The two integration routes write the cost into different fields. The native callback sends it as usage.total_cost of the v2 schema, and duplicates it in the trace metadata as litellm_response_cost. The OTLP preset emits langfuse.observation.cost_details as a JSON string of the form {"total": cost}, and langfuse.observation.usage_details with input, output and total.

The rule on the Langfuse side is documented and is the right one for this case: when there is ingested cost and inferred cost, the ingested one takes priority. Langfuse only computes from its own pricing table when the emitter sends no cost, and changes to a model definition apply to new generations, with no retroactive recalculation. A self-hosted model therefore needs no pricing definition in Langfuse: it is enough for LiteLLM to send cost_details.

One detail remains that explains a puzzling symptom. Langfuse stores the ingested cost and the computed one in separate columns, totalCost and calculatedTotalCost, and the session header totals add up only the first. A mixed deployment where part of the traffic brings cost and part lets Langfuse compute it shows sessions at zero while the individual generations show cost. There is an open issue about it.

The tokens of a streaming response

Almost all of an assistant’s traffic is streamed, and there the OpenAI usage object only appears if the client asks for stream_options.include_usage. LiteLLM solves this by injecting the option on its own: on the asynchronous completion routes with stream: true, if the caller did not set it and all the candidate deployments support it, it adds it and marks _litellm_strip_stream_usage so the usage block is used in the cost calculation and not returned to the client. The general_settings.always_include_stream_usage flag forces or disables that behaviour, and is worth setting to True when capability detection does not recognise the vLLM deployment.

When the usage is missing anyway, LiteLLM reassembles the response with stream_chunk_builder() and counts tokens with the tokeniser. It is an approximation, and it produces the second path towards a cost of zero. The two paths to zero are told apart by looking at total_tokens in LiteLLM_SpendLogs: zero tokens with a correct price is a usage problem, correct tokens with zero cost is a price problem.

Correlation: from the client to the engine

The operational question that justifies the whole setup is being able to take a user complaint, find their request and see where the time went. That requires the trace identifier to survive three hops.

What does work

If the client sends traceparent, LiteLLM’s spans hang off the existing trace. In addition, and independently of the OTel subsystem, litellm_pre_call_utils.py extracts by regular expression the 32-character hexadecimal trace-id and assigns it to data["litellm_trace_id"] as a last resort, and takes the session_id from the W3C baggage header. That makes the Langfuse trace identifier equal to the W3C trace identifier, which is the most robust correlation key available: with it, an identifier copied from the application log opens the trace in Langfuse with no translation.

For the routes that are not OTel v2, moreover, headers prefixed with langfuse_ inject per-request metadata. langfuse_trace_id, langfuse_trace_user_id and langfuse_trace_metadata are documented, but the code strips the prefix and writes the rest into the metadata, so any body key works as a header. Two warnings about this: header values arrive as strings and are not interpreted as JSON, and Nginx drops headers with underscores by default (underscores_in_headers off), which produces the symptom that the metadata works locally and disappears behind the ingress.

What does not work without effort

Towards the engine, propagation is off. The flag is litellm.forward_traceparent_to_llm_provider, it is False, and its implementation has three behaviours to know about before enabling it:

The function returns immediately if open_telemetry_logger is None. A deployment with callbacks: ["langfuse_otel"] and without the generic otel callback propagates nothing, whatever the flag is set to.
When it does propagate, it copies the client’s traceparent header as is, instead of injecting the context of LiteLLM’s active span. The vLLM span ends up hanging off the client’s span-id, a sibling of the gateway call’s span. The trace is one, the hierarchy is wrong.
If the client sends no traceparent, none is created.

Add to that the fact that LiteLLM’s general policy is not to forward client headers to the provider, and that the forward_client_headers_to_llm_api allowlist covers headers prefixed with x-, among which traceparent is not. This is the only route.

The vLLM side, and why the Collector stops being optional

vLLM enables traces with --otlp-traces-endpoint, and --collect-detailed-traces with model, worker or all adds forward and execute times, with a performance impact its own documentation warns about. The attributes it emits are in vllm/tracing/utils.py and are the most useful part of the setup: gen_ai.latency.time_in_queue, time_to_first_token, time_in_scheduler, time_in_model_prefill, time_in_model_decode, time_in_model_forward, time_in_model_execute, time_in_model_inference and e2e, plus the token counters. That breakdown is what turns a “it’s slow” into “the time goes in the queue, not in decode”, which is the conversation you want to have during an incident. It fits with what was already covered in instrumenting vLLM with OTel and with the metrics in DCGM and vLLM.

There are three constraints that decide the architecture:

vLLM exports over gRPC unless OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf is set, and its gRPC exporter is built with insecure=True hard-coded. Langfuse, for its part, does not support gRPC. And vLLM emits the old convention names (gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens), renamed in the specification to input_tokens and output_tokens, plus a gen_ai.latency.* family that was never standard and that the file itself documents as its own attributes until they are normalised.

An OpenTelemetry Collector in between solves all three at once: it receives gRPC, exports HTTP with the Basic header towards /api/public/otel/v1/traces, and renames attributes with a transform. It is also the only place where tail sampling can be done, which is the next point. And add the warning from the llm-d community, which holds for any deployment with Envoy or sidecars in between: OTEL_SERVICE_NAME has to be set per engine, because vLLM sends no service name and without it every span arrives as unknown_service.

The four serial queues

A request’s telemetry crosses four bounded buffers before it is queryable. All four drop once full and none blocks the request.

Queue	Capacity	What it does once full
LiteLLM logging thread pool	100 threads, 10,000 pending tasks	Drops the new ones with a rate-limited warning
Asynchronous `LoggingWorker`	50,000 events, concurrency 100	Aggressively flushes 50 % of the queue; `enqueue()` documents that it never blocks
OTel’s `BatchSpanProcessor`	2,048 spans (`OTEL_BSP_MAX_QUEUE_SIZE`), batch 512, sent every 5 s	Drops spans silently
Langfuse’s `ClickhouseWriter`	Bounded retries	Drops rows after exhausting retries, with no dead-letter queue

The first is documented in LiteLLM’s own code with a sentence that saves half an hour of discussion: logging is best-effort and, once the cap is reached, new entries are dropped with a rate-limited warning instead of being queued forever. The third is a barely visible consequence: LiteLLM builds the BatchSpanProcessor without adjusting max_queue_size, so the OTel SDK defaults govern, and no LiteLLM documentation mentions the OTEL_BSP_* variables that change them. The Langfuse SDK does the same on its own account: it sets batch size and interval, and leaves the queue size at the default.

On top of this comes a fifth loss point that is not a queue: LiteLLM does not call flush() per request on the native route, so a SIGKILL on the pod takes the last batch with it. On Kubernetes, this is an argument for a generous terminationGracePeriodSeconds and a preStop that gives some slack.

None of this is a defect. It is the right design decision for a piece that sits on the critical path of inference. What changes is what can be asserted with the resulting data.

What cannot be promised to the auditor

A dashboard built on this pipeline answers statistical questions well: what is the p95 of time to first token for the model group, which team consumes the most, what proportion of requests fails, how cost per request evolves. That is what it is designed for, and that is what it is good for.

It does not answer completeness questions well, which are the ones an ENS auditor or an ISO/IEC 42001 management system auditor asks. Faced with “show me that I have a record of every interaction with the AI system during March”, the honest answer is that the traces are a best-effort record with four documented points of silent dropping, with no dead-letter queue on the last hop and no durability guarantee on any of them.

The clean separation is the one already proposed in technical controls ENS, 42001 and the AI Act: the regulatory audit record is a different artefact, with a different write path, durability guarantees and retention of its own. In the pair this article deals with, the LiteLLM_SpendLogs table in Postgres is a much better candidate than Langfuse for that function, because it is a transactional write against a relational database, and because it already contains request_id, api_key in hashed form, team_id, organization_id, end_user, model, tokens, startTime, endTime, status and requester_ip_address. Writing the audit event from the same place and in the same transaction as the spend is the sensible route; delegating it to the tracing pipeline is not.

Here an operational tension appears that has to be resolved deliberately. LiteLLM’s performance recommendation for production includes disable_error_logs: true and, in very loaded deployments, disable_spend_logs: true, because provider errors inflate the table and concurrent writes against the same rows produce locks and exhaust the Postgres connections. Disabling spend_logs to gain performance and at the same time leaning on that table for the audit record are incompatible decisions. Above a thousand requests per second or more than ten instances, the way out is use_redis_transaction_buffer: true, which batches the spend updates in Redis before pushing them down to Postgres, watching the litellm_in_memory_spend_update_queue_size and litellm_redis_spend_update_queue_size metrics.

PII: switching off the content before it leaves

An LLM trace contains, by definition, the user’s prompt. In an on-premise deployment for a regulated customer, that content may be exactly what cannot leave the application’s domain, not even towards the observability backend in the same cluster.

The controls on the proxy side, from most global to most fine-grained:

litellm_settings.turn_off_message_logging: true cuts messages and responses in every callback, and keeps the metadata and the spend. The redaction replaces the content with the REDACTED_BY_LITELLM sentinel in messages, choices, reasoning_content, streaming deltas, tool call arguments and in the standard logging object.
The x-litellm-enable-message-redaction: true header enables redaction per request, and LiteLLM-Disable-Message-Redaction: true disables it. The second is checked first and wins over everything else, including the global configuration, which makes it a control to block at the ingress if the policy is that content is never logged.
redact_user_api_key_info: true removes the hashed token, the user_id and the team_id from the log.
Per request and per callback, metadata: {"mask_input": true, "mask_output": true}.
The body parameter "no-log": true disables logging of that call. It is a body parameter, not a header, and it can be disabled globally with global_disable_no_log_param: true so a client cannot make itself invisible.

On the OTel v2 route the policy is inverted towards the safe side: content is off by default and is opened with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, with values no_content, span_only, event_only and span_and_event. The documentation underlines that the gate is applied centrally and that a user’s request cannot force their prompt towards the backend while capture is disabled. It is also the stance the OpenTelemetry specification recommends, whose preferred pattern for production is to store the content in external storage and leave only the reference in the span.

One specific version to check in the inventory: below LiteLLM 1.98.0, with per-team Langfuse credentials, the team’s own credentials ended up inside its traces. That version changed the origin of the metadata emitted to the StandardLoggingPayload to cut it off. The same change removed a couple of dozen metadata fields (model_group, deployment, queue_time_seconds, attempted_retries and others), so it breaks saved dashboards and alerts that filter on them. It is an upgrade with work attached, and it has to be done anyway. The same applies to the general hardening of the stack, covered in hardening and secrets of the sovereign stack.

The price in latency

LiteLLM publishes two overhead figures that differ by a factor of thirty, and both are theirs.

The benchmarks page, with Locust, a thousand users with think time and a fake OpenAI endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms, at 1,170 requests per second. For two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures hold with around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, of July 2026, measures the Python proxy at 257.7 ms of added p99 with 329.5 MB of peak memory, against 0.7 ms for the Rust variant in beta. That measurement has no callbacks, no spend tracking and no persistence.

The operational reading is that the first figure is the best case and the second the order of magnitude to plan for agentic load, which is the one that arrives in a closed loop against fast responses. To watch it there are two always-on headers: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms, and the documentation sets the useful threshold of the second at 100 ms, above which the diagnosis is that the payloads are too large.

Three production settings that appear in LiteLLM’s documentation and that are wrong by default:

request_timeout is 6,000 seconds. A hundred minutes holding a connection against a hung upstream. The recommendation is to lower it to 600.

LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and the documentation puts the cost of that serialisation at 2 to 5 seconds with payloads over 2 MB. It is the first cause of latency their troubleshooting guide mentions.

Background jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every periodic job. The LITELLM_JOB_ROLE variable separates the pods that serve traffic from the replica that runs jobs.

And one failure mode with an open issue that deserves to be in the runbook: under sustained 429s from the upstream, pods have been reported unable to answer readiness probes during startup, with the ensuing restart loop. It fits with what was covered in incident response runbooks.

Start-up checklist

Ten checks before signing off the installation of the pair:

callbacks: ["langfuse_otel"] instead of success_callback: ["langfuse"], unless you depend on prompt management, which the OTLP preset does not support.
LANGFUSE_HOST defined explicitly. The preset’s default value points to the US region.
LiteLLM at 1.98.0 or higher, because of the team credential leak into the traces, and at 1.95.0 or higher because of the v4 ingestion header.
A price registered for every self-hosted model, per token and in USD, verified with /spend/calculate before opening traffic.
A test request with a known traceparent, checking that the same identifier appears in Langfuse.
An OpenTelemetry Collector between vLLM and Langfuse, with OTEL_SERVICE_NAME per engine and renaming of the old token attributes.
request_timeout: 600 and LITELLM_LOG out of DEBUG.
Tail sampling in the Collector, with a latency and status code policy, to keep 100 % of what is slow and what failed.
A written decision about prompt content, and blocking of the LiteLLM-Disable-Message-Redaction header at the ingress if the policy is not to log.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms and Langfuse’s ingestion queue depth (langfuse.queue.ingestion.depth), which is the autoscaling signal for its workers.

Traps and things that are not what they look like

The trace arrives, but ten minutes late. The x-langfuse-ingestion-version: 4 header is missing, which LiteLLM sends from 1.95.0. It is not a ClickHouse capacity problem.

The metadata works in development and disappears in production. Nginx drops headers with underscores by default, and all of this integration’s have one.

Session cost comes out as zero and generation cost does not. Langfuse adds up only the ingested cost in the session totals, not the one it computes itself.

Enabling OTel v2 breaks nothing visible and ungroups the conversations. The v2 mapper does not read session_id or trace_user_id from the request.

A team’s budget is never exhausted. Its spend is zero because the model has no price, and the calculation failure is only logged at debug level.

The alerts stop firing after the migration to v2. The success status becomes UNSET; no rule on OK is satisfied.

Forwarding the traceparent is enabled and it does not reach the engine. The generic otel callback is missing, without which the function that applies it returns before looking at the flag.

All the engines’ spans appear under unknown_service. vLLM emits no service name; OTEL_SERVICE_NAME has to be set per deployment.

Closing

The gateway and the tracing backend are the two pieces of the inference stack that take the least time to install and the longest to operate well. The installation fits in a ConfigMap. The operation consists of four decisions no getting-started guide raises: which integration route to take knowing that one expires in November, where the price per token of a model nobody invoices comes from, what to do with the traceparent at each hop, and what to tell an auditor about data the system drops by design when things get tight.

The answer to the fourth is the one that orders the other three. This pipeline is operational instrumentation, with statistical precision and best-effort guarantees, and separating it from day one from the regulatory audit record avoids the awkward conversation two years from now. With that separation made, the LiteLLM and Langfuse pair is the pair of pieces that turns a fleet of GPUs into a platform with an owner, a known cost and explainable latency.

Sources

LiteLLM, Langfuse integration (rutas de integración, metadata aceptada, cabeceras langfuse_*): https://docs.litellm.ai/docs/observability/langfuse_integration.
LiteLLM, OpenTelemetry v2 y Migración a OTel v2 (preset, captura de contenido, cambios de semántica): https://docs.litellm.ai/docs/observability/opentelemetry_v2 · https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.
LiteLLM, Logging y Config settings (redacción, turn_off_message_logging, variables de entorno): https://docs.litellm.ai/docs/proxy/logging · https://docs.litellm.ai/docs/proxy/config_settings.
LiteLLM, Custom pricing y Spend tracking (claves de precio, SpendLogs, endpoints de gasto): https://docs.litellm.ai/docs/proxy/custom_pricing · https://docs.litellm.ai/docs/proxy/cost_tracking.
LiteLLM, Production best practices (topología, request_timeout, buffer de transacciones en Redis, LITELLM_JOB_ROLE): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Benchmarks y Latency overhead (cifras de sobrecarga y cabeceras de diagnóstico): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, Rust AI Gateway benchmarks (la cifra de 257,7 ms de p99 del proxy Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, notas de versión v1.95.0, v1.98.0 y v1.100.0: https://docs.litellm.ai/release_notes/v1.95.0/v1-95-0 · https://docs.litellm.ai/release_notes/v1.98.0/v1-98-0 · https://docs.litellm.ai/release_notes/v1.100.0/v1-100-0.
Langfuse, OpenTelemetry (endpoint, autenticación, tabla de mapeo de atributos, ausencia de gRPC): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, Compatibility y Migrate v3 to v4 (la fecha del 16 de noviembre de 2026 y los modos de escritura): https://langfuse.com/docs/compatibility · https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4.
Langfuse, Token and cost tracking (prioridad del coste ingerido sobre el inferido): https://langfuse.com/docs/observability/features/token-and-cost-tracking.
Langfuse, Scaling y Sampling (dimensionado, señal de autoescalado, muestreo de cabecera): https://langfuse.com/self-hosting/configuration/scaling · https://langfuse.com/docs/observability/features/sampling.
Langfuse, issue #13468 — descarte de filas en ClickhouseWriter sin cola de mensajes muertos: https://github.com/langfuse/langfuse/issues/13468. Issue #15109 — coste de sesión a cero: https://github.com/langfuse/langfuse/issues/15109. Issue #5173 — Nginx y cabeceras con guión bajo: https://github.com/langfuse/langfuse/issues/5173.
vLLM, vllm/tracing/utils.py y vllm/tracing/otel.py en la etiqueta v0.28.0 (atributos emitidos, protocolo por defecto, insecure=True): https://github.com/vllm-project/vllm/blob/v0.28.0/vllm/tracing/utils.py.
Langfuse, vLLM integration (la advertencia de que vLLM solo exporta contadores y latencias): https://langfuse.com/integrations/model-providers/vllm.
LiteLLM, litellm/constants.py y litellm/litellm_core_utils/logging_worker.py (capacidades de las colas y política de descarte): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
OpenTelemetry, Semantic Conventions for Generative AI y el repositorio semantic-conventions-genai (estado de desarrollo, renombrados, captura de contenido): https://github.com/open-telemetry/semantic-conventions-genai.
OpenTelemetry Collector, tailsamplingprocessor (políticas de muestreo por cola y la restricción de instancia única por traza): https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md.
llm-d, End-to-end and fine-grained tracing in llm-d (OTEL_SERVICE_NAME y los spans huérfanos tras Envoy): https://llm-d.ai/blog/end-to-end-and-fine-grained-tracing-in-llm-d.

Full TCO of an on-premise GPU cluster: from capex to the all-in €/GPU-hour and the break-even against cloud

Tue, 16 Jun 2026 06:00:00 +0200

Notation: amounts in N € or N USD (source denominated in dollars); decimal point; comma as thousands separator. The dollar sign is not used (it is the formula delimiter). Data centred on Europe/Spain. Generic example hardware: a cluster of N nodes, each with 4×H100 SXM5 80 GB.

TL;DR

A 4×H100 SXM5 node costs between 150,000 USD and 200,000 USD in total capex (GPUs + server + network + storage + prorated rack). Amortised over 3 years with European opex (energy at ~0.116 €/kWh industrial, PUE 1.54 average or 1.2 with liquid, 0.3 FTE of staff), the all-in cost ranges from 3.10 USD/GPU-hour (100 % utilisation) to 6.20 USD/GPU-hour (50 % utilisation). The break-even against AWS p5 on-demand (~6.88 USD/GPU-hour) is crossed at around 70 % utilisation; against a neocloud 3-year reserved (~1.49–2.10 USD/GPU-hour), on-prem never closes the gap in that scenario. Utilisation is the variable that decides the cost axis, not the price of the hardware.

The model: declared assumptions

Every calculation below starts from these assumptions. Changing any of them moves the conclusion; the sensitivity section quantifies by how much.

Parameter	Base value	Sensitivity range
Node	4×H100 SXM5 80 GB (HGX baseboard)	—
Cluster	N nodes (per-node model; scales linearly)	1–32 nodes
Capex amortisation	3 years (straight line)	3–5 years
GPU utilisation	70 %	30 %–100 %
Energy price	0.116 €/kWh (industrial Spain, Sept. 2025)	0.06–0.20 €/kWh
PUE	1.54 (global average, Uptime Institute 2025)	1.15–1.80
Exchange rate	1 USD = 0.93 € (reference Jun. 2026)	—

Energy source: GlobalPetrolPrices · Spain Business Electricity, Sept. 2025. PUE source: Uptime Institute Global Data Center Survey 2025 — the global average PUE has been stuck at 1.54 for the sixth consecutive year; hyperscalers 1.10–1.15; colocation/enterprise 1.58–1.80; facilities less than 5 years old, 1.48. PUE 1.2 is achievable with direct-to-chip liquid cooling.

Capex breakdown per 4×H100 SXM5 node

GPUs

Component	Unit price (USD)	Quantity	Subtotal (USD)	Source and date
H100 SXM5 80 GB (card)	30,000–40,000	4	120,000–160,000	GMI Cloud, Apr. 2026 · Introl, Apr. 2026

The range reflects market variability and volume discounts (5–15 % for orders above 50 units). H100 SXM5 cards require NVIDIA’s HGX baseboard; they are not sold loose for direct installation in standard servers.

Server / HGX baseboard

A complete 4×H100 SXM node uses NVIDIA’s HGX H100 4-GPU baseboard plus a compatible host server. Reference models: Supermicro SYS-421GU-TNXR (4U, dual Intel Xeon 4th Gen, HGX H100 4-GPU) and its Dell equivalent.

Component	Estimated cost (USD)	Note
Server chassis + CPU (2× Xeon) + RAM (512 GB DDR5) + redundant PSU	18,000–25,000	Based on a Supermicro SYS-821GE bare without GPUs at ~24,806 USD (xicomputer.com, Jun. 2026); scaled to the 4-GPU
HGX H100 4-GPU baseboard	included in the GPU price	NVIDIA HGX platform; no separate public price
Inter-GPU NVLink (inside the node)	included in the baseboard	4 GPUs connected by NVLink 4.0 on the HGX baseboard

Marketing claim (no independent verification): Supermicro announces datacenter energy cost reductions of up to 40 % with liquid cooling in its HGX H100 servers (Supermicro press release).

InfiniBand NDR network

For a multi-node cluster with tensor parallelism across nodes, the GPU-to-GPU network is critical. NDR InfiniBand (400 Gb/s per port) is the de facto standard for HGX clusters.

Component	Estimated cost per node (USD)	Source / Note
NVIDIA Quantum-2 NDR 400G switch (64 ports, prorated across N nodes)	2,000–4,000	Switch ~35,000 USD (Introl, Apr. 2026); at 16 nodes, ~2,200 USD/node
InfiniBand NDR cables/transceivers (4 ports per node × ~1,000 USD/port)	4,000	Estimate based on ~1,000 USD per optical transceiver (Introl, Apr. 2026)
InfiniBand network (prorated per 4-GPU node)	~6,000–8,000	—

For inference serving inside a single node (4 GPUs with NVLink), the inter-node network is less critical than for multi-node training. For disaggregated prefill-decode workloads spread across nodes, InfiniBand NDR is necessary.

NVMe storage

Component	Estimated cost (USD)	Note
Local NVMe (4 TB × 2 U.2/E1.S drives, working datasets and checkpoints)	2,000–4,000	~500–1,000 USD/TB enterprise NVMe, 2025
Shared object storage (NAS/MinIO, prorated per node)	2,000–5,000	Varies with the total capacity of the cluster
Total storage per node	~4,000–9,000	—

Introl models 50 TB per GPU for effective operation in training clusters (Introl, Apr. 2026); for pure inference the requirement is significantly lower (model weights + logs).

Rack, PDU and datacenter connectivity

Component	Estimated cost per node (USD/year)	Source
Rack colocation (high density, 10–15 kW per node)	5,000–12,000/year	Encoradvisors · Colocation Pricing 2026: high density 3,000–6,000 USD/month per rack; at 2 nodes per rack, ~1,500–3,000 USD/month per node = 18,000–36,000 USD/year in tier-1; lower in Spain
Rack PDU, electrical cabling (prorated)	500–1,000 per node (amortised capex)	Inside the colocation line item or in an own datacenter

Colocation in Spain/Europe is structurally cheaper than in US tier-1 markets (New York, Silicon Valley). For an own datacenter, replace it with the cost of your own space plus amortisation of the electrical and cooling infrastructure.

Capex summary per 4×H100 SXM5 node

Line item	Range (USD)	Midpoint
GPUs (4× H100 SXM5)	120,000–160,000	140,000
Server chassis + CPU/RAM/PSU	18,000–25,000	21,500
InfiniBand NDR network (prorated)	6,000–8,000	7,000
NVMe + object storage	4,000–9,000	6,500
PDU/rack/other (capex)	2,000–5,000	3,500
Total capex per node	150,000–207,000	178,500

Sources: GMI Cloud (Apr. 2026), Introl (Apr. 2026), Spheron (Apr. 2026), xicomputer.com (Jun. 2026).

Opex breakdown per 4×H100 SXM5 node (annual)

Energy

A 4×H100 SXM5 node at full load draws approximately:

$$P_{\text{node}} = 4 \times 700\,\text{W (TDP H100 SXM5)} + 800\,\text{W (server)} \approx 3.6\,\text{kW (IT)}$$

The total datacenter power includes the cooling overhead, expressed by the PUE:

$$P_{\text{total}} = P_{\text{IT}} \times \text{PUE}$$ $$\text{annual energy cost} = P_{\text{IT}} \times \text{PUE} \times 8{,}760\,\text{h} \times \text{kWh price}$$

With the base values (PUE 1.54; 0.116 €/kWh):

$$\text{energy/year} = 3.6\,\text{kW} \times 1.54 \times 8{,}760\,\text{h} \times 0.116\,\text{EUR/kWh} \approx 5{,}475\,\text{EUR}$$

With a Spanish solar PPA (Q3 2025 reference price: ~34 €/MWh = 0.034 €/kWh according to PV Tech, Oct. 2025):

$$\text{energy/year (solar PPA)} = 3.6 \times 1.54 \times 8{,}760 \times 0.034 \approx 1{,}604\,\text{EUR}$$

Energy scenario	Price (€/kWh)	Energy cost/year per 4-GPU node
Spanish solar PPA (Q3 2025)	0.034	~1,604 €
Industrial Spain (Sept. 2025)	0.116	~5,475 €
European average (industrial tariff)	0.160	~7,550 €
Worst case (no PPA, high tariff)	0.200	~9,437 €

Staff / operations

Staff cost is the most variable line item with cluster size. For a small cluster (2–8 nodes), the rule of thumb is 0.3–0.5 FTE per cluster of GPU infrastructure support (Spheron, Apr. 2026).

Cluster size	Estimated FTE	FTE cost (€/year, Western Europe)	Cost per 4-GPU node (€/year)
2–4 nodes	0.3 FTE	~120,000	36,000–18,000
8–16 nodes	0.5 FTE	~120,000	7,500
32+ nodes	1–2 FTE	~120,000	3,750–7,500

Indicative salary reference: a GPU infrastructure engineer with CUDA, InfiniBand and Kubernetes knowledge in Western Europe, ~90,000–140,000 €/year fully loaded. Introl’s figures (Apr. 2026) in USD (~275,000 USD/year for the US) reflect the North American market, which is appreciably higher.

Maintenance and support

Line item	Annual cost (% of hardware capex)	Per 4-GPU node (midpoint)
Vendor maintenance / support	5–10 % of capex	~7,000–14,000 USD → ~6,500–13,000 €
GPU failure rate (~5 % annual) × replacement cost	5 % × 4 GPUs × ~35,000 USD = ~7,000 USD expected	~6,500 € (amortised as a provision)
Minor spares (cables, modules)	~500–1,000 €	—

Introl quotes GPU failure rates of 2–3 % per year in small clusters; Google Research documented ~9 % annualised in Meta’s 16,384-GPU H100 cluster (Introl, Apr. 2026). 5 % is used here as a conservative intermediate value.

Depreciation (for accounting purposes)

Straight-line depreciation turns capex into an annual flow comparable to the cost of committed cloud:

$$\text{annual depreciation} = \frac{\text{node capex}}{\text{amortisation years}}$$

Node capex (USD)	3-year amortisation (USD/year)	5-year amortisation (USD/year)
150,000 (minimum)	50,000	30,000
178,500 (average)	59,500	35,700
207,000 (maximum)	69,000	41,400

H100 hardware depreciates fast: secondary-market analyses put the residual value at 20–40 % of the purchase price after 3 years (Introl, Apr. 2026). The arrival of Blackwell GB200/GB300 accelerates the perceived obsolescence.

Annual opex summary per 4×H100 SXM5 node (base scenario, 8-node cluster)

Line item	Base scenario (€/year)	Range
Energy (PUE 1.54; 0.116 €/kWh)	5,475	1,604–9,437
Staff (0.5 FTE × 8 nodes, prorated)	7,500	3,750–36,000
Maintenance / support / failures	9,000	5,000–15,000
Rack colocation (Spain, high density)	6,000	3,000–15,000
Total opex per node	~28,000	~13,000–75,000

The extreme range reflects the difference between a well amortised own datacenter with a solar PPA and cheap energy (minimum opex) and tier-1 colocation with market tariffs and senior staff.

Deriving the all-in €/GPU-hour

Formula

$$\text{EUR/GPU-hour all-in} = \frac{\frac{\text{node capex}}{\text{years}} + \text{node annual opex}}{4\,\text{GPUs} \times 8{,}760\,\text{h} \times u}$$

where $u$ is the average annual utilisation (0 to 1).

See the cost-per-token identity in cost per token and per request for the connection with throughput.

€/GPU-hour table by utilisation and scenario

Average capex (178,500 USD → ~166,000 €), 3-year amortisation → 55,300 €/year.

Utilisation	Opex/year (base, €)	Total cost/year (€)	Useful GPU-hours/year	€/GPU-hour
30 %	28,000	83,300	10,512	7.93
50 %	28,000	83,300	17,520	4.75
70 %	28,000	83,300	24,528	3.39
80 %	28,000	83,300	28,032	2.97
100 %	28,000	83,300	35,040	2.38

Low-opex scenario (solar PPA, own datacenter, large cluster): opex/year ~13,000 €.

Utilisation	Total cost/year (€)	€/GPU-hour
50 %	68,300	3.90
70 %	68,300	2.78
80 %	68,300	2.43
100 %	68,300	1.95

High-opex scenario (market tariff, expensive colocation, small cluster): opex/year ~75,000 €.

Utilisation	Total cost/year (€)	€/GPU-hour
50 %	130,300	7.44
70 %	130,300	5.31
80 %	130,300	4.65
100 %	130,300	3.72

From €/GPU-hour to €/1M tokens

The cost-per-token identity connects the hardware cost with the inference cost:

$$\text{EUR/1M tokens} = \frac{\text{EUR/GPU-hour} \times 10^6}{\text{throughput (tok/s)} \times 3{,}600}$$

For reference throughputs on H100 SXM5 with vLLM (see capacity planning for on-premise inference):

Model	Typical throughput (tok/s per GPU)	Source
Llama-3 70B FP8, high batch	~2,800	Series B benchmarks
Llama-3 8B FP16, medium batch	~9,000	Series B benchmarks
Mixtral 8×7B, high batch	~4,500	Series B benchmarks

€/1M tokens table in the base scenario (€/GPU-hour 3.39 at 70 % utilisation):

Model	Throughput (tok/s)	€/1M tokens
Llama-3 70B FP8	2,800	~0.336
Llama-3 8B FP16	9,000	~0.105
Mixtral 8×7B	4,500	~0.209

At 50 % utilisation (€/GPU-hour 4.75):

Model	€/1M tokens
Llama-3 70B FP8	~0.471
Llama-3 8B FP16	~0.147

Occupancy (batching) multiplies the effective throughput and lowers the €/1M tokens without changing the hardware; it is analysed in GPU utilisation as a FinOps lever.

Break-even on-prem vs cloud

The break-even formula

Break-even happens when the total annual on-prem cost equals the annual cloud cost at the same utilisation:

$$\text{annual cloud cost} = \text{cloud GPU-hour price} \times 4\,\text{GPUs} \times 8{,}760\,\text{h} \times u$$ $$\text{break-even}: \quad \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times u} = \text{cloud GPU-hour price}$$

Solving for the break-even utilisation:

$$u^* = \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times \text{cloud GPU-hour price}}$$

Break-even table by cloud mode and on-prem scenario

Base on-prem scenario (capex/year 55,300 €, opex/year 28,000 €, total 83,300 €/year per 4-GPU node):

Cloud reference (price/GPU-hour)	USD equiv.	Break-even utilisation $u^*$	Note
Neocloud on-demand (Lambda/Spheron ~2.90 USD)	2.90 USD (~2.70 €)	>100 %, on-prem does not compete	Neocloud on-demand is cheaper even at full utilisation
Neocloud 3-year reserved (CoreWeave ~1.49–2.10 USD)	~1.80 USD (~1.67 €)	>100 %, impossible	Neocloud reserved beats on-prem in every scenario of this model
AWS p5 on-demand (6.88 USD/GPU-hour)	6.88 USD (~6.40 €)	~47 %	Above 47 %, the average on-prem beats AWS on-demand
AWS p5 3-year reserved (~2.97 USD/GPU-hour)	2.97 USD (~2.76 €)	>100 %	—
GCP A3 on-demand (~10.98 USD/GPU-hour)	10.98 USD (~10.21 €)	~29 %	Above 29 %, on-prem beats GCP on-demand
Azure ND H100 v5 on-demand (~12.29 USD/GPU-hour)	12.29 USD (~11.43 €)	~26 %	—

Low-opex scenario (total 68,300 €/year):

Cloud reference	Break-even utilisation
AWS p5 on-demand (6.88 USD ≈ 6.40 €)	~38 %
Neocloud on-demand (2.90 USD ≈ 2.70 €)	~91 %
Neocloud 3-yr reserved (1.80 USD ≈ 1.67 €)	>100 %

High-opex scenario (total 130,300 €/year):

Cloud reference	Break-even utilisation
AWS p5 on-demand (6.88 USD ≈ 6.40 €)	~72 %
GCP A3 on-demand (~10.21 €)	~45 %
Azure on-demand (~11.43 €)	~41 %

Reading the break-even table

Against neoclouds (on-demand or reserved), on-prem TCO does not reach break-even in any scenario of the base model. Neocloud reserved beats on-prem even at 100 % utilisation, because its hourly price is below the all-in cost of your own hardware. This is consistent with the analysis in cloud GPU: on-demand, reserved and spot.
Against on-demand hyperscalers (AWS, GCP, Azure), on-prem does have an achievable break-even: around 26–72 % utilisation depending on the scenario. At medium-high utilisation (>70 %), on-prem clearly beats AWS/GCP/Azure on-demand.
The variable that moves the break-even most is opex (staff above all), not hardware capex. A well sized cluster in cheap colocation with PPA energy can lower the threshold by 20 percentage points compared with the high scenario.
For GDPR data, the break-even against US hyperscalers is skewed: the sovereignty axis rules out US hyperscalers before cost does (see sovereign on-premise vs hyperscalers).

Sensitivity analysis

TCO vs utilisation

The all-in cost per GPU-hour varies inversely with utilisation because capex is fixed:

$$\frac{d(\text{EUR/GPU-hour})}{du} = -\frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times u^2} < 0$$

Going from 50 % to 80 % utilisation cuts the €/GPU-hour by $\frac{4.75 - 2.97}{4.75} \approx 37\,\%$ in the base scenario. That 37 % reduction requires no hardware change, only more efficient scheduling (see GPU utilisation as a FinOps lever).

Utilisation	€/GPU-hour (base scenario)	Change vs 50 %
30 %	7.93	+67 %
50 %	4.75	reference
70 %	3.39	−29 %
80 %	2.97	−37 %
100 %	2.38	−50 %

TCO vs energy price

Energy price (€/kWh)	Energy opex/year	€/GPU-hour (70 % util.)	Change vs base
0.034 (solar PPA)	1,604 €	3.00	−12 %
0.116 (industrial ES, base)	5,475 €	3.39	reference
0.160 (European average)	7,550 €	3.54	+4 %
0.200 (high tariff)	9,437 €	3.67	+8 %

Energy has a moderate impact on total TCO (8–12 % variation between the extremes), because hardware capex dominates. Over a very long amortisation (5 years) with a solar PPA, however, energy drops from 6 % to 1 % of total TCO and the differential is amplified. The price of energy matters more for the carbon footprint (CSRD) than for TCO when capex is dominant.

TCO vs PUE

PUE	Cooling overhead	Energy/year (0.116 €/kWh)	€/GPU-hour (70 % util.)
1.15 (liquid cooling, new facilities)	+15 %	2,166 €	3.21
1.20 (liquid, modern datacenter)	+20 %	2,259 €	3.23
1.48 (facilities <5 years, Uptime 2025)	+48 %	3,490 €	3.33
1.54 (global average, Uptime 2025)	+54 %	3,627 €	3.39
1.80 (legacy colocation)	+80 %	4,260 €	3.47

The difference between PUE 1.15 (liquid) and 1.80 (legacy) is barely ~8 % of the €/GPU-hour at 70 % utilisation, because energy is only a fraction of TCO. PUE matters far more for the absolute energy cost and for CSRD reporting than for total TCO when hardware is the dominant component.

TCO vs amortisation years

Amortisation	Capex/year (average node, USD)	€/GPU-hour (70 % util., base opex scenario)
3 years	59,500 USD (~55,300 €)	3.39
4 years	44,625 USD (~41,500 €)	2.99
5 years	35,700 USD (~33,200 €)	2.72

Stretching the amortisation from 3 to 5 years lowers the €/GPU-hour by ~20 %, assuming the hardware remains competitive and the resale market supports the residual value. With the refresh cycle accelerated by Blackwell GB200/GB300, a 5-year amortisation carries a higher risk of technological obsolescence.

Sensitivity heat map (€/GPU-hour at 70 % utilisation, base scenario)

	PUE 1.15	PUE 1.54	PUE 1.80
3-yr amort., solar PPA (0.034 €)	2.72	2.74	2.76
3-yr amort., industrial (0.116 €)	3.21	3.39	3.47
5-yr amort., industrial (0.116 €)	2.54	2.72	2.80
3-yr amort., high tariff (0.200 €)	3.44	3.67	3.78

Decision table: cost/control/sovereignty Pareto

The table below crosses the four dimensions with no implicit hierarchy; the ordinal reading depends on each organisation’s constraints.

Option	€/GPU-hour	Initial capex	Full stack control	EU sovereignty	Elasticity	Operational risk
On-prem (util. >70 %, low opex)	2.40–3.00	high (150–207k USD/node)	total	total	none	hardware failure, idle
On-prem (util. <50 %, base opex)	4.75–7.93	high	total	total	none	capex with no return
Neocloud 3-year reserved (CoreWeave, Lambda)	1.49–2.10 USD	none	partial (API)	depends on the provider	rigid contract	minimal interruption
Neocloud on-demand (Lambda, Spheron)	2.49–3.44 USD	none	partial	depends	total	no interruption
AWS p5 on-demand	6.88 USD	none	minimal	NO (CLOUD Act)	total	no interruption
AWS p5 3-year reserved	~2.97 USD	financial commitment	minimal	NO (CLOUD Act)	rigid	no interruption
Sovereign EU cloud (Scaleway, Nebius EU)	2.15–3.85 USD	none	partial	yes (EU)	total	no interruption
Hybrid on-prem base + EU cloud peak	2.00–3.50 (weighted)	medium	high	yes (EU)	elastic peak	operational complexity

“EU sovereignty” column: US hyperscalers (AWS, GCP, Azure) are subject to the US CLOUD Act regardless of the datacenter region. Nebius has a Dutch legal entity; CoreWeave is a US company. See the full analysis in sovereign on-premise vs hyperscalers.

“Full stack control” column: on-prem lets you choose the driver version, the kernel, the NCCL configuration, MIG partitioning, and any system parameter. Cloud options offer control at container/pod level, with the hypervisor and firmware opaque.

The cost/sovereignty Pareto frontier for GDPR data excludes US hyperscalers, leaving: on-prem, sovereign EU cloud, and the hybrid. Among those three, the deciding variable is sustained utilisation and the predictability of traffic (see capacity planning for on-premise LLM inference).

Integration with the series FinOps model

The all-in €/GPU-hour of on-prem is the number that feeds the cost allocation pipeline of the series:

Cost-per-token identity (cost per token and per request): engine throughput × €/GPU-hour → €/1M tokens.
Chargeback and showback (chargeback and showback in GPU multitenancy): the all-in €/GPU-hour is the internal price charged to each tenant of the multi-tenant cluster.
Utilisation as a lever (GPU utilisation as FinOps): raising utilisation from 50 % to 80 % cuts the €/GPU-hour by 37 % without changing the hardware, the highest ROI in on-prem FinOps.
Capacity planning (capacity planning for on-premise LLM inference): the number of nodes to buy depends on the percentile of base load you want to cover in iron.
Cloud comparison (cloud GPU: on-demand, reserved and spot): the all-in €/GPU-hour goes head to head with the cloud price in table A7 to compute the break-even.

Sources

Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis (abr. 2026) — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
Introl · GPU Infrastructure TCO Model: 5-Year Cost Analysis for Enterprise AI (abr. 2026) — https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model
GMI Cloud · NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis (abr. 2026) — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Uptime Institute · Global Data Center Survey 2025 (PDF oficial) — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf
Uptime Institute · Global Data Center PUE Stalls at 1.54 (comunicado, oct. 2025) — https://mgrid.org/2025/10/01/uptime-institute-data-center-pue-stagnation-2025-liquid-cooling/
GlobalPetrolPrices · Spain Business Electricity Price (sept. 2025) — https://www.globalpetrolprices.com/Spain/electricity_prices/
PV Tech · European Solar PPA Prices Fall Below 35 €/MWh in Q3 2025 (oct. 2025) — https://www.pv-tech.org/european-solar-ppa-prices-fall-below-35-mwh-q3-2025/
Xi Computers · Supermicro SYS-821GE-TNHR 8U GPU Server Pricing (jun. 2026) — https://www.xicomputer.com/Solutions/Data-Center-Servers/SYS-821GE-TNHR.asp
Encoradvisors · Data Center Colocation Pricing 2026 — https://encoradvisors.com/data-center-colocation-pricing/
CloudZero · H100 GPU Cost In 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/
Spheron · AI Inference Power Consumption and GPU Electricity Costs: 2026 Guide — https://www.spheron.network/blog/ai-inference-power-electricity-cost-2026/
NVIDIA · DGX SuperPOD H100 Electrical Specifications (documentación oficial) — https://docs.nvidia.com/dgx-superpod/design-guides/dgx-superpod-data-center-design-h100/latest/electrical.html

Sovereign on-premise vs hyperscalers: the case with data (cost, energy, performance and sovereignty)

Sun, 14 Jun 2026 05:30:00 +0200

Notation: amounts in euros (N €), decimals with a point; when a source quotes dollars it is marked “USD”. Data centred on Europe and Spain. The dollar symbol is not used (formula delimiter).

What this article covers

Synthesis article of the series (S2), and the heart of the proposal: the case with data for sovereign on-premise against the hyperscalers and the European cloud. Up to here, each track measured its own axis, cost per token (FinOps), goodput (benchmarking), energy and carbon; here the four are crossed, with a fourth dimension no US technical comparison puts up front: data sovereignty. The aim is to answer, with numbers and not with ideology, the question that underpins any investment in an AI platform: serve on your own iron, on a European cloud or on a hyperscaler? And to do it honestly: on-prem does not always win, and saying when it wins and when it does not is what makes the recommendation credible.

The framework: four axes, not one number

The mistake in almost every comparison is reducing the decision to the hourly cost of a GPU. The real decision crosses four axes, and only by seeing them together do you decide well:

Axis	Question	Who measures it in the series
Cost (TCO)	how much does it cost to serve, all in?	FinOps (A2–A8)
Performance	does it meet the SLO, at what goodput?	Benchmarking (B2–B8)
Energy and carbon	how many watts and grams per token?	Energy (C2–C8)
Sovereignty	under which jurisdiction does the data live?	GDPR / EU AI Act / CSRD

The first three are quantifiable and meet in the cost per token; the fourth is a constraint that can rule out an option however cheap it is. The synthesis consists of scoring each option on all four and deciding on the Pareto frontier, not on whichever axis happens to suit.

The cost axis: TCO and break-even

The real on-premise cost is amortised capex plus opex, and its cost per effective hour depends on utilisation:

$$\text{effective cost/GPU-hour (on-prem)} = \frac{\text{annual amortised capex} + \text{annual opex}}{8760 \times \text{utilisation}}$$

This formula is the key to the whole debate: the cost per useful hour of on-prem rises as utilisation falls, because capex is paid whether the GPU is working or idle. The 2026 data:

Data point	Value	Source
On-prem cost 8×H100 (floor, high util.)	~2.83 USD/GPU-hour all-in	Spheron
Annual on-prem cost (floor)	~237,000 USD/year	Spheron
AWS H100 (p5.48xlarge)	4.10–6.88 USD/GPU-hour	Spheron
AWS 8-GPU on-demand annual (100 % util)	287,000–482,000 USD/year	Spheron
Sovereign European cloud (Lyceum/Scaleway)	from 2–2.73 €/GPU-hour, zero-egress	Lyceum, Scaleway

The break-even against AWS on-demand falls around 50–83 % utilisation depending on region and tariff; below ~70 % utilisation, the cloud wins on TCO, and above it, on-prem (Spheron). For very high-utilisation workloads, on-prem pays for itself in less than 4 months (Lenovo).

Three-year TCO: the full calculation for an 8×H100 node

Abstract numbers do not convince a committee; a three-year model with declared line items does. Take a sovereign 8×H100 node in Spain and compare it, for the same work, with AWS and with a European cloud. Declared assumptions: amortisation over 3 years, energy at a solar PPA of 32.5 €/MWh (with grid backup), PUE 1.3, and two utilisation scenarios (50 % and 80 %).

On-premise (own node), annual line items:

Line item	Annual value	Note
Amortised capex (node ~270,000 € / 3 years)	~90,000 €	8×H100 server + network + storage
Energy (≈10.4 kW × PUE 1.3 × 8760 h)	~3,850 € (at 32.5 €/MWh)	with solar PPA; at grid tariff, ~12–18 k €
Operation, cooling, maintenance	~25,000 €	prorated staff, support, spares
Datacenter space (rack, connectivity)	~12,000 €	colocation or own datacenter
Annual total	~131,000 €	independent of utilisation

At a fixed 131,000 €/year, the cost per token depends only on how many tokens you generate, that is, on utilisation:

Utilisation	Useful GPU-hours/year	Cost/1M tokens
50 %	~35,000	~2.9 €
65 %	~45,500	~2.2 €
80 %	~56,000	~1.8 € (with cheap grid, ~1.1 €)

Sovereign European cloud (Scaleway/Lyceum), pay per use: at ~2.2 €/GPU-hour with zero-egress, the cost per token is constant with utilisation (you only pay for what you use): ~1.5–2.2 €/1M tokens depending on model and batching, with no capex and no idle risk.

Hyperscaler (AWS p5), on-demand: at 4.10–6.88 USD/GPU-hour (≈3.8–6.4 €), the cost per token works out at ~2–3.5 €/1M tokens, and on top of that you have to add the egress, quite apart from the fact that for GDPR data the sovereignty axis rules it out already.

The reading of the model is the thesis of the whole of S2: at 50 % utilisation, on-prem (~2.9 €) does not beat the European cloud (~1.8 €); at 80 % with cheap energy (~1.1 €), it beats it comfortably. The crossover sits, as the literature says, around 65–70 %. Investing in on-prem is, fundamentally, a bet that you will sustain high utilisation, and that bet is won with scheduling, not with hardware.

The uncomfortable reality: the utilisation almost nobody reaches

Here is the honest figure missing from the “on-prem is always cheaper” speeches: most inference teams in production operate at 40–65 % GPU utilisation, because of traffic variability and the limits of batching; the 80–90 % assumption that makes on-prem attractive is rarely reached outside batch-only pipelines (Spheron).

This changes the naive conclusion: if your real utilisation is 50 %, on-prem is not cheaper than the cloud, because the capex you pay for the idle GPU eats you. That is why utilisation is not a detail, it is the variable that decides the cost axis, and it connects directly with the FinOps track (the idle of A2, the chargeback of A5) and with scheduling: raising utilisation is what makes on-prem pay. A badly used cluster of your own is more expensive than the cloud; a well scheduled one is far cheaper. The cost question is not “on-prem or cloud?”, it is “can I sustain high utilisation?”.

The hidden costs of the cloud: egress

The cloud has its own small print: egress costs (taking data out of the provider). In the hyperscalers, moving data out or between regions is billed, and in AI workloads with a lot of data movement (datasets, checkpoints, embeddings) it can be a significant line item that does not appear in the GPU-hour price. The advantage of the sovereign European cloud: most of them (Lyceum among others) have adopted the zero-egress model, they do not charge for moving data out or between regions (Lyceum). When comparing, the real hyperscaler cost is GPU-hour + egress + other charges, not just the GPU-hour; ignoring it artificially inflates the hyperscaler’s competitiveness.

An example of the order of magnitude: a platform moving 50 TB/month outbound (datasets, checkpoints, responses served to systems outside the provider) at a typical egress tariff of ~0.08–0.09 €/GB pays ~4,000–4,500 €/month, that is ~50,000 €/year on egress alone, a line item the size of a third of the cost of a node of your own, invisible in the GPU-hour price. On the European cloud with zero-egress that line item is zero; on-prem, internal traffic is not billed either. That is why a fair comparison must model egress according to the real data pattern: for workloads with a lot of outbound movement, it can invert the ranking between hyperscaler and European cloud. The cloud bill is not the GPU-hour; it is the GPU-hour plus everything you move.

To this is added contract and lock-in risk: the hyperscaler’s on-demand GPU tariffs can change, commitment discounts (reserved/savings plans) tie you for 1–3 years, and migrating out, because of egress and coupling to proprietary services, has a real exit cost. On-prem and the European cloud with standard APIs (Kubernetes, S3 compatible) reduce that coupling: the same manifest and the same vLLM run on your cluster or on Scaleway without rewriting. Operational sovereignty, being able to move the workload without rebuilding it, is a value that does not appear in the tariff but weighs on a three-year decision.

The performance axis: the provider does not decide, goodput does

One point that simplifies the synthesis: performance does not depend on the provider, it depends on the hardware and the configuration. An H100 gives the same goodput on your cluster, on Scaleway or on AWS, served with the same vLLM and the same config. What decides performance is the goodput under your SLO (track B), not who hosts the GPU. So, in a comparison at equal hardware, the performance axis neutralises itself: what changes between options is cost, energy and sovereignty. The exception: if a provider gives you access to newer hardware (B200, GB200) ahead of your on-prem buying cycle, then the cloud can win on performance per GPU, a real argument in favour of the cloud for staying at the hardware frontier without capex.

The energy axis: the European and Spanish advantage

Here on-prem (or cloud) in Spain or France has a structural advantage over a hyperscaler in a dirty region. Recalling the data from the energy track:

Location	Grid carbon (gCO₂/kWh)	Price (indicative)
France (nuclear)	~20–60	low and stable
Spain (renewables + gas)	~150–170	low, volatile; solar PPA ~32.5 €/MWh
Germany	~363	high
Hyperscaler (region depends on provider)	depends; often not selectable	provider tariff

The same workload in France emits ~9× less carbon per token than in Germany, and in Spain, with a solar PPA at 32.5 €/MWh (an all-time low), the electricity cost, 30–50 % of the TCO, is low and, under contract, predictable. A sovereign cluster in Spain or France controls where the energy is consumed and with what carbon; a hyperscaler gives you the region it gives you, often with no choice of grid intensity. For CSRD reporting, that selectability is a quantifiable advantage of on-prem and the European cloud.

In concrete numbers: the example 8×H100 node (~10.4 kW × PUE 1.3 ≈ 118,000 kWh/year) emits, depending on the grid, ~2.4 t CO₂/year in France (~20 gCO₂/kWh) against ~43 t CO₂/year in Germany (~363 gCO₂/kWh), the same machine, the same work, ~18× the difference in reportable footprint just by choosing the location. That decision, which a hyperscaler in an imposed region does not let you take, is exactly what on-prem and the sovereign European cloud put in your hands. The energy axis is not an environmental detail: it is cost (the price of the kWh), compliance (CSRD) and sovereignty (control of location) all at once.

The sovereignty axis: the one that does not depend on utilisation

And here is the axis that invalidates the cheapest option if the data is sensitive. US hyperscalers are subject to the US CLOUD Act: US authorities can demand data held by a US company even if it sits in a European datacenter. For data subject to the GDPR, that is a compliance risk. Sovereign European clouds operate under EU/EFTA jurisdiction, providing data residency and GDPR compliance, and are exempt from the US CLOUD Act (Lyceum · sovereign providers). On-prem of your own is the maximum degree of sovereignty: the data does not leave your cluster.

The key difference from the other axes: sovereignty does not depend on utilisation or on volume. However much a hyperscaler cheapens the GPU-hour, for GDPR data it is not an option, the jurisdiction risk is not offset by price. It links with the ENS × ISO 42001 × EU AI Act controls and the EU AI Act mapping: compliance is a hard constraint, not an axis to optimise.

The four instruments that turn sovereignty into a concrete constraint rather than a slogan:

Instrument	What it requires	Implication for the architecture
US CLOUD Act	gives the US access to data held by US companies, wherever it is	a US hyperscaler does not guarantee jurisdictional residency even if the datacenter is in the EU
GDPR	residency and processing of personal data under EU law	requires an EU/EFTA provider or your own iron for personal data
EU AI Act	traceability, risk management and records for AI systems	favours the full control of the stack (logs, datasets, models) that on-prem gives
CSRD	verifiable reporting of environmental footprint	the selectable energy (clean grid, PPA) of on-prem and the European cloud is auditable

The operational conclusion: for a European entity processing personal data or deploying high-risk AI, three of the four technical axes can favour the hyperscaler and it can still lose, because the fourth axis, sovereignty, acts as a prior filter. That is why S2 orders the decision like this: first the sovereignty filter (which rules out the hyperscaler for GDPR data), then the optimisation of cost, performance and energy among the options that pass the filter (sovereign on-prem and European cloud).

The scorecard: the three options scored

Crossing the four axes for the three realistic options of a European platform (order-of-magnitude, illustrative figures):

Option	Cost/1M tok	Break-even	Energy/carbon	Sovereignty
Sovereign on-prem (ES/FR)	~1.1 € (high util.) / ~3 € (low)	>65–70 % util.	controllable (clean grid, PPA)	total (EU)
Sovereign European cloud	~1.5–2.2 €	no capex, pay per use	EU, zero-egress	high (EU)
Hyperscaler (US)	~2–3.5 € + egress	no capex	imposed region	not EU (CLOUD Act)

The reading of the scorecard: for data subject to the GDPR, the US hyperscaler is ruled out by the sovereignty axis, however competitive its tariff. The real decision comes down to sovereign on-prem vs sovereign European cloud, and there it is decided by utilisation and volume.

When each option wins

The honest recommendation, by scenario:

Scenario	Winning option	Why
High, sustained volume (util. >65–70 %), GDPR data	Sovereign on-prem	lowest cost/token + total sovereignty
Variable or growing volume, GDPR data	Sovereign European cloud	sovereignty without capex/idle risk
Low volume or sporadic peak	European cloud (per use)	you do not amortise the capex
No sovereignty requirement, hardware frontier	Hyperscaler	access to new hardware without capex
Hybrid (base + peak)	On-prem + European cloud (burst)	cheap base of your own, sovereign elastic peak

On-prem makes sense when there is very high, predictable utilisation (80 %+), strict sovereignty requirements, or a hyperscaler contract that works out expensive (Spheron). For sovereign platforms with a sustained base load, the winning pattern is usually the hybrid: sovereign on-prem for the high-utilisation base (where the cost per token is unbeatable) and sovereign European cloud for the peak and for growth (elastic, no capex, keeping EU jurisdiction). The best of both without giving up sovereignty.

Sizing the hybrid: how much on iron, how much on cloud

The hybrid is not “a bit of each”; it is sized with one figure: the base load percentile. The rule is to put on on-prem the load that is almost always present (the load that keeps the GPU at 75–85 %) and to send to the European cloud only the peaks that, if covered with iron, would leave GPUs idle most of the time. An example with a realistic traffic profile:

Load band	% of the time	Where to serve	Why
Base (p0–p70)	always	on-prem (1 node 8×H100 at ~80 %)	minimum cost/token, high util. guaranteed
Middle (p70–p95)	daily peak hours	on-prem if it fits, otherwise European cloud	elasticity without idle capex
Peak (p95–p100)	sporadic	sovereign European cloud (burst)	absurd to buy iron for a rare peak

With this split, the base amortises your own node at high utilisation (~1.1–1.8 €/1M tokens) and the peak is paid per use with no idle penalty (~1.5–2.2 €/1M tokens), all under EU jurisdiction. The expensive mistake is the opposite: sizing the on-prem for the peak, in which case the GPU spends most of its time idle at 30–40 %, the cost per token shoots up above 3 € and the cloud would have been cheaper. You size the iron for the base, not for the peak; the peak is exactly what the cloud does well. This principle connects with the capacity planning and the scheduling (Kueue/Volcano) of the series: the hybrid only works if the scheduler fills your own node before overflowing to the cloud.

Assumptions and sensitivity

The whole comparison hangs on assumptions that have to be declared, because moving them moves the conclusion:

Assumption	If it rises	Effect
Utilisation	50 % → 80 %	on-prem goes from losing to clearly winning
Energy price	expensive region → France/PPA	lowers on-prem TCO and carbon
Amortisation period	24 → 36 months	lowers the on-prem cost per hour
Volume	< 2M tok/day → much more	crosses the break-even towards on-prem
Egress (hyperscaler)	low → high	makes the hyperscaler dearer than the European cloud

The rule: no on-prem vs cloud comparison is valid without fixing these assumptions. One that says “on-prem is 3× cheaper” without declaring the assumed utilisation is propaganda; one that fixes utilisation, energy price, period and volume is a data point. The dossier must present the case with explicit assumptions and a sensitivity analysis, which is what makes it defensible before a committee that questions them.

Decision checklist

To take S2 from theory to decision, the questions that order the choice, in order:

Is the data subject to the GDPR, or is the system high-risk under the EU AI Act? If so, the US hyperscaler is ruled out on sovereignty; you choose between on-prem and European cloud. If not, the hyperscaler enters the cost comparison.
Can I sustain utilisation above 65–70 % on the base load? If so, on-prem wins on cost for that base. If not, the European cloud avoids paying capex for idle GPUs.
Does the traffic profile have marked peaks? If so, hybrid: base on iron, peak on European cloud. Size the iron for the base, never for the peak.
How much data do I take out of the provider each month? Model the egress; with a lot of movement, the zero-egress of the European cloud, or on-prem, win clearly.
Which electricity grid and at what price? France or Spain with a PPA lower TCO and carbon; include it in the model and in the CSRD report.
Have I fixed utilisation, energy, period and volume in writing? Without those four declared assumptions, the number is not defensible.

Whoever answers these six questions with data, not with intuition, has the case built. The series' recommendation for a sovereign European platform with a sustained base load is stable: sovereign on-prem for the high-utilisation base plus sovereign European cloud for the peak, with the hyperscaler reserved only for workloads with no sovereignty requirement where frontier hardware is needed without capex.

Limits and traps (data-driven)

Unrealistic assumed utilisation. The 80–90 % that makes on-prem win rarely happens in production (40–65 % typical). Model your real utilisation, not the ideal one.
Comparing only the GPU-hour. TCO includes energy, operation, cooling, egress (cloud) and capex (on-prem). Compare totals with the same assumptions.
Ignoring sovereignty. For GDPR data, the sovereignty axis rules out the hyperscaler before cost does; it is not negotiable with price.
Forgetting the hybrid. It is not “all on-prem or all cloud”; the base-plus-peak pattern usually dominates.
Data in USD. US comparisons are in dollars and with dirty regions; convert them to euros and to your region’s grid (Spain/France) for your case.

The synthesis of S2, in one sentence: for sovereign European data, the decision is not on-prem vs cloud in the abstract, but sovereign on-prem (high utilisation) plus sovereign European cloud (peak) against a hyperscaler that the sovereignty axis rules out, and utilisation is the variable that splits the base between the first two. The rest of the series gives the numbers for each axis; this one crosses them into the recommendation. The next synthesis article (S3) sizes the investment; this one decides the architecture.

Sources

Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
Lenovo Press · On-Premise vs Cloud: Generative AI TCO (2026) — https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
Lyceum · EU Sovereign Inference Platform Comparison (2026) — https://lyceum.technology/magazine/eu-sovereign-inference-platform-comparison/
Lyceum · Sovereign Cloud Providers 2026 — https://lyceum.technology/magazine/sovereign-cloud-providers-2026/
Scaleway · H100 GPU instance (precio €, soberanía UE) — https://www.scaleway.com/en/h100/
Nerd Level Tech · GPU Cloud TCO 2026: hidden fees, egress costs — https://nerdleveltech.com/gpu-cloud-comparison-2026-the-real-cost-of-ai-compute

FinOps and multi-tenancy on the GPU cluster: who pays for what

Thu, 11 Jun 2026 12:20:00 +0000

Third instalment in an operational series on how to squeeze a generic on-premise LLM inference cluster of 4×H100 SXM 80 GB. The sibling pieces are Multimodal VLM on-premise with vLLM, serving vision-language models on the same cluster, and Speeding up cold start with Tensorizer, cutting the model load time from disk to HBM. This post closes the economic question the other two leave open: once the cluster serves several models for several teams, who pays for what?

TL;DR

An on-premise GPU cluster does not have the property that makes FinOps easy in the cloud: no monthly invoice arrives telling you what each compute hour cost. The cost has to be manufactured from capex and energy. The right unit is the token, and it is derived in two steps. First the €/GPU-hour: capex depreciation spread over the hours of useful life, plus energy (power × PUE × price_kWh). For an H100 SXM with an assumed capex of 30,000 € and 4 years of life at 90 % availability, that gives ≈ 1.07 €/GPU-hour (0.95 of depreciation + 0.12 of energy at 0.12 €/kWh and PUE 1.4). Second, the €/1M tokens: divide that €/GPU-hour by the sustained throughput. At 2,000 useful tok/s, ≈ 0.148 €/1M tokens; but that number only holds if the GPU is at 100 % useful utilisation. At 20 % utilisation the same token costs 4× more (0.74 €/1M), because capex and energy keep being paid even if the GPU is idle. That is the central thesis: the FinOps goal is not to negotiate the price of the token down, there is no provider to negotiate with, but to raise useful utilisation, which is the only term that moves cost by an order of magnitude. Attribution (chargeback / showback) is made real with LiteLLM virtual keys: per-team and per-user budget, RPM/TPM rate limit, automatic spend tracking and tags to assign spend to cost centres. Isolation between tenants is achieved with MIG (hard GPU partitioning), namespaces, ResourceQuota and PriorityClass. The FinOps Foundation rule worth carving in stone: showback always, chargeback only if the organisation’s accounting policy supports it; neither of the two is “more mature”. And the governance metric of the shared cluster is not the €/token but DCGM_FI_DEV_GPU_UTIL.

The problem the cloud hides from you and on-premise forces you to solve

In the cloud, the cost of a GPU is an observable fact: AWS, GCP or Azure charge you per GPU-hour and the invoice arrives itemised. FinOps in that world consists of reading an invoice that already exists and attributing it. On-premise there is no such invoice. The GPU was bought once, eighteen months ago, on a purchase order nobody remembers; the electricity bill arrives for the whole building with no breakdown of what fraction was the cluster’s; and the datacenter PUE is a number the facilities team knows but the AI team never asked about.

The first job of on-premise FinOps, then, is not to attribute a cost but to manufacture one. You have to build, from explicit assumptions, the equivalent of the “price per GPU-hour” that in the cloud is given to you. And once manufactured, you have to understand that this number is not a property of the hardware, like VRAM or TDP, but a function of how the hardware is used. An idle H100 costs exactly the same as an H100 at 100 %: the capex is already paid and idle energy is not zero. The only thing that changes is how many useful tokens that fixed cost produced. That is where the whole thesis of this post comes from.

The analogy: the coworking space with metered services

A coworking space manages a finite physical space and rents it to teams. The cost model has exactly the structure of the GPU cluster, and the analogy holds down to the last detail.

The desk-hour is the spread capex. The manager paid the rent on the premises, the furniture and the refurbishment, a one-off outlay or a fixed monthly cost, and spreads it over the available desk-hours in the month. Each desk has a base cost that does not depend on whether anybody uses it: the walls, the table and the chair cost the same empty as occupied. This is the depreciation of the GPU: capex divided by the hours of useful life.

The metered kWh is the energy. On top of the desk-hour, the coworking space measures the actual electricity consumption, the plugged-in laptop, the screen, the air conditioning in that room, and bills it separately. This is the energy of the GPU: power × hours × price_kWh, multiplied by the datacenter’s PUE, which is the coworking space also charging you for the fraction of the building’s air conditioning that cools your room.

The empty desk is paid for by whoever booked it. Here is the heart of the matter. If a team books ten desks for the whole month and only uses two, it pays for all ten anyway. The empty desk still takes up space the manager cannot resell, still depreciates furniture, still counts as committed capacity. In the cluster: a GPU assigned to a tenant that uses it at 20 % costs the same as if it used it at 100 %, and the idle 80 % is wasted capex that somebody is paying for.

The manager optimises occupancy, not tariff. A competent coworking manager does not obsess over lowering the price of the desk-hour, it is a sunk cost, the premises are already paid for. They obsess over the occupancy rate: every empty desk is margin lost forever, because an unsold desk-hour cannot be stored. The FinOps equivalent: the goal is not to lower the €/token, it is to raise utilisation, because the €/token falls only as a consequence of utilisation rising. There is no provider to squeeze; the only margin lies in not leaving GPUs idle.

The analogy has an honest limit worth naming: in a coworking space, two people cannot physically share the same chair. On a GPU you can, with time-slicing, MPS or MIG (see Sharing a GPU), and that is precisely the technical lever that lets you raise occupancy beyond what the physical coworking space would allow. But the economics of fixed cost + metered cost is identical.

The token as the unit of cost

Why the token and not the GPU-hour, or the request, or the model? Because the token is the only unit comparable across heterogeneous workloads. A GPU-hour of Llama 8B and one of Llama 70B produce radically different amounts of “useful work”; a RAG request with a 50-token answer and one generating a 3,000-token report are not comparable. The token, specifically the pair (input_tokens, output_tokens), normalises all of that. It is the unit LiteLLM accounts for natively, the one that appears in the OTel attributes gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (see LLM tracing with OpenTelemetry GenAI), and the one commercial providers already use to charge. Adopting the token as the internal currency makes on-premise chargeback directly comparable with the cloud alternative, which is exactly the comparison management wants to see.

The two derived metrics that matter:

Tokens/€ — how many useful tokens each euro of total cluster cost buys. It rises when utilisation rises. It is the metric of economic efficiency.
Tokens/W — how many tokens each watt consumed produces. It is the metric of energy efficiency and, in a datacenter with limited power (the usual case on-premise), it is often the real hard constraint: you cannot add more GPUs because there are no more kW in the rack, so every watt has to perform.

Cost model of a GPU: from capex to €/token

The hourly cost of a GPU has two addends. Let us put all the assumptions in writing, because, as in capacity planning, a cost calculation without written assumptions is a disposable calculation.

Depreciation (spread capex)

$$\text{depreciation cost/h} = \frac{\text{capex}_{\text{GPU}}}{\text{hours of useful life}}$$

where the hours of useful life are the depreciation years × 8,760 h/year × the effective availability (the GPU is not available 100 % of the time: there is maintenance, restarts, update windows). Explicit, generic assumptions (they are not figures from any provider or any real purchase):

Capex per GPU: 30,000 € — a reasonable assumption that includes not just the GPU but its pro-rata share of the server, power supply, cooling, NVLink/InfiniBand network and rack. A “bare” H100 SXM costs less, but honest FinOps spreads the cost of the whole node across its GPUs.
Accounting useful life: 4 years. That is aggressive-realistic for a datacenter GPU; some depreciate over 3, others over 5.
Effective availability: 90 %.

$$\text{hours of life} = 4 \times 8{,}760 \times 0.90 = 31{,}536 \text{ h}$$ $$\text{depreciation cost/h} = \frac{30{,}000}{31{,}536} \approx 0.95 \text{ €/GPU-hour}$$

Energy (power × PUE × price_kWh)

$$\text{energy cost/h} = P_{\text{GPU}} \cdot \text{PUE} \cdot \text{price kWh}$$

Assumptions:

GPU power: the TDP of the H100 SXM is 700 W according to the NVIDIA datasheet (H100 datasheet). Under sustained inference load it hovers around that value; we use 0.7 kW as the average power under load. (At idle it drops a lot, 60–100 W, but the FinOps of reserved capacity reasons about power under load, which is what limits the rack.)
PUE (Power Usage Effectiveness): how many watts enter the datacenter for every watt that reaches the chip. A modest, well-managed on-premise datacenter sits at 1.4; a bad one at 2.0; the hyperscalers boast of 1.1. We use 1.4.
Price per kWh: 0.12 €/kWh — a generic assumption for an industrial tariff; it varies enormously by country and contract.

$$\text{energy cost/h} = 0.7 \cdot 1.4 \cdot 0.12 \approx 0.1176 \approx 0.12 \text{ €/GPU-hour}$$

The total €/GPU-hour

$$\boxed{\text{€/GPU-hour} = 0.95 + 0.12 \approx 1.07 \text{ €/GPU-hour}}$$

Two important readings of this number. First: depreciation dominates (89 % of the cost); energy is 11 %. This inverts many people’s intuition, who believe “the expensive part is the electricity”. With these assumptions, the expensive part is having bought the GPU, and that is why leaving it idle hurts so much. Second: the number is per-GPU; for the 4×H100 node it is ≈ 4.28 €/hour, and for the complete generic cluster of 4 nodes (16 GPUs), ≈ 17.1 €/hour ≈ 150,000 €/year of fixed cost that is paid whether it is used or not.

From €/GPU-hour to €/token

Now we divide the hourly cost by the tokens the GPU produces in an hour. Here sustained throughput comes in. Let us suppose, an order-of-magnitude figure, checkable with vllm bench serve, that a Llama 70B replica on 4×H100 (TP=4) sustains 2,000 useful tokens/s aggregated under good concurrency.

$$\text{tokens/hour} = 2{,}000 \times 3{,}600 = 7.2 \times 10^6 \text{ tok/h}$$

The cost of that replica (4 GPUs) is $4 \times 1.07 = 4.28$ €/hour. Therefore:

$$\text{€/1M tokens} = \frac{4.28}{7.2} \approx 0.59 \text{ €/1M tokens}$$

Per individual GPU, if we take a smaller replica (e.g. Llama 8B FP8 on 1 GPU at ~2,000 tok/s):

$$\text{€/1M tokens} = \frac{1.07 \text{ €/h}}{7.2 \times 10^6 \text{ tok/h}} \times 10^6 \approx 0.149 \text{ €/1M tokens}$$

These numbers are the theoretical floor at 100 % utilisation. Nobody operates at 100 %. And that is the whole story.

Utilisation is the lever (with numbers)

The €/token in the previous section assumes the GPU produces 7.2 M tokens every hour, hour after hour. In practice it produces that only during peak hours. The rest of the time it is partly idle: at night, at weekends, between traffic peaks. The useful utilisation $u$ is the fraction of the theoretical capacity that really turns into billable tokens.

The hourly cost is fixed (1.07 €/GPU-hour is paid come rain or shine), but the tokens produced scale with $u$:

$$\text{€/1M tokens}(u) = \frac{\text{€/GPU-hour}}{\text{max tok/hour} \cdot u} = \frac{1.07}{7.2 \times 10^6 \cdot u} \times 10^6 = \frac{0.149}{u}$$

The requested contrast, worked out:

Useful utilisation $u$	Real tokens/h	€/1M tokens	Multiplier vs 80 %
100 %	7.2 M	0.149 €	0.80×
80 %	5.76 M	0.186 €	1.00× (baseline)
50 %	3.6 M	0.298 €	1.60×
20 %	1.44 M	0.744 €	4.00×
10 %	0.72 M	1.488 €	8.00×

Direct reading: going from 20 % to 80 % utilisation divides the cost per token by four ($0.744 \to 0.186$). No tariff negotiation, no quantization, no hardware change gives you that factor of 4 so cheaply. Quantizing from BF16 to FP8 can double throughput, and therefore halves the €/token, but it degrades quality and demands evals (see Quantization for inference); raising utilisation from 20 to 80 % does not touch the quality of a single token.

This reformulates GPU cluster FinOps in a single sentence: the €/token is not something you negotiate, it is something you earn by filling the GPUs. And raising utilisation is an engineering problem that already has its pieces:

Autoscaling that switches replicas off when traffic drops, so as not to pay for idle GPU (see LLM autoscaling on Kubernetes with KEDA). Switching off a 4-GPU replica for eight hours overnight saves ≈ 34 € a day of real cost; multiplied by replicas and by days, that is the difference between a profitable cluster and one that bleeds.
Sharing a GPU between small workloads with MIG or time-slicing, so that two tenants that individually would use 30 % together fill a GPU to 60 % (see Sharing a GPU).
Overnight batch that fills the off-peak hours with non-urgent work (re-embeddings, evals, light fine-tuning) instead of leaving the GPUs switched off or ticking over.
Reducing cold start so that scaling to zero and starting again is cheap and therefore viable as a utilisation strategy (see the sibling piece Speeding up cold start with Tensorizer).

The diagram: the €/token cascade by utilisation

The cost per token is not a number, it is a cascade that multiplies as utilisation falls:

Attributing the cost: LiteLLM virtual keys

With the €/token manufactured, it remains to attribute it to whoever consumed it. The piece that already sits in front of the engines, the L7 inference router, is also the natural place to count. LiteLLM Proxy makes chargeback real with four mechanisms, all documented:

Virtual keys with a budget. Each team or user receives a virtual key with a max_budget and a budget_duration. You can define several independent budget windows (for example one of 24 h and another of 30 d) that reset on their own cycle (Virtual Keys). When a team exhausts its monthly token budget, the proxy rejects with a budget exceeded error instead of carrying on spending GPU-hours nobody will be able to charge anywhere.

RPM/TPM rate limit. Each key carries requests per minute (RPM) and tokens per minute (TPM) limits (Budgets, Rate Limits). This is not just anti-abuse protection: it is the tool that stops a noisy tenant monopolising the cluster’s utilisation at everybody else’s expense. The rate limit is the capacity quota; the budget is the spend quota. They are orthogonal and you use both.

Automatic spend tracking. LiteLLM accounts for the spend of all known models by key, user and team, recording the API key, the user, the team_id, the request’s tags, the end user, the model group and the token counts (Spend Tracking). For the cluster’s self-hosted models, the cost per token is configured with the €/token we manufactured above, and that is the hook between the cost model and the real accounting.

Tags for cost centres. The tags let you track spend and set budgets per label, categorising costs by project, department or cost centre (Setting Tag Budgets). A tag is attached when the key is created and every request made with that key inherits the tag automatically; the proxy applies the label’s budget. That way chargeback maps cleanly onto the organisation’s accounting hierarchy without every team having to remember to tag by hand.

A declarative catalogue fragment, which fits into the litellm-config of the router post, materialising the manufactured cost and two tenants:

litellm_settings:
 # €/token manufactured in this post for the self-hosted model.
 # input/output in €/token (not per 1M); 0.186 €/1M = 1.86e-7 €/token at u=80%.
 model_cost_map:
 llama-70b-onprem:
 input_cost_per_token: 0.000000186
 output_cost_per_token: 0.000000186

# Virtual keys per team (via the /key/generate API or config):
# equipo-datos: max_budget=500 €/month, tpm_limit=200000, tags=["cc-datos"]
# equipo-soporte: max_budget=150 €/month, tpm_limit=60000, tags=["cc-soporte"]
# The proxy counts tokens, multiplies by model_cost_map, deducts from the budget,
# and attributes the spend to the tag → showback per cost centre, without touching the engine.

The complete attribution flow, from the request to the per-team report:

Showback vs chargeback: the distinction the FinOps Foundation insists on

The two words are constantly confused and the difference is one of accounting formality, not technology:

Showback — giving each team visibility of what it consumed and its cost, without really billing it to their budget. The report arrives, the team sees it, but the money does not move between cost centres.
Chargeback — really transferring the cost to the team’s or product’s budget, putting it in their P&L. The spend stops being a central IT cost and becomes an allocated cost.

The FinOps Foundation is explicit on two points worth carving in stone (Invoicing & Chargeback, Chargeback vs Showback). First: showback is a requirement of any FinOps practice; chargeback depends on the organisation’s accounting policy. Not all organisations can or want to move money between departments for GPU consumption; showback you can always do. Second, and counterintuitive: neither of the two is “more mature” than the other. The narrative that chargeback is the “grown-up” version of showback is false according to the framework itself. The practical recommendation: start with showback, giving visibility, then build cost allocation aligned to the organisational hierarchy, and only then, if the accounting policy supports it, switch chargeback on.

For the on-premise GPU cluster this means: the LiteLLM machinery (keys, tags, spend tracking) always produces the showback. Turning it into chargeback is a decision for finance, not for the platform team. The platform team guarantees that the numbers are correct and reproducible; who really pays is a governance decision.

Isolation: making the cost genuinely attributable

Chargeback is only honest if consumption is isolable. If two tenants share a GPU without partitioning and one saturates the HBM, the other suffers degradation it did not cause but which contaminates its cost attribution. Isolation has two planes.

Hard GPU isolation: MIG. Multi-Instance GPU physically partitions an H100 into up to 7 instances with dedicated memory and SMs; in a 7-way partition each instance has ~10 GB of HBM3 and its own SMs. MIG gives the strongest isolation: tenant A in its MIG instance cannot touch tenant B’s performance, and attribution is trivial because each instance is accountable separately (see Sharing a GPU for the detail of MIG vs time-slicing vs MPS). The cost: MIG instances are fixed, they are not resized hot, and if they sit empty they are fragmented and wasted capex, the coworking desk subdivided into booths nobody rents.

Logical Kubernetes isolation. On top of the cluster:

Namespaces per tenant — the boundary for RBAC, NetworkPolicy and quotas.
ResourceQuota — limits how many nvidia.com/gpu (or nvidia.com/mig-1g.10gb) a namespace can request. It is the GPU capacity quota at the scheduler level: the tenant cannot claim more GPUs than its quota grants, which bounds its maximum cost by construction.
PriorityClass — defines which workloads can evict others under pressure. The overnight re-embeddings batch runs with low priority and gives way to interactive inference; that way it fills the utilisation valleys without risking the paying tenant’s SLO. It is the piece that stops “raising utilisation with batch” from cannibalising quality of service.

The combination that works in the generic cluster: MIG to partition the GPUs between tenants that need hard isolation and clean attribution, ResourceQuota per namespace to bound each tenant’s cost, and PriorityClass so that filler work raises utilisation without touching the priority tenants.

Applied to the generic 4×H100 cluster

Let us bring it all down to the series’ cluster: 4 nodes × 4×H100 SXM 80 GB, 16 GPUs, ≈ 17.1 €/hour of fixed cost (≈ 150,000 €/year). Three teams share it: Data (production RAG, office-hours traffic), Support (ticket assistant, daytime peaks) and Platform (evals and re-embeddings batch, no urgency).

Splitting the cost. The cluster’s fixed cost (150,000 €/year) is split through LiteLLM’s spend tracking in proportion to the tokens consumed by each team, valued at the manufactured €/token. If in one month Data consumed 1,700 M tokens, Support 480 M and Platform 320 M, at 0.186 €/1M the showback is ≈ 316 € / 89 € / 60 €. The residue, the fixed cost of the GPUs nobody used because average utilisation was, say, 45 %, is the cost of idleness, and the governance decision is whether it is split between the tenants (penalises everybody equally) or charged to Platform as “cost of unsold capacity” (gives Platform an incentive to raise utilisation). The FinOps Foundation would say: make it visible first (showback of the cost of idleness), decide the split afterwards.

Quotas per tenant. Each team has its namespace with ResourceQuota: Data can claim up to 8 GPUs (2 replicas at TP=4), Support up to 4, Platform up to 4 but with a low PriorityClass, giving up its GPUs when Data or Support need them at peak. In LiteLLM, each team has its virtual key with a monthly max_budget and a tpm_limit that reflects its capacity quota.

MIG to isolate and attribute. For the small models (embeddings, reranker, a utility Llama 8B), an H100 split 7-way with MIG gives seven separately attributable instances: three for Data’s embeddings, two for Support’s reranker, two free for filler. Each instance is an independent accounting “desk”; the per-tenant showback comes straight from the scheduler.

The governance metric is utilisation, not the €/token. Here we close the circle. The KPI the cluster owner should look at daily is not “what the token costs”, that number only falls as a consequence, but DCGM_FI_DEV_GPU_UTIL, the DCGM exporter metric that indicates what fraction of the time the GPU is not idle (DCGM exporter). And with a critical nuance that already appeared in capacity planning: DCGM_FI_DEV_GPU_UTIL measures “not idle”, it does not measure useful work. A GPU can show 100 % GPU-util with the HBM saturated and low SM occupancy, producing few tokens. That is why serious FinOps crosses three signals: DCGM_FI_DEV_GPU_UTIL (is the GPU busy?), DCGM_FI_PROF_SM_OCCUPANCY and DCGM_FI_PROF_GR_ENGINE_ACTIVE (is it doing real work?), and vllm:gpu_cache_usage_perc (is the HBM well used?). The economic governance metric is useful tokens per GPU-hour, and it is watched from the GPU observability with DCGM dashboard.

An end-to-end assembly detail, LibreChat on top of LiteLLM with RAG, which closes the flow from tenant to cost, is covered in the (forthcoming) post on the end-to-end sovereign assistant; here it is enough to know that the metering point is always the proxy, never the engine.

Four traps of on-premise GPU FinOps

Trap 1 — comparing the on-premise €/token with the commercial API’s list price. The commercial API has margins, SLA and scale the on-premise cluster does not, but on-premise has the cost of idleness the API hides from you (they fill their GPUs with thousands of customers). The honest comparison is on-premise at its real utilisation vs the API at its real negotiated price, not the fantasy of on-premise at 100 %.

Trap 2 — forgetting the cost of idleness in the showback. If you only charge teams for the tokens they consumed, the cost of the idle GPUs disappears from the report and nobody sees it. That cost exists and somebody is paying it. Making it visible is the first step to reducing it.

Trap 3 — confusing high GPU-util with efficiency. A GPU at 100 % DCGM_FI_DEV_GPU_UTIL with the HBM saturated and low SM occupancy spends capex without producing proportional tokens. The goal is not “GPU at 100 %”, it is “useful tokens per euro”. Always cross util with SM occupancy and real throughput.

Trap 4 — chargeback before showback. Switching chargeback on, moving money, before the teams trust that the numbers are correct generates disputes that burn the whole FinOps programme. First visibility, then trust in the data, then, if the accounting policy allows it, the money.

References

NVIDIA — H100 Tensor Core GPU Datasheet (TDP 700 W, HBM3 3,35 TB/s): resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306.
LiteLLM — Virtual Keys (docs.litellm.ai/docs/proxy/virtual_keys), Budgets & Rate Limits (/docs/proxy/users), Spend Tracking (/docs/proxy/cost_tracking), Setting Tag Budgets (/docs/proxy/tag_budgets), Team Budgets (/docs/proxy/team_budgets).
FinOps Foundation — Invoicing & Chargeback Capability (finops.org/framework/capabilities/invoicing-chargeback/) y Data Analysis and Showback (finops.org/framework/previous-capabilities/analysis-showback/).
CloudZero — Chargeback vs. Showback: Cloud Cost Allocation Models Explained (cloudzero.com/blog/chargeback-vs-showback/).
NVIDIA — DCGM Exporter (docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html): DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_GR_ENGINE_ACTIVE.

GitOps for the inference stack with Flux: operating the assistant as code

Thu, 11 Jun 2026 10:20:00 +0000

This post is part of the operational series on how to squeeze a generic on-premise 4×H100 SXM 80 GB LLM cluster. The sibling pieces: the document ingestion pipeline for RAG that fills the vector store we deploy here, the embeddings and reranker service with TEI that is one of the services GitOps manages, and the secrets hardening of the sovereign stack, which goes deep into the secrets problem we only state here. The complete end-to-end assistant (LibreChat + LiteLLM + RAG) that orchestrates all of this will get its own post.

TL;DR

An LLM assistant in production is a distributed system, not a binary. At a minimum it has an inference engine (vLLM or similar), an L7 gateway that routes per model and applies rate limiting, a chat front end, a vector store for RAG, an embeddings/reranker service, and an observability stack. Six substantial services, spread across three or four namespaces, with start-up dependencies (the front end is useless if the gateway is not there, the gateway is useless if the engine has not loaded the model, RAG does not answer if the vector store is empty). Operating it by hand, a kubectl apply here and a helm upgrade there, produces a cluster whose real state nobody can reconstruct: there is no record of what was applied, or when, or why. GitOps solves this with four principles (OpenGitOps): the desired state is declarative, it is versioned and immutable in git, it is applied automatically by an agent (nobody SSHes in to deploy) and it is continuously reconciled by comparing the actual state against the declared one. Flux is that agent: six controllers (source, kustomize, helm, notification, image-reflector, image-automation) that clone the repo, render Kustomize/Helm, apply to the cluster, correct drift and, optionally, write back to git to bump the tag of a new image. The unit of work is the Kustomization or the HelmRelease, with interval (how often it reconciles), prune (deletes what is no longer in git), dependsOn (orders the start-up) and health checks. The interval sets the MTTR of drift: with interval=1m, a manual edit to the cluster is reverted in ≤ 1 min on average. Secrets cannot sit in the clear in git, a chicken-and-egg problem, and it is solved with SOPS/age, sealed-secrets or External Secrets + Vault. Rollback is a git revert. GitOps is not free: the learning curve is real, and debugging why a Kustomization is not reconciling is a new skill.

The analogy: the master plan and the foreman who does not negotiate

Picture a large building site with a signed and filed master plan, and a foreman who has a single order: the site must be identical to the plan, at all times. The foreman does not improvise. Every so often he walks the site with the plan in hand and compares: if a wall is where the plan says, he leaves it; if someone moved a partition overnight without updating the plan, he puts it back; if the plan says there is a column and it does not exist, he builds it; if a column exists but no longer appears on the plan, he knocks it down. The plan is the single source of truth: to change the site you do not touch the site, you change the plan, and the foreman takes care of the rest on his next round.

That is exactly the mechanics of GitOps. Git is the master plan: declarative (it describes the end state, not the steps), versioned (every change is signed into the history, with author and reason), immutable (a commit is not rewritten). Flux is the foreman: every interval it walks the cluster, compares against git and converges. And here is the lesson that separates GitOps from “keeping the YAML in a repo”: the foreman undoes manual changes. If an operator comes in with kubectl edit and raises the inference engine replicas from 2 to 4 at three in the morning to put out a fire, on the next round Flux takes it back to 2, because the plan says 2. This enrages anyone coming from the imperative world, and it is precisely the point: if you want 4 replicas permanently, you update the plan. The cluster stops being a system with its own opaque memory and becomes a reproducible projection of git. Delete the entire cluster, point a fresh Flux at the same repo, and the site is rebuilt identically.

The analogy also marks the boundary: the plan describes the building, not who holds the keys to the materials store. Secrets (passwords, tokens, keys) cannot go into the public plan. That is the chicken-and-egg problem we cover further down and that the hardening piece develops in full.

The problem: an assistant is six services, not one

Before Flux, it is worth fixing the size of the problem. A minimally serious sovereign LLM assistant, deployed on the generic reference cluster, has this service topology:

Service	Function	Typical namespace	State
Inference engine (vLLM)	Serves the tokens of the general LLM and the code one	`llm-serving`	Stateless (model from object store)
Embeddings + reranker (TEI)	Vectorises queries and reorders candidates for RAG	`llm-serving`	Stateless
Vector store (Qdrant)	Stores and searches document embeddings	`data`	Stateful (PVC)
L7 gateway (Envoy AI Gateway / LiteLLM)	Per-model routing, rate limiting, auth	`gateway`	Semi-stateless
Chat front end	Assistant UI	`apps`	Stateless
Observability (Langfuse, OTel Collector, dashboards)	LLM-aware traces, metrics and logs	`observability`	Stateful (Langfuse Postgres)

Six services, four or five namespaces, two components with persistent state, and a real dependency graph: the front end calls the gateway, the gateway calls the engine and the embeddings service, RAG depends on the vector store being full, and everything emits traces to observability. Deploying this by hand means remembering the order, the values of each helm install, the ConfigMaps, the Secrets, the nodeSelectors that pin the GPU pods to the right nodes. Do it twice (a staging environment and a production one) and the two diverge within days. That is exactly what GitOps eliminates.

The six Flux controllers

Flux is not a monolithic binary but a set of cooperating controllers, the GitOps Toolkit. A default installation brings four; the two image automation ones are added with --components-extra (Flux installation):

source-controller: clones and keeps the sources up to date, whether git repos (GitRepository), Helm repos (HelmRepository), buckets or OCI artifacts. It is the one “holding the plan”: it exposes the repo contents as an internal artifact that the others consume.
kustomize-controller: takes a source artifact, renders Kustomize (bases + overlays) and applies the result to the cluster. It is responsible for prune, dependsOn and the health checks of the Kustomizations.
helm-controller: reconciles HelmRelease objects, installing and updating Helm charts declaratively, without anybody running helm from a terminal (helm-controller).
notification-controller: the bridge to the outside world in both directions. It receives webhooks (to reconcile instantly on every push, instead of waiting for the interval) and emits events and alerts to Slack, an internal chat or an incident system.
image-reflector-controller: scans container registries and stores the tags it finds in an internal database. It is the “eyes” of image automation.
image-automation-controller: uses what the eyes see to write back to git, committing the new image tag into the manifests when a version appears that satisfies the policy.

A cluster without image automation does not need the last two. But for an inference stack where the model server is updated fairly often, they are what automates version promotion without touching anything by hand.

The diagram has three paths. The grey one is the main flow: git → source-controller → kustomize/helm-controller → cluster. The dashed red one is the reconciliation loop that runs every interval: it compares actual state against desired, fixes drift and prunes what is left over. The purple one is image automation: the reflector sees a new tag in the registry, the policy decides whether it qualifies, and the automation-controller writes back to git. That last path is the one that closes the circle and turns git into a system that updates itself.

Repo structure: clusters, infrastructure, apps

The most widespread convention separates three levels of responsibility. It is not the only one, but it ages well:

gitops-repo/
├── clusters/
│ └── prod/
│ ├── infrastructure.yaml # Kustomization → ./infrastructure
│ └── apps.yaml # Kustomization → ./apps (dependsOn infrastructure)
├── infrastructure/
│ ├── controllers/ # ingress, cert-manager, GPU operator, KEDA...
│ └── configs/ # ClusterIssuer, RuntimeClass, StorageClass...
└── apps/
 ├── base/ # manifests common to all environments
 │ ├── llm-engine/ # vLLM HelmRelease
 │ ├── gateway/ # L7 gateway HelmRelease
 │ ├── chat-front/ # Deployment + Service + Ingress
 │ ├── vector-store/ # Qdrant HelmRelease (+ PVC)
 │ └── observability/ # Langfuse + OTel HelmRelease
 ├── staging/ # overlay: low replicas, small model
 └── prod/ # overlay: high replicas, large model, MIG

The key piece is Kustomize’s base / overlay separation. The base/ describes the service once; each overlay (staging/, prod/) applies a patch with the environment’s differences: number of replicas, model size, MIG profile, nodeSelector, --gpu-memory-utilization. That avoids duplicating complete manifests per environment. The clusters/ folder is the entry point Flux reconciles first: it holds the root Kustomizations that point to infrastructure/ and apps/, with a dependsOn that guarantees the infrastructure (CRDs, operators, storage classes) is ready before the applications.

How many manifests: the calculation that justifies overlays

This is where the numbers make the argument. Without overlays, each service needs a complete set of manifests per environment. With $N$ services and $M$ environments, the cost in files to maintain is:

$$ \text{files}_{\text{naïve}} = N \times M \times k $$

where $k$ is the average number of manifests per service (Deployment/HelmRelease + Service + ConfigMap + PVC + Ingress ≈ 5). For our assistant, $N=6$ services and $M=3$ environments (dev, staging, prod), with $k=5$:

$$ 6 \times 3 \times 5 = 90 \text{ complete files, each maintained separately.} $$

With the base/overlay structure, the base is written once and each overlay contains only the patch with the differences (typically 1 short file per service per environment):

$$ \text{files}_{\text{overlay}} = \underbrace{N \times k}_{\text{base}} + \underbrace{N \times M}_{\text{patches}} = 6 \times 5 + 6 \times 3 = 30 + 18 = 48 $$

It is not just that there are fewer files (48 against 90): it is that the bulk of a change happens in a single place. Changing the engine’s memory limit for every environment is one edit in base/, not three synchronised edits. The saving factor grows with $M$: for 5 environments, the naïve version is 150 files and the overlay one is 60. Duplication is the enemy of auditability, and overlays attack it at the root.

The reconciliation loop: desired vs actual

The heart of Flux is a control loop that never ends: every interval it reads the desired state (git), reads the actual state (cluster), computes the difference and applies it. It is the same principle as a thermostat: it reads the target temperature (git), reads the actual temperature (cluster) and switches the boiler on or off until they match. It does not “deploy once”; it converges forever.

A Kustomization manifest for the applications, with the pieces that matter (Kustomization reference):

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: apps
 namespace: flux-system
spec:
 interval: 1m # reconciles every minute: sets the drift MTTR
 path: ./apps/prod  # production overlay
 prune: true # deletes from the cluster whatever is removed from git
 sourceRef:
 kind: GitRepository
 name: gitops-repo
 dependsOn:
 - name: infrastructure  # does not apply apps until infra is Ready
 wait: true # waits for the resources to be healthy
 timeout: 5m  # fails the reconciliation if it does not converge in 5 min
 healthChecks:
 - apiVersion: apps/v1
 kind: Deployment
 name: gateway
 namespace: gateway

Four fields do the heavy lifting:

interval defines how often the foreman’s round runs. It is the direct determinant of the drift MTTR (next section).
prune: true switches on garbage collection: objects that were applied before but are no longer in the current git revision are deleted from the cluster automatically. Without prune, a service withdrawn from the repo keeps running, and the plan and the site diverge silently.
dependsOn orders the graph: Flux does not apply the manifests of a Kustomization until every referenced one is in state Ready: True. It is the mechanism that guarantees data → gateway → front end.
wait + healthChecks: with wait: true, Flux monitors every applied resource and waits for them to be ready before marking the reconciliation as successful; healthChecks lets you tune exactly which resources to watch. That is what makes a dependsOn mean something: the dependency is not satisfied until it is healthy, not merely applied.

MTTR and drift detection as a function of the interval

The interval is the only parameter the operator chooses to govern the speed of the loop, and it translates directly into operational metrics. If a deviation (drift) happens at a random instant within the reconciliation period $T$, the waiting time until Flux detects it is uniformly distributed over $[0, T]$, with mean:

$$ \mathbb{E}[t_{\text{detection}}] = \frac{T}{2} $$

To that you add the correction time $t_c$ (render, diff, apply), normally a few seconds. The drift MTTR comes out as:

$$ \text{MTTR}_{\text{drift}} = \frac{T}{2} + t_c $$

With interval=1m and $t_c \approx 10\text{s}$, drift is corrected in $\frac{60}{2} + 10 = 40$ s on average, with a worst case of $60 + 10 = 70$ s. With interval=10m, the mean rises to $5\text{min}\;10\text{s}$ and the worst case to more than 10 min: a window in which a manual change stays active. The temptation is to set interval=10s and forget about it, but there is a cost: every reconciliation consumes CPU and, above all, makes requests to the cluster API and to the registry. With dozens of Kustomizations reconciling every 10 s, the kube-apiserver and the image-reflector start to feel the pressure. The practical rule: a short interval (1m) for critical applications whose drift hurts, a long interval (10–30m) for stable infrastructure that almost never changes, and notification-controller webhooks for instant reconciliation on every push, so that the interval governs only drift and not deployment latency.

Helm as code: the inference engine HelmRelease

For the engine, the gateway and the front end, the usual approach is to package them as Helm charts and declare them with HelmRelease. The helm-controller reconciles them without anyone ever running helm:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: llm-engine
 namespace: llm-serving
spec:
 interval: 10m
 chart:
 spec:
 chart: vllm
 sourceRef:
 kind: HelmRepository
 name: vllm-charts
 values:
 image:
 tag: "0.8.4" # managed by image automation (see below)
 replicaCount: 2
 nodeSelector:
 nvidia.com/gpu.product: H100-SXM
 resources:
 limits:
 nvidia.com/gpu: 1
 extraArgs:
 - "--gpu-memory-utilization=0.90"
 - "--tensor-parallel-size=1"

Everything that in the imperative world would be a vllm serve flag, --gpu-memory-utilization or --tensor-parallel-size, now lives as versioned data. Changing the fraction of VRAM the engine reserves is a commit, reviewable in a PR, with history. The relationship with sharing a GPU is direct: the MIG profile and the --gpu-memory-utilization are deployment parameters, and here they are code.

Image automation: bumping the tag without touching anything by hand

The inference engine is updated fairly often (vLLM patches, security fixes). Without automation, every new version requires somebody to edit the tag in the HelmRelease. Flux image automation does it on its own, with three objects (automate image updates):

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
 name: vllm
 namespace: flux-system
spec:
 image: registry.example.local/inference/vllm
 interval: 5m  # scans the registry tags every 5 min
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
 name: vllm
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: vllm
 policy:
 semver:
 range: ">=0.8.0 <0.9.0" # only patches and minors within 0.8.x–0.8.x
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
 name: vllm
 namespace: flux-system
spec:
 interval: 5m
 sourceRef:
 kind: GitRepository
 name: gitops-repo
 git:
 commit:
 author:
 name: fluxbot
 email: fluxbot@example.local
 messageTemplate: "auto: bump vllm to {{ .NewTag }}"

The division of roles: ImageRepository scans all the tags of the image repository and stores them in the reflector’s internal database. ImagePolicy reads those tags and picks the “latest” according to the policy; the policy field is mandatory and defines how the selection is made (ImagePolicy). ImageUpdateAutomation takes the chosen tag, edits the manifest in the repo (wherever there is a # {"$imagepolicy": "flux-system:vllm"} marker) and commits (ImageUpdateAutomation).

The choice of policy matters:

semver: interprets tags as semantic versions and picks the highest one that satisfies the range (range: ">=0.8.0 <0.9.0"). By default it excludes prereleases (0.8.0-rc.1 does not qualify) unless they are asked for explicitly. It is the right option for production: you stay inside a tested version range and do not jump to a new major by accident.
regex / alphabetical / numerical: for tag schemes that are not pure semver, for example main-<sha>-<timestamp> from an internal pipeline. More flexible, but it forces you to trust that tag order reflects version order, which is fragile.

A safety guideline: instead of having ImageUpdateAutomation push directly to main, configure it to write to a branch and open a PR. That way the version bump goes through human review or a validation pipeline before reaching the cluster. Automation proposes; the human (or an automatic gate) disposes. And for production, the version bump is not the deployment: it is the trigger for the progressive rollout described in canary, blue-green and shadow.

Secrets in GitOps: the chicken-and-egg problem

GitOps demands that all the desired state is in git. But secrets, the Langfuse Postgres password, the registry token, the gateway API keys, cannot go into a repo in the clear, not even a private one: git history is immutable, and a secret committed once stays there forever. This is the fundamental tension: GitOps wants everything in git; security forbids secrets in git. There are three families of solution, all with the same underlying idea, that only the encrypted secret goes into git and the cluster holds the key to decrypt it:

SOPS + age/KMS: you encrypt the sensitive values of a manifest with SOPS (it leaves the keys readable and encrypts only the values). The encrypted YAML goes to git; the kustomize-controller carries an age key or reaches a KMS to decrypt it at apply time. Simple, with no extra components in the cluster.
sealed-secrets: a controller in the cluster holds a private key. You encrypt the secret against its public key (with kubeseal), the encrypted SealedSecret goes to git, and the controller decrypts it into a normal Secret inside the cluster. The private key never leaves the cluster.
External Secrets + Vault: not even the encrypted secret goes into git, only a reference (ExternalSecret) saying “the value of this key is in Vault, at this path”. The External Secrets operator resolves it at apply time. It is the cleanest pattern for many secrets and frequent rotation, at the cost of operating a Vault.

The choice depends on the volume of secrets and on whether you already have a central manager. For a small stack, SOPS+age is enough and has no dependencies. The sibling hardening piece goes into the detail of each option, key rotation and the threat model.

Promotion and rollback: git revert is the panic button

The most elegant consequence of having the cluster as a projection of git is that rollback is a git revert. If a deployment breaks production, the new engine gives worse latency or the gateway starts returning 5xx, there is no need to remember which version was there before or to rebuild the state by hand. You revert the commit that introduced the change, and on the next reconciliation (≤ interval, or instantly with a webhook) Flux returns the cluster to the previous state. The git history is, literally, the deployment history: every commit is a restore point with author, date and reason.

This fits with the promotion strategies of the canary, blue-green and shadow post: Flux manages what is declared, and the progressive rollout tool manages how traffic moves between the old version and the new one. The git revert is the last-resort rollback (it returns everything to a known state); the canary is the mechanism that avoids needing it. For services where the version change is delicate, the right approach is for image automation to do the bump on a branch, for the canary to validate the regression gates, and only then to promote the commit to main.

The seams: GitOps is not free

It would be dishonest to sell GitOps as magic. It has real costs that any team pays:

Learning curve. The declarative model is a change of mindset. The operator who has spent years fixing incidents with kubectl edit has to unlearn the reflex: now the cluster reverts their changes and that, at first, feels like Flux “fighting them”. Understanding that the change is made in git, not in the cluster, takes weeks of discomfort.

Debugging the reconciler. When a Kustomization does not apply, the error is not in the pod; it is in the reconciliation chain. Did the source-controller clone the right revision? Did the kustomize-controller render the overlay properly? Did a dependsOn health check fail? Did the timeout fire before the large model finished loading? Diagnosing this needs flux get, flux logs, reading the status.conditions of the Flux objects and understanding which link got stuck. It is a new skill, different from debugging plain Kubernetes.

Timeouts and slow loads. The timeout of a Kustomization with wait: true has to be longer than the start-up time of the slowest service. An inference engine that takes 4 minutes to load a large model from the object store will fail a Kustomization with timeout: 2m, even though everything is fine. Calibrating timeouts per service is fine-grained work.

The drift you did want. Sometimes the operator needs an urgent temporary change and Flux reverts it. The correct answer, suspending reconciliation of that Kustomization with flux suspend, making the change, then reflecting it in git and resuming, is discipline that has to be built. Without that discipline, people end up fighting the foreman at 3 in the morning.

None of these costs cancels the benefit. But a team adopting GitOps expecting “everything to be easier from day one” gets frustrated. It is easier from month two onwards, once auditability, reproducibility and trivial rollback have paid off the curve.

Applied to the generic 4×H100 cluster

On the reference cluster (4×H100 SXM 80 GB per GPU node, NVLink, plus CPU and control nodes), the assistant stack is managed entirely by GitOps. Hardware decisions become data in the overlays.

nodeSelectors and MIG profiles as code. The prod overlay pins each service to the right node and declares the GPU profile. The general engine uses a whole GPU; embeddings and the small LLM fit in MIG slices, exactly the split from the sharing a GPU post, but now versioned:

# apps/prod/llm-engine-patch.yaml
spec:
 values:
 nodeSelector:
 nvidia.com/gpu.product: H100-SXM
 extraArgs:
 - "--gpu-memory-utilization=0.92"
 - "--tensor-parallel-size=1"
---
# apps/prod/embeddings-patch.yaml — on a MIG slice
spec:
 values:
 nodeSelector:
 nvidia.com/mig.config: "3g.40gb"
 resources:
 limits:
 nvidia.com/mig-3g.40gb: 1

The MIG profile (3g.40gb), the VRAM fraction (0.92), the tensor-parallel setting: all of it is text in a PR. Changing the GPU split between services is a reviewable commit, not a hand-run nvidia-smi mig session that nobody records.

Start-up order with dependsOn. The assistant’s dependency graph is encoded with chained dependsOn. The right order is data → embeddings/engine → gateway → front end:

# apps/prod/gateway.yaml
spec:
 dependsOn:
 - name: llm-engine  # the gateway is useless without the engine
 - name: embeddings
 # ...
---
# apps/prod/chat-front.yaml
spec:
 dependsOn:
 - name: gateway  # the front end is useless without the gateway

With wait: true on each Kustomization, Flux does not mark llm-engine as Ready until the engine pod passes its health check, which in the case of vLLM means model loaded and endpoint responding, not just pod started. Only then does it begin applying the gateway. This avoids the classic cascade of failures of manual deployment: bring up the front end first, see a 502 because the gateway is not there, bring up the gateway, see a 503 because the engine is still loading the model. With dependsOn + wait, the order is guaranteed by the reconciler, not by the operator’s memory.

The result: deleting the assistant’s entire namespace and letting Flux rebuild it from git produces exactly the same stack, in the same order, with the same GPU configuration. The 4×H100 cluster stops having a configuration known only to whoever built it, and becomes a reproducible projection of a repo that anyone can audit.

Conclusion

GitOps is not a tool, it is an inversion of the direction of control: instead of pushing changes to the cluster, you declare the state in git and let an agent pull the cluster towards it. For an LLM assistant (six services, several namespaces, start-up dependencies, delicate GPU configuration) that inversion turns a fragile and opaque system into a reproducible and auditable one. Flux is the foreman who keeps the site identical to the plan and undoes any change that has not gone through the plan first. The price is a real learning curve and a new debugging skill. The prize is that the cluster stops keeping secrets about itself: everything it is, is written down, signed and reproducible in git.

References

OpenGitOps — GitOps principles — opengitops.dev
Flux installation — fluxcd.io/flux/installation
Flux Kustomization — fluxcd.io/flux/components/kustomize/kustomizations
helm-controller — github.com/fluxcd/helm-controller
Flux Image Policies — fluxcd.io/flux/components/image/imagepolicies
Flux Image Update Automations — fluxcd.io/flux/components/image/imageupdateautomations
Automate image updates to Git — fluxcd.io/flux/guides/image-update
SOPS — github.com/getsops/sops

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

Tue, 09 Jun 2026 17:00:00 +0200

Operational companion to The contractor with the master key. That post explains the why and the where: the threat model, the five isolation families, which domain uses each. This one is the how, with commands. If you have not read it, read it first: here I take for granted what the blast radius is, why bwrap runs without root and what Tetragon watches. The procedure comes in two independent tracks, client and cluster, because, as the sibling post argues, the control extrapolates but the primitive gets rewritten.

TL;DR

Two reproducible procedures. Client (workstation): install ai-jail (it wraps bubblewrap), generate the per-project .ai-jail, audit with --dry-run, fix the allowlists with --bootstrap, use --lockdown for anything you do not trust, and leave the agent without git push permission. Cluster (RKE2 with Cilium + Tetragon): put the pod baseline in place (unprivileged securityContext, seccomp: RuntimeDefault, default-deny NetworkPolicy), put the untrusted agent’s pod in a microVM with runtimeClassName: kata, and deploy the Tetragon TracingPolicy files in two phases: observe with action: Post to raise the baseline, then promote to action: Sigkill on tcp_connect (egress) and security_file_open (secret paths). The golden rule of the Tetragon phase: adopt first, block later; never put a Sigkill into production without having seen the events in observation mode first.

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

ai-jail wraps the sandbox; on Linux it needs bubblewrap separately, on macOS it needs no extra dependency.

# ai-jail (macOS and Linux)
brew tap akitaonrails/tap && brew install ai-jail
# or, with cargo:
cargo install ai-jail
# or, with mise:
mise use -g ubi:akitaonrails/ai-jail

# bubblewrap on Linux (pick your distro)
sudo pacman -S bubblewrap # Arch
sudo apt install bubblewrap # Debian / Ubuntu
sudo dnf install bubblewrap # Fedora

Check the binary is there and that bwrap runs without root (it must not ask for sudo):

ai-jail --version
bwrap --ro-bind / / --unshare-all echo "bwrap ok without root"

If bwrap fails asking for privileges, your kernel has unprivileged user namespaces disabled; enable them (sysctl kernel.unprivileged_userns_clone=1 on older Debian/Ubuntu) before continuing.

A1 — The per-project .ai-jail file

On the first launch inside the project, ai-jail creates a .ai-jail (TOML) that is committable to the repo: any colleague who clones inherits the same policy.

cd ~/Projects/mi-app
ai-jail claude # creates .ai-jail and launches Claude Code inside the sandbox

The generated file:

# .ai-jail — sandbox configuration (commit it to the repo)
command = ["claude"]
rw_maps = [] # extra writable directories
ro_maps = [] # extra read-only directories

Before trusting the sandbox, audit it. --dry-run --verbose prints every mount point, every isolation flag and the full bwrap command, without executing anything:

ai-jail --dry-run --verbose claude

Read the output and confirm three things: that $HOME is mounted as a tmpfs (not the real one), that ~/.ssh, ~/.aws and ~/.gnupg do not appear among the mounts, and that the only writable directory is the project one. If you need an extra directory:

ai-jail --rw-map ~/Projects/shared-lib claude # extra, writable
ai-jail --map /opt/datasets claude # extra, read-only

Other agents, same binary:

ai-jail codex
ai-jail opencode
ai-jail bash # bare shell to debug the sandbox
ai-jail -- python script.py # any command

A2 — Permission allowlists with –bootstrap

--bootstrap generates each agent’s permission configuration, with sensible allow/deny/ask, and takes a backup before overwriting:

ai-jail --bootstrap

What it produces, in summary:

Agent	File	Base policy
Claude Code	`~/.claude/settings.json`	allow: `git status/diff/log`, `ls`, `grep`, `cargo`, `npm`, `python`, `docker compose` · ask: `git push`, `rm`, `docker run` · deny: `rm -rf`, `sudo`, `chmod 777`, `git push --force`
Codex	`~/.codex/config.toml`	`approval_policy = "on-request"`
OpenCode	`~/.config/opencode/opencode.json`	permissions for `bash`, `edit`, `write`

The operational key: git push is under ask, not allow, and git push --force under deny. The agent can commit, branch and rebase locally as much as it wants; none of that touches the remote. (If you use the Claude Code /sandbox, also set "allowUnsandboxedCommands": false to close the dangerouslyDisableSandbox escape hatch, which is opt-out out of the box.)

A3 — Lockdown for anything you do not trust

To audit third-party code or run an agent over a project you do not know, --lockdown goes further: project mounted read-only, GPU/Docker/display disabled, --rw-map/--map ignored, $HOME a pure tmpfs with no host dotfiles, network cut with --unshare-net and the environment wiped with --clearenv.

ai-jail --lockdown bash

It is the most restrictive sandbox possible short of a VM. Use it as your mental default for anything that is not your own code on your own machine.

A4 — The safety net: git with no push

It is not a flag, it is a property of the environment that changes the risk arithmetic. If the project is in git with a remote, and the agent does not have push permission, the worst case, that it corrupts every file in the project, is reverted with:

git checkout . # back to the last commit
# and if it touched .git (unlikely): delete the dir and re-clone

The remote was never touched. Sandbox for the filesystem + git for the code + manual push is already a reasonable level for daily use: ai-jail protects your data and the system, git protects the code, and the decision to publish is still yours.

Track B — Cluster (RKE2 with Cilium + Tetragon)

The untrusted agent, or the inference that executes generated code, runs as a pod. The same principle as on the client, different primitives. We assume a generic RKE2 cluster with Cilium as CNI and Tetragon already deployed (the eBPF agent DaemonSet on every node).

B0 — The pod baseline

Before any eBPF, the standard kit. Unprivileged securityContext, read-only root, default seccomp:

apiVersion: v1
kind: Pod
metadata:
 name: ai-agent
 namespace: agentes
 labels:
 app: ai-agent
spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 10001
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: agent
 image: registry.interno/ai-agent:pinned
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop: ["ALL"]
 volumeMounts:
 - { name: work, mountPath: /work }  # the only writable one
 volumes:
 - name: work
 emptyDir: {}

And the default egress cut, the cluster twin of --unshare-net. A default-deny egress NetworkPolicy in the namespace, opening only DNS and the bare essentials:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-egress
 namespace: agentes
spec:
 podSelector: {}
 policyTypes: ["Egress"]
 egress:
 - to:
 - namespaceSelector:
 matchLabels: { kubernetes.io/metadata.name: kube-system }
 ports:
 - { protocol: UDP, port: 53 }
 - { protocol: TCP, port: 53 }

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

For genuinely untrusted code, take it out of the shared kernel. With Kata deployed there is a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
 name: kata
handler: kata

And the pod asks for it with one line, runtimeClassName: kata, running in its own microVM with a dedicated kernel instead of sharing the node’s:

spec:
 runtimeClassName: kata  # ← the pod runs in a microVM, not on the node's kernel
 # ...the rest the same as B0

It is the cluster twin of isolation by construction: a kernel exploit inside the pod does not reach the node.

B2 — Tetragon, observation phase (Post)

Now the layer that separates a platform with runtime visibility from one without. Observe first, never kill from the outset. A TracingPolicyNamespaced, scoped to the namespace and to the agent’s label, that reports (does not kill) three things: process executions, network connections and opens of sensitive paths. action: Post only emits the event.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-observa
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- outbound connections ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchActions:
 - action: Post
 # --- opens of sensitive files ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets"
 - "/work/.git/config"
 matchActions:
 - action: Post

(Process executions need no kprobe: Tetragon emits process_exec/process_exit natively.) Deploy it and watch the events live from the node’s Tetragon pod:

kubectl apply -f agente-observa.yaml
# readable events, filtered by namespace:
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
 tetra getevents -o compact --namespace agentes

Leave this running for a typical working day of the agent. Note which destinations it really connects to (your internal registry, your HF mirror, your vLLM endpoint) and which paths it opens. That is your baseline: the list of what is legitimate. Without this step, a Sigkill kills good work and generates an availability incident, precisely what ENS asks you to avoid.

B3 — Tetragon, enforcement phase (Sigkill)

With the baseline in hand, promote to blocking. Two rules. The first: kill any connection whose destination is not on the allowlist — NotDAddr inverts the match: it fires for everything that is not those networks. The second: kill any attempt to open a secrets path.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-enforce
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- egress: kill everything that is NOT on the allowlist ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr"
 values:
 - "127.0.0.1"
 - "10.0.0.0/8" # internal cluster network
 - "172.16.10.20" # internal registry (example)
 matchActions:
 - action: Sigkill
 # --- reading secrets: kill the process ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets/kubernetes.io/serviceaccount/token"
 - "/work/.ssh"
 matchActions:
 - action: Sigkill

kubectl apply -f agente-enforce.yaml

Now the agent can do whatever it likes inside the pod, but the instant it tries to connect to a disallowed destination or read the service account token, Tetragon kills it in the kernel, before the packet leaves or the read returns bytes. It is the cluster twin of the curl blocklist and the unmounted ~/.ssh, but applied at runtime and over any binary, not just the ones you know about.

Operational warning. Enforcement with Sigkill requires a recent kernel with eBPF support for the action (5.10+ is safe). Deploy agente-enforce in a test namespace first, and keep agente-observa active in parallel: if the block fires, the Post event tells you exactly what triggered it. Adopt first, block later.

The client ↔ cluster equivalence table

The same vector, the two primitives. This is “extrapolating the technology” made explicit:

Threat vector	Client (workstation)	Cluster (RKE2)
`$HOME` / writable root	`$HOME` as ephemeral tmpfs (`bwrap`)	`readOnlyRootFilesystem: true` + `emptyDir`
Arbitrary egress	`curl`/`wget` blocklist · `--unshare-net`	default-deny NetworkPolicy + Tetragon `NotDAddr`→`Sigkill`
Reading secrets	`~/.ssh`/`~/.aws`/`~/.gnupg` not mounted	secrets outside the pod + Tetragon `security_file_open`→`Sigkill`
Kernel escape	Landlock (2nd VFS barrier)	`runtimeClassName: kata` (microVM, own kernel)
No escape hatch	process inside `bwrap`, no way out	no `privileged`, `drop ALL`, `allowPrivilegeEscalation:false`
Damage to the code	git remote with no `push` → `git checkout .`	GitOps + PR review, the agent does not apply to `main`
Visibility	`--dry-run --verbose` (static, pre-run)	Tetragon `tetra getevents` (dynamic, at runtime)

Checklist of gotchas

Do not put in a Sigkill without passing through Post. The observation baseline is not optional: it is what separates “blocking a C2” from “killing your own fine-tuning job”.
The .ai-jail gets committed; secrets do not. The TOML is policy, not credentials. Check you are not putting paths holding sensitive data into rw_maps.
readOnlyRootFilesystem breaks apps that write to /tmp. Mount an emptyDir at /tmp as well as the work one.
A NetworkPolicy with no DNS rule leaves the pod blind. Open port 53 to kube-system or nothing resolves.
Kata is not free. It adds startup latency and not every workload with special devices (GPU passthrough) fits; reserve it for the untrusted, not for everything.
The Claude Code /sandbox does not cover MCP or hooks unless you enable sandbox-runtime. If your agent uses MCP servers, assume they run with full permissions until you do.
NotDAddr with literal IPs ages badly. Document the allowlist and review it when the registry or the inference endpoint changes; consider stable internal CIDRs instead of loose IPs.

References

ai-jail (Fabio Akita), GPL-3.0: https://github.com/akitaonrails/ai-jail
bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io
Tetragon — TracingPolicy: https://tetragon.io/docs/concepts/tracing-policy/
Tetragon — enforcement (Sigkill/Override): https://tetragon.io/docs/concepts/enforcement/
Kata Containers — Kubernetes RuntimeClass: https://katacontainers.io
Kubernetes — Pod Security & seccomp: https://kubernetes.io/docs/tutorials/security/seccomp/
Kubernetes — Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Cilium: https://cilium.io

The contractor with the master key: isolating AI agents from workstation to cluster

Tue, 09 Jun 2026 16:00:00 +0200

First post of a pair on AI agent isolation. This one fixes the what and the where: the complete map of isolation primitives and which domain each belongs to. The sibling runbook fixes the how, with commands: ai-jail and bubblewrap on the client, Tetragon TracingPolicy and RuntimeClass on the cluster. If you are only going to read one, this one gives you the mental model; the other one gives you the files to copy and paste.

TL;DR

An AI agent that runs code needs access to your filesystem and your tools: compiler, linter, grep, make, cargo, npm. That is the minimum to be useful. The problem is that alongside that access travels the ability to read ~/.aws/credentials, exfiltrate your SSH keys or fire an rm -rf outside the project directory. And no malicious model is needed: a compromised dependency in an npm install is enough, because the well-meaning agent and the poisoned post-install script run with the same permissions. The answer is not to trust the LLM’s good intentions; it is to isolate in order to bound the blast radius. This post walks the five isolation families of 2026, from the process sandbox to the full VM, and sorts them into two columns: what applies on the client (the developer’s workstation: bubblewrap, ai-jail, sandbox-exec, Landlock, the native Claude Code and Codex sandboxes) and what applies on the cluster (where the agent or the inference runs on Kubernetes: namespaces+seccomp, gVisor, Firecracker/Kata microVMs and eBPF/Tetragon as the layer for live observation and enforcement). The thesis: the threat model is the same in both places; the tools are not. Policy extrapolates; the primitive gets rewritten.

The analogy: the contractor with the master key

You hire a builder for a renovation. He is competent and acts in good faith. But two things happen that you do not control. The first: he may misread the instruction and knock down the wrong partition wall. The second, worse: his toolbox may have been tampered with before he came through your door — somebody put something inside — and when he opens it in your living room, that something activates.

Nobody sensible gives him the master key to the whole building. You open the room he works in, you leave him the tools he needs, and you keep the study with the safe and the server room locked. If the renovation goes wrong, by error or by sabotage, the damage stays in that room.

An AI agent is that contractor. The sandbox is the key policy: you give it the project room and the tools, not the master key to the system. And here is the twist that justifies two posts: the builder works in two different buildings. One is your flat, the developer’s workstation, with your credentials, your ~/.ssh, the browser’s password vault. The other is the datacenter, the cluster where inference and autonomous agents serve customers, with data from several tenants at once. The key policy is identical in both: least privilege, bound the radius. But the lock on your flat’s door is not the same as the one at the datacenter. In the flat you fit a cylinder lock (bubblewrap). At the datacenter you put a guard watching every door and a separate wing of the building (Tetragon + microVM). Same principle, different hardware. That is extrapolating the technology, not copying it.

The threat model: what a runaway agent can do

Before choosing a lock it helps to enumerate the burglar. The attack surface of an agent that executes arbitrary bash breaks down into five concrete threats. Not all are defended by the same layer and, an uncomfortable fact that the security documentation itself acknowledges, no single layer covers them all.

Threat	What the agent does	Minimum layer that cuts it
Filesystem out of scope	Reads `.env`, `~/.ssh/id_rsa`, system secrets; modifies sources outside the project	Process sandbox (path allowlist)
Arbitrary network egress	Exfiltrates data, receives instructions from a remote C2, calls unauthorised APIs	Network blocking / NetworkPolicy / microVM
Kernel syscall surface	A kernel exploit from inside the container escalates to the host (shared kernel)	gVisor or microVM (dedicated kernel)
Cross-tenant leakage	One customer’s workload reads another’s data on a multi-tenant platform	microVM (de facto standard)
Secret exfiltration	Extracts tokens and environment variables via `/proc` or the environment	`--clearenv` / tmpfs `$HOME` / secrets outside the pod

There is a sixth threat that no sandbox solves: prompt injection. If an attacker manages to slip instructions into the agent’s context — a poisoned comment in the code, a malicious file the agent reads, an adversarial response from a tool — the agent will execute those instructions with whatever permissions the sandbox grants it. Isolation shrinks the impact radius of a successful injection; it does not prevent the injection. That is why the sandbox is a layer, not the solution: on top go input validation, tool-call allowlists and output auditing. The sentence to internalise: isolation does not make the agent trustworthy; it bounds what an untrustworthy agent can break.

Two domains, one policy

The builder works in two buildings. The distribution of tools reads better in two columns:

The client: isolating the agent on the workstation

Here the agent is a coding assistant — Claude Code, Codex, OpenCode, Cursor — that a developer launches on their machine. One process, one user, and right next to it the juiciest assets that exist: ~/.aws/credentials, ~/.ssh, ~/.gnupg, the browser’s password store. The matching tier is the lightest one: the process sandbox.

bubblewrap (Linux) and sandbox-exec (macOS). bubblewrap (bwrap) is the same sandbox Flatpak uses to isolate each desktop app: a ~50 KB binary, ~4,000 lines of C, maintained by the GNOME team and, the key property, it runs without root via CLONE_NEWUSER, creating namespaces with no elevated privileges. It mounts $HOME as an ephemeral tmpfs and only exposes the project directory, writable; the rest of the system becomes invisible. On macOS the equivalent is sandbox-exec with SBPL profiles: a legacy Apple API, officially deprecated and with no public replacement, but working today. Parity between the two is not exact — on macOS the GPU (Metal) and the display (Cocoa) are system services and sandbox-exec does not restrict them — but both protect what matters: access to the sensitive areas of the filesystem.

Landlock as a second barrier. bubblewrap isolates via namespaces and mounts; Landlock, a Linux Security Module available since kernel 5.13, restricts access at the VFS level, independently of namespaces. It does not replace bwrap: it complements it. It closes vectors that mount-based isolation does not cover on its own (escape routes via /proc, symlink tricks inside allowed mounts) and acts as a safety net should the namespace machinery have a bug. It is defence in depth inside the client itself, and it degrades cleanly to a no-op on kernels that do not support it.

Dev containers, when reproducibility is needed. A dev container (devcontainer.json, what Codespaces and Cursor use) is a Docker container with a configuration layer on top. It gives reasonable filesystem isolation and easy reset (destroy and recreate), but it shares the host kernel, the same limitation as any Docker, and it tends to be long-lived, accumulating state. For an agent running your own team’s code on your own machine it is a good repeatability story; it is not, on its own, the isolation layer for untrusted code.

What wraps all of this. A hand-written bash script works, but it does not scale to a team. The tools that package it:

ai-jail (Rust, GPL-3.0): wraps bwrap/sandbox-exec with per-project config in a .ai-jail file (TOML, committable to the repo, so the whole team inherits the same policy), auto-detection of GPU/Docker/display, a --lockdown mode (project read-only, network cut with --unshare-net, --clearenv), --dry-run for auditing, and --bootstrap to generate each agent’s permission allowlists. It is tool-agnostic: the same binary serves Claude, Codex, OpenCode or Crush. It also applies Landlock automatically on 5.13+ kernels as defence in depth.
The Claude Code /sandbox: since October 2025 Claude Code ships its own sandbox that uses, exactly, bubblewrap on Linux and sandbox-exec on macOS. Its Sandboxed Bash isolates shell commands, but not the file tools, the MCP servers or the hooks, which run with the process’s full permissions unless you enable the beta sandbox-runtime package, which wraps the whole process. There is one nuance worth knowing: if a command fails because of a restriction, the agent can retry with dangerouslyDisableSandbox — it is opt-out, not opt-in.
Codex CLI: three modes via --sandbox (read-only, workspace-write, danger-full-access); the recommended default is workspace-write. The philosophy is deliberate: Codex does not provide the isolation, it delegates it to the environment that wraps it. danger-full-access only makes sense inside a microVM.
Cursor: its cloud agents run in isolated VMs; /worktree creates a single-use isolated worktree per task, and /best-of-n launches several attempts in parallel in separate worktrees.

The cluster: isolating the agent in production

The second building is the datacenter. Here the “agent” can be an autonomous agent running with no human in front of it, or the inference service itself executing generated code, or a multi-tenant workload where one customer’s pod must not touch another’s data. The process is no longer one: they are pods in a Kubernetes cluster (RKE2/RKE3 on a typical sovereign platform). The primitives change in nature.

The pod baseline. First of all, the standard kit: Linux namespaces, seccomp (RuntimeDefault) to trim the syscall surface, cgroups for resource limits, an unprivileged securityContext (runAsNonRoot, readOnlyRootFilesystem, dropping all capabilities) and NetworkPolicy to cut egress. It is the cluster equivalent of the process sandbox allowlist. Necessary, but it shares the kernel with the host: insufficient for genuinely untrusted code.

gVisor (runsc). Google’s user-space kernel: it intercepts the workload’s syscalls before they reach the host kernel and serves them inside a Linux kernel reimplemented in Go (the Sentry). The surface exposed to host kernel vulnerabilities drops sharply, while keeping fast startup and a low footprint. It is the middle ground when the risk of a kernel escape is real but the overhead of a microVM is not acceptable.

Firecracker / Kata microVMs. The de facto standard for untrusted code in 2026. Firecracker (AWS’s VMM written in Rust, on top of KVM) gives each sandbox a dedicated Linux kernel: a kernel exploit inside the microVM does not reach the host by construction. It is what sits underneath Vercel Sandbox (GA January 2026) and E2B. On Kubernetes, Kata Containers brings that model to a RuntimeClass: you mark the untrusted agent’s pod with runtimeClassName: kata and it runs in its own microVM instead of sharing the node’s kernel. For multi-tenant with generated code, this is the baseline, not the luxury.

eBPF / Tetragon: the layer we already have. Here is the piece that separates a platform with runtime observability from one that only trusts its configuration. The previous layers are static: they define what the pod can do before it starts. Tetragon, Cilium’s runtime security component, based on eBPF, is dynamic: it observes, in the kernel and at minimal cost, every process execution, every network connection and every file open in every pod, and it can act inline. It does not replace the sandbox; it watches it from inside the kernel. Where bubblewrap on the client blocks curl with a command blocklist, Tetragon on the cluster hooks tcp_connect in the kernel and, if the destination is not allowed, kills the process with Sigkill before the packet leaves. Where the client hides ~/.ssh behind a tmpfs, Tetragon hooks security_file_open and reports, or kills, any attempt to read a sensitive mounted path. It is the guard patrolling the corridors while the microVMs are the walls. And it is exactly the kind of control that materialises the ENS monitoring and traceability measures (op.mon, op.exp) without instrumenting the application: visibility lives in the kernel, not in the agent’s code.

The landscape table

The five families, their relative isolation strength, their startup cost and the domain where they live:

Tier	Primitive	Isolation	Startup	Natural domain
Process sandbox	Seatbelt · bubblewrap	Baseline	~0 ms	Client (Claude Code default)
Dev container	Docker + seccomp	Moderate	seconds	Client / cluster (repeatability)
User-space kernel	gVisor (`runsc`)	Strong	ms	Cluster (medium multi-tenant)
microVM	Firecracker · Kata	Strongest (practical)	<1 s	Cluster (untrusted code)
Full VM	KVM · EC2	Maximum	30 s+	Cluster (external boundary, compliance)
Runtime enforcement	eBPF · Tetragon	Cross-cutting	always on	Cluster (watches+kills on top of any tier)

Tetragon gets its own row on purpose: it is not a rung on the ladder, it is a cross-cutting layer that operates on top of any of the others. It stacks with all of them.

A numerical note on why the “startup” column decides as much as the “isolation” column. A full VM wins on raw isolation but takes tens of seconds to provision; for an agent that needs a fresh environment per request or per session, that cost is prohibitive. A Firecracker microVM starts in under 1 second and a process sandbox in ~0 ms. That is why the dominant pattern in 2026 is not “the most isolated VM”, but a full VM as the external boundary plus a microVM as the per-request execution unit inside, the architecture of Vercel, AWS Lambda and E2B. On the client the arithmetic is the opposite: the developer launches the agent dozens of times a day interactively, and a startup measured in seconds would break the flow; hence the process sandbox, with its microsecond overhead, is the correct default.

Extrapolate, do not copy

The thesis of this pair of posts fits in one sentence: the threat model is invariant across domains; the primitive that implements it is not. The client and the cluster defend exactly the same five vectors — filesystem, network, kernel, multi-tenant, secrets — but with toolboxes that do not overlap. Every control has its twin on the other side:

$HOME as an ephemeral tmpfs (client) ↔ readOnlyRootFilesystem + emptyDir (cluster).
A curl/wget blocklist in bwrap (client) ↔ a TracingPolicy on tcp_connect in Tetragon plus NetworkPolicy (cluster).
--unshare-net in lockdown (client) ↔ default-deny NetworkPolicy (cluster).
No escape hatch, the process lives inside bwrap (client) ↔ no privileged, no hostPath, RuntimeClass kata (cluster).
~/.ssh and ~/.aws never mounted (client) ↔ secrets outside the pod plus Tetragon watching security_file_open (cluster).

The sibling runbook turns each of these equivalences into concrete files. What matters here is the method: when someone shows you an agent sandbox, be it the Claude Code /sandbox on a laptop or a microVM on a PaaS, the useful question is not “which tool does it use?”, but “which of the five vectors does it close, and which does it leave open?”. The tool gets replaced; the threat map stays.

What no sandbox solves

Three limits that Anthropic’s own documentation states, and which are worth keeping in view so as not to sell smoke:

Egress remains a risk in any sandbox that allows outbound connections. If the agent can open a connection, it can exfiltrate. That is why the client lockdown cuts the network and the cluster uses default-deny NetworkPolicy plus Tetragon: you do not trust “filtering well”, you trust “not letting anything out”.
Code modification remains possible in any sandbox with the project directory mounted writable. The remedy is not a sandbox-technical one, it is git: with the remote untouched and no push permission, the worst case is corrupting the local working copy — git checkout . and start again. The damage never reaches the remote.
No sandbox prevents a compromised prompt from reaching the API. Isolation bounds the impact of an injection; it does not prevent it. The complementary defences — input validation, tool-call allowlists, output auditing — are mandatory alongside isolation, not instead of it.

The operational conclusion: isolation shrinks the blast radius; defence in depth is what closes the circle. A process sandbox for trusted code on a known machine is appropriate and practically free. For an agent acting on user prompts, executing generated code or running multi-tenant, the acceptable minimum in 2026 is a microVM, with Tetragon watching from above. Pick the tier that matches your real threat, check which vector it leaves open, and stack complementary controls on top.

References

bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io · https://docs.rs/landlock
ai-jail (Fabio Akita): https://github.com/akitaonrails/ai-jail
gVisor: https://gvisor.dev
Firecracker: https://firecracker-microvm.github.io
Kata Containers: https://katacontainers.io
Tetragon (Cilium): https://tetragon.io
Vercel Sandbox — concepts: https://vercel.com/docs/vercel-sandbox/concepts
E2B: https://github.com/e2b-dev/E2B
Claude Code sandboxing: https://docs.claude.com/en/docs/claude-code
Codex CLI: https://github.com/openai/codex

The corridors and the security guard: PCIe topology, GPUDirect P2P and ACS

Mon, 08 Jun 2026 06:00:00 +0200

This continues the below the engine series. The NVLink and NCCL post explained the shared bench over which the GPUs pass data to each other at 450 GB/s. But that bench only connects GPUs to each other. Everything else, disk, network, the host, travels over another bus, PCIe, and through its corridors. The cold start already brushed against this with GPUDirect Storage; this post opens up the complete floor plan of the corridors and the guard who watches them.

TL;DR

On a 4×H100 SXM node, the GPUs talk over NVLink (450 GB/s per direction, around 7× PCIe), and for the all-reduce of tensor parallel that is the path. But PCIe does not go away: it is where everything else comes in. The weights come up from the NVMe over PCIe (the cold start), the data from another node arrives via the NIC over PCIe (RDMA), and a KV that moves between nodes travels over PCIe. GPUDirect is the family that lets those bytes go straight from the device into the HBM without bouncing through host RAM: P2P (GPU↔GPU), RDMA (GPU↔NIC) and Storage (GPU↔NVMe). The obstacle is a guard called ACS (Access Control Services): a PCIe security feature that by default forces peer-to-peer traffic up to the root complex for inspection, which destroys the direct path and inserts a detour through the CPU. The IOMMU (VT-d) does something similar if it is not in passthrough mode. Disabling them gives performance; keeping them gives isolation and virtualisation, and that is a real decision in an ENS environment. This post explains the topology (nvidia-smi topo -m), GPUDirect, why ACS and IOMMU break P2P with numbers, the 10 knobs and the trap of removing the guard without knowing what it was watching. On the generic 4×H100 SXM cluster.

Where you are: the corridors, not the bench

Picture the kitchen as a building. The elite cooking stations, the GPUs, are in a room with a huge central bench (NVLink/NVSwitch) over which they pass ingredients at full speed without standing up. That bench is for them and them alone.

But the building has more to it: the pantry (the NVMe storage), the loading door (the network, the NIC) and reception (the CPU and its RAM). To reach any of those, the stations do not use the central bench: they use the corridors of the building, the PCIe bus. And here the character of the post appears: at the entrance of each corridor there is a security guard (ACS) who, by default, does not let two stations hand each other something directly along the corridor: it forces them to take the parcel up to reception to be checked, and only then does it come down to its destination. It is safe, but it is an absurd detour when the two stations are right next to each other. GPUDirect is the permission to skip that detour; ACS and IOMMU are the reasons why, often, you cannot.

The topology of a node: two buses, not one

The most common mistake is to think that a node has “a bus”. There are (at least) two, and they do different things:

NVLink / NVSwitch — the high-speed GPU↔GPU mesh. On H100 SXM, 18 links × 50 GB/s = 900 GB/s bidirectional between any two GPUs, with NVSwitch giving an all-to-all with no contention (NVLink, NVIDIA). It is the shared bench.
PCIe Gen5 — the general I/O bus. An x16 link gives 128 GB/s bidirectional (around 64 per direction) (H100 product brief). It connects each GPU with the CPU, the RAM, the NICs and the NVMes. It is the corridor.

The difference is 7×: NVLink moves in one second what PCIe takes seven to move. That is why intra-node tensor parallel goes over NVLink and nobody argues about it. PCIe matters for the other things: pulling weights off disk, receiving from the network, moving KV between nodes.

The tool for seeing this is nvidia-smi topo -m, which prints a matrix of how each pair is connected (NVIDIA forum):

 GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA
GPU0 X NV18 NV18 NV18 PXB 0-47 0
GPU1 NV18 X NV18 NV18 PXB 0-47 0
GPU2 NV18 NV18 X NV18 SYS 48-95 1
GPU3 NV18 NV18 NV18 X SYS 48-95 1

The legend is what matters: NV18 = 18 NVLink links (the bench); PXB = crosses PCIe switches but not the host; PHB = goes through the host bridge; NODE = same NUMA, crossing PCIe; SYS = crosses the inter-socket interconnect (the worst case, it traverses NUMA). The fact that GPU0↔NIC0 is PXB and GPU2↔NIC0 is SYS tells you exactly which GPU should handle that NIC’s traffic, GPU 0, without crossing NUMA. This links directly with the NUMA post and the network NUMA one: PCIe affinity is NUMA affinity.

GPUDirect: skipping reception

Without GPUDirect, moving a piece of data from the NIC (or the NVMe) to the GPU takes a mandatory detour: device → host RAM → GPU. That bounce through RAM consumes CPU bandwidth, spends copies and adds latency. GPUDirect removes the bounce by letting the data go straight from the device into the HBM. Three flavours:

GPUDirect P2P — GPU↔GPU over PCIe (when there is no NVLink between them, or for traffic that does not use the bench).
GPUDirect RDMA — GPU↔NIC: the network card writes directly into the HBM. It is what makes efficient multi-node viable (NCCL over InfiniBand/RoCE).
GPUDirect Storage (GDS) — GPU↔NVMe: the disk writes straight into the HBM, with no host buffer. It is the lever of the cold start.

On an SXM node, the GPU↔GPU traffic of tensor parallel does not use P2P over PCIe: it uses NVLink. That is why GPUDirect matters above all at the edges of the node: the network (RDMA, for multi-node) and the disk (GDS, for startup). That is where ACS does damage.

The guard: why ACS and IOMMU break P2P

ACS (Access Control Services) is a PCIe security feature designed for virtualisation and isolation: it guarantees that one device cannot read from or write to another directly without the root complex mediating. To achieve that, it forces peer-to-peer transactions up to the root complex and back down (GDS best practices, NVIDIA). It is exactly the opposite of what GPUDirect wants: the direct path stops being direct.

The IOMMU (VT-d on Intel, the equivalent on AMD) translates addresses and isolates devices. If it is active and not in passthrough mode, it also redirects P2P traffic through the root complex, with the same effect: performance through the floor or, in extreme cases, hangs (NCCL troubleshooting).

Summarised without detours (Giraud, debugging P2P): ACS forces the trip through the root for security checks; IOMMU forces it for isolation and virtualisation. Both break the goal of P2P (direct communication with no intermediaries) and add overhead. If you do not need that security/virtualisation on that path, disabling them recovers the performance. The operational recipe for maximum GPUDirect performance: ACS off on the switches along the path and IOMMU in passthrough (iommu=pt) or disabled.

The maths that matter: how much the detour costs

Take a KV SWAP of 5 GB (preemption from the scheduler that sends KV to the host, or a transfer between nodes in disaggregated serving):

Path	Effective BW	Time for 5 GB
NVLink (GPU↔GPU intra-node)	~450 GB/s	~11 ms
Direct PCIe Gen5 x16 (P2P, ACS off)	~55 GB/s	~91 ms
PCIe via root complex (ACS on)	~25-30 GB/s*	~170-200 ms

* The detour through the root does not just “add latency”: it saturates the host bridge bandwidth, contends with other traffic and, depending on the topology, can fall well below the direct path. The figure is an order of magnitude, to show the size of the problem, not a benchmark.

The reading: on the path that does use PCIe (network, disk, swap), having ACS on can double or triple the time. And if that time is on the critical path, a cold start, a preemption swap, an inter-node all-reduce, it shows up in the latency the user sees. What disabling ACS does not fix: traffic that already went over NVLink (intra-node TP). There, ACS is irrelevant.

The real tension: performance vs isolation (and ENS)

Here the post gets serious, because the “disable ACS and IOMMU” recipe has a cost that in a regulated environment is not free. ACS and IOMMU exist for a reason: to isolate devices. On a dedicated bare-metal node for inference, with no virtualisation or multi-tenancy, you are not isolating anything that matters and disabling them is reasonable. But:

If you do GPU passthrough to VMs or use containers with strong isolation, the IOMMU is necessary, not optional.
In a multi-tenant scenario where several workloads share a node, ACS provides a guarantee that one device does not snoop on another.
Under ENS (see technical controls), workload isolation and access traceability can be requirements; disabling bus isolation to gain 80 ms is a decision you have to justify and document, not a silent tweak.

The design way out, when you need both: keep the isolation where compliance requires it and design so that the hot path does not depend on P2P over PCIe. Concretely, on an SXM node, the bulk of the critical traffic (TP) already goes over NVLink and ACS does not affect it. For the network, dedicate one NIC per GPU on its own PCIe switch (PXB) and use GPUDirect RDMA only on the path you control. For the disk, cache weights on local NVMe. That way you do not pay the choice between performance and isolation: you avoid it on the path that matters.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	ACS off (switches on the path)	P2P detour through root	you lose bus isolation
2	`iommu=pt` / off	P2P redirection through root	breaks VM passthrough if off
3	`nvidia-smi topo -m`	audit the real topology	— (always worth it)
4	`p2pBandwidthLatencyTest`	measure P2P for real	— (verify before assuming)
5	NIC placement	same PCIe switch as the GPU	SYS if it crosses NUMA
6	NVMe placement	NUMA-local to the GPU	H2D crossing UPI
7	`nvidia-peermem` (GDR)	enables RDMA into HBM	correct driver/kernel
8	PCIe gen/lanes (x16)	width of the corridor	GPU silently at x8
9	relaxed ordering / ASPM	PCIe latency and power	jitter if misconfigured
10	persistence mode	avoids re-init of the path	idle GPU still paid for

How it connects with the rest of the stack

With NVLink and NCCL. The NVLink post covers the GPU↔GPU bench; this one covers the other bus, the one connecting to disk, network and host. They are complementary: ACS affects PCIe, not NVLink.

With the cold start. The GPUDirect Storage of the disk→HBM post is GPUDirect over the storage path; ACS on strangles it just as it strangles P2P.

With NUMA. The PCIe affinity of topo -m is the NUMA affinity of the host post; placing NIC and NVMe on the correct NUMA avoids the SYS path.

With the network. NIC placement and GPUDirect RDMA is the subject of the network NUMA post; the same principle of “get the CPU out of the way”.

With PagedAttention and the scheduler. The preemption SWAP (scheduler) moves KV blocks over PCIe; that is why V1 prefers RECOMPUTE and why this bus matters.

With disaggregated serving. Transferring KV between pools in disaggregated serving travels PCIe→NIC→PCIe; ACS and placement decide whether it is viable.

With ENS. Bus isolation is a technical control; see ENS/42001/AI Act controls.

Traps and things that are not what they look like

“Disable ACS everywhere, it goes faster.” On a dedicated node, fine. On one with virtualisation, multi-tenancy or isolation requirements (ENS), you are removing a security control. The correct decision is per path and documented, not global and silent.

“NVLink and PCIe are the same bus, more or less.” No. They are two buses with a 7× difference and different purposes. TP goes over NVLink; disk, network and host go over PCIe. Confusing them leads to “optimising” ACS for traffic that does not even go through PCIe.

“P2P just works, there is nothing to check.” P2P is silently disabled with ACS/IOMMU active, and many distros enable them by default. Check with p2pBandwidthLatencyTest and nvidia-smi topo -m; do not assume you have the direct path just because the GPUs are on the same node.

“IOMMU off always, for performance.” If you do GPU passthrough to virtual machines, the IOMMU is mandatory; disabling it breaks the passthrough. The correct mode is usually passthrough (iommu=pt): it keeps the necessary mapping without penalising P2P.

“More PCIe lanes = faster GPU.” PCIe is the I/O path, not the compute path. A GPU at x8 instead of x16 takes longer to load and to communicate over PCIe, but generates tokens at the same speed once the weights are inside. The damage from x8 is in the cold start and in multi-node, not in decode throughput.

“GPUDirect fixes any I/O bottleneck.” GPUDirect removes the bounce through the CPU; if your bottleneck is the device itself (saturated NVMe, NIC at full tilt) or the topology (a SYS path crossing NUMA), GPUDirect does not touch it. Measure where the bottleneck is first.

Conclusion

This whole series has been going down floors looking for where the time goes, and this one reaches the wiring of the building. Intuition treats the node as a homogeneous box where “the GPUs talk to everything”; the reality is that there are two buses with opposite purposes, an elite bench for the GPUs (NVLink) and some service corridors for everything else (PCIe), and a security guard in the corridors who, with the best of intentions, makes every parcel go up to reception before it is delivered. GPUDirect is the permission for direct delivery; ACS and IOMMU are the legitimate reasons why sometimes you are not given it. The lesson is not “disable the guard”: it is understanding which path is critical (almost never the one you think) and what the guard was watching before sending it home. On a dedicated node, the direct path is almost free and worth taking. On one that shares workloads or lives under ENS, bus isolation is a control you sacrifice with justification or do not sacrifice at all. Good design does not choose between performance and isolation blindly: it puts the critical traffic on the bench that needs no guard, and leaves the corridors for what can wait.

References

NVIDIA, GPUDirect Storage Best Practices Guide (ACS, IOMMU, paths): https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html.
NVIDIA, NCCL Troubleshooting (IOMMU/VT-d and P2P): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html.
M. Giraud, Multi-GPU (NVIDIA) P2P capabilities and debugging tips: https://morgangiraud.medium.com/multi-gpu-nvidia-p2p-capabilities-and-debugging-tips-fb7597b4e2b5.
NVIDIA, H100 Product Brief (PCIe Gen5, NVLink 900 GB/s): https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/h100/PB-11773-001_v01.pdf.
NVIDIA, GPUDirect RDMA documentation: https://docs.nvidia.com/cuda/gpudirect-rdma/index.html.

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

Mon, 08 Jun 2026 05:40:00 +0200

Part of the under the engine series. The PagedAttention post explained where the KV lives (in paged blocks). This one explains who reads it and how: the attention kernel. And it connects with FlashAttention v1-v4, which took that kernel apart from the inside; here we look at the level above, how vLLM chooses between several kernels and why it needs more than one.

TL;DR

An LLM forward is, for the most part, standard matrix multiplications that any library does well. The exception that decides performance is attention, and having one good kernel is not enough: you need two, because the two phases of inference are physically opposite problems. Prefill processes the whole prompt: many queries against many keys, dense and compute-bound, the home ground of FlashAttention’s IO-aware tiling. Decode generates one token: a single query against all the accumulated KV, thin and memory-bound, where the only thing that matters is saturating HBM bandwidth while reading the paged KV. That is why vLLM does not have “the attention kernel” but a switchable backend (FLASH_ATTN, FLASHINFER, TRITON_ATTN and others) plus logic that picks according to the GPU: by default FA4 on Blackwell (SM100), FA3 on Hopper (SM90), FA2 elsewhere, with FlashInfer as the alternative that compiles bespoke kernels (JIT) and knows how to do cascade attention for shared prefixes. This post explains why prefill and decode are opposites (with arithmetic intensity), how the backend reads paged KV, how the engine chooses, what FlashInfer brings, the 10 knobs and the trap of pinning a backend blindly. On the generic 4×H100 SXM cluster.

Where you are: the specialist, not the commis

In the kitchen, almost all the work is chopping and sautéing: standard operations any competent commis chef executes, the matrix multiplications of the feed-forward layers and the projections. There is one dish that is not delegated: the signature dish, the one that defines the restaurant. That dish is attention, and it has a quirk: it is cooked in two radically different ways depending on the moment of service.

During prefill, when a new order arrives with its whole prompt, you cook big: a lot of raw material at once, a lot of fire, an intense operation that fills the stoves. During decode, when a table asks for “one more dish”, you cook à la carte: a single dish, but you have to go to the larder and bring back every ingredient that table has accumulated over the whole meal. One is a problem of firepower; the other, of larder speed. The same specialist does not do both well. That is why vLLM has several, and a chef who decides which one steps up depending on the GPU and the phase. That is the attention backend.

Why prefill and decode are opposite problems

This is the central idea, and one sum proves it: arithmetic intensity (FLOPs per byte read). An operation with high intensity is limited by compute; one with low intensity, by memory.

Prefill. We serve $N$ queries (the whole prompt) against $N$ keys. The $QK^\top$ operation and the $\text{softmax}\cdot V$ do on the order of $N^2 d$ FLOPs and read on the order of $N d$ of data. Intensity grows with $N$:

$$I_\text{prefill} \sim \frac{N^2 d}{N d} = N$$

With large $N$ (a prompt of thousands of tokens), intensity is high: compute-bound. This is where FlashAttention’s tiling squeezes the tensor cores and where you get close to the GPU’s peak TFLOPS.

Decode. We serve one query (the new token) against $L$ keys (all the accumulated KV). FLOPs on the order of $L d$; bytes read on the order of $L d s$ (the entire KV has to be read from HBM). Intensity is:

$$I_\text{decode} \sim \frac{L d}{L d s} = \frac{1}{s} \quad (\approx 0.5 \text{ FLOP/byte in FP16})$$

Constant and tiny: memory-bound. The decode kernel is not limited by how much the GPU can compute but by how fast it reads the KV from HBM. It makes no difference that the H100 has 132 idle SMs (see the SM post): the bottleneck is the 3.35 TB/s of bandwidth, and the decode kernel exists so as not to waste a single one of those bytes/s.

The design consequence: a kernel optimised for prefill (dense tiling, maximum tensor-core occupancy) is not the optimum for decode (coalesced reads of the paged KV, minimum latency). Serious servers have different kernels, or one kernel with two paths. In models with MLA (multi-head latent attention), vLLM goes as far as using separate backends for prefill and decode, selectable independently (attention backends, vLLM).

The scheduler’s trick: prefill and decode in the same forward

Here the circle closes with the scheduler post. Because vLLM V1 mixes requests in prefill and in decode in every step, one single forward has to serve both. The backend receives metadata telling it, for each sequence in the batch, how many queries it brings and how much KV it has to read, and applies the right path to each one. That is why the attention backend and the scheduler are coupled: the first has to digest the heterogeneous batch the second builds.

How the backend reads the paged KV

The kernel does not receive a contiguous KV tensor: it receives the block table from the block manager and does a gather over the physical blocks. This imposes a real constraint on the backend: it has to support the paged layout and vLLM’s block_size. Not every kernel in the world does; the ones vLLM integrates (FlashAttention, FlashInfer, Triton) are adapted to read KV in fixed-size blocks scattered across HBM. That is why you cannot plug in any attention kernel from a paper: it has to speak the language of the pigeonhole larder.

The backends and how the engine chooses

vLLM exposes a backend abstraction with several implementations (vLLM deepwiki):

FLASH_ATTN — the FlashAttention family. By default the version is picked by architecture: FA4 on SM100 (Blackwell), FA3 on SM90 (Hopper), FA2 elsewhere, configurable with flash_attn_version.
FLASHINFER — an attention engine with JIT compilation and specialisable kernels; strong on heterogeneous KV and shared prefixes.
TRITON_ATTN — written in Triton, portable and with no dependency on precompiled CUDA binaries (Triton backend deep dive, vLLM, mar-2026).
Specific backends for MLA and for non-NVIDIA hardware.

Selection is automatic unless you force it with VLLM_ATTENTION_BACKEND. The heuristic tries FlashAttention first; on Blackwell (SM100) the fallback order for MLA is TRT-LLM Ragged → FlashInfer → others; on other GPUs only FlashAttention is considered for the main path (attention backends, vLLM). The decision depends on: architecture (SM), dtype (FP16/BF16/FP8), head dimension, and whether the load needs a feature only one backend has (cascade attention, certain soft caps, FP8 in KV).

What FlashInfer brings: JIT and cascade attention

FlashInfer does not compete with FlashAttention on “being a bit faster”; it attacks a different problem: the heterogeneity of the KV in real serving (FlashInfer, arXiv 2501.01005). Two ideas:

JIT compilation. Instead of a monolithic kernel, FlashInfer generates bespoke kernels for the attention variant, the problem shape and the KV layout you have, injecting functors (query/key/logit transformations, masks). It specialises rather than generalises.

Cascade attention. Here is the jewel for serving with shared prefixes. If $R$ requests share a prefix of $P$ tokens (a common system prompt), naive attention would read that prefix $R$ times. Cascade attention computes it once against the shared prefix and then combines with each request’s own suffix:

$$\text{reads: } \underbrace{R \cdot (P + s_i)}_{\text{naive}} \;\longrightarrow\; \underbrace{P + \textstyle\sum_i s_i}_{\text{cascade}}$$

With $R=50$ requests and a prefix $P=1000$, that is reading 50,000 prefix tokens versus 1,000. It is the natural complement to prefix caching: the block manager shares the prefix’s memory, and cascade attention shares the compute of attending over it.

The arithmetic that matters: when switching backend gives you something

The backend only moves the needle where attention is the bottleneck. In memory-bound decode, a kernel that exploits HBM bandwidth better gives a real improvement; in compute-bound prefill with long sequences, FA3/FA4 getting close to the tensor-core peak gives a real improvement. But if your bottleneck is in another layer, launch overhead, a badly sized scheduler, the cold start, then switching backend does not touch that part. The rule, once again: measure the regime before optimising.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	`VLLM_ATTENTION_BACKEND`	force a backend	mismatch with hardware/feature
2	`flash_attn_version` (2/3/4)	FA version	version unsupported on your SM
3	enable FlashInfer	JIT + cascade	initial JIT compilation time
4	cascade attention	prefix compute reuse	only helps with a heavily shared prefix
5	`kv_cache_dtype` (FP8)	FP8 support in the kernel	not every backend/SM supports it
6	`block_size`	layout the kernel must read	consistency with PagedAttention
7	MLA prefill backend	kernel for the dense phase	MLA models only
8	MLA decode backend	kernel for the thin phase	MLA models only
9	soft cap / sliding window	features that restrict backends	fewer kernel options
10	head_dim / variant	which kernels are eligible	exotic models with no support

How it connects to the rest of the stack

With FlashAttention. The FA post explains the kernel from the inside (tiling, online softmax, FA1-4); this is the level above, how vLLM chooses between kernels and why it needs more than one.

With PagedAttention. The backend reads the KV that the block manager places in blocks; it has to speak block-table.

With the scheduler. The scheduler builds mixed prefill+decode batches; the backend has to serve both regimes in a single forward.

With CUDA graphs. Attention kernels are captured in the CUDA graphs; a backend that launches many small kernels benefits more from capture.

With prefix caching. Cascade attention is the compute side of what prefix caching does in memory.

With FP8. Attending over KV in FP8 requires the backend to have the FP8 path; not all of them do on every architecture.

Traps and things that are not what they look like

“FlashInfer is always faster than FlashAttention.” No. FlashInfer wins when its specialisation (cascade, heterogeneous KV, a particular attention variant) applies to your load; in classic dense prefill, FA3/FA4 usually does as well or better. It depends on the regime, there is no universal winner.

“A good attention kernel works for everything.” The underlying error of this post. Prefill and decode are compute-bound and memory-bound respectively; a kernel tuned for one wastes effort on the other. That is why there are separate paths (and separate backends in MLA).

“Decode is compute-bound because the GPU is at 100%.” nvidia-smi at 100% is misleading (see the SM post): decode is memory-bound, the GPU is moving KV, not computing. Optimising decode’s compute is polishing what is not the bottleneck.

“I pin VLLM_ATTENTION_BACKEND and forget about it.” Pinning a backend by hand can leave you on a suboptimal one when you change GPU or version, or force a slow fallback if your hardware does not support what you asked for. Autoselection usually gets it right; pin it only with a measurement that justifies it.

“Cascade attention always helps.” Only with a heavily shared prefix across many concurrent requests. If each request has its own context, there is nothing to share and the overhead of organising the cascade is not amortised.

“The attention backend is the bottleneck, that is why I am slow.” Almost always the bottleneck is higher up (launch, scheduling, memory) or lower down (bandwidth). The backend matters where attention dominates; measure it with nsys/DCGM before changing it.

Conclusion

Of everything an LLM does while generating text, almost all of it is matrix multiplications that any library resolves. Performance is decided in a single kernel, attention, and the surprise is that it is not even one kernel: it is two opposite problems wearing the same name. Prefill wants fire, dense compute over thousands of tokens, and decode wants a fast larder, reading a token’s entire KV with the minimum waste of bandwidth. That is why vLLM did not pick a winning kernel but an abstraction that switches: FlashAttention tuned to each architecture for the general case, FlashInfer compiling bespoke kernels when there is heterogeneity or prefixes to share, Triton for portability. The head chef does not cook the signature dish just one way: he looks at who is ordering and at what point of service, and sends out the right specialist. The lesson for whoever is tuning is the usual one in this series: before changing specialist, make sure the signature dish really is what is holding you back.

References

vLLM, Attention Backends (selección, FA2/3/4 por arquitectura, MLA): https://docs.vllm.ai/en/latest/design/attention_backends/.
vLLM / DeepWiki, FlashAttention and FlashInfer: https://deepwiki.com/vllm-project/vllm/8.2-flashattention-and-flashinfer.
vLLM, Triton Attention Backend Deep Dive (mar-2026): https://vllm.ai/blog/2026-03-04-vllm-triton-backend-deep-dive.
Z. Ye et al., FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (arXiv 2501.01005): https://arxiv.org/abs/2501.01005.
T. Dao, FlashAttention-2 / FlashAttention-3 (kernel IO-aware, async Hopper): https://github.com/Dao-AILab/flash-attention.

The pantry of pigeonholes: PagedAttention and the vLLM block manager

Mon, 08 Jun 2026 05:20:00 +0200

Part of the under the engine series. The scheduler post ended with a loose end: the scheduler has a second budget, the KV blocks, and when they run out it preempts. This post opens up that budget. It is the piece the KV cache post took for granted, what gets stored, in order to explain how it is managed in memory. And it is the one the FlashAttention post had been promising for months.

TL;DR

The KV cache grows a little with every token generated, and the problem was never its total size but the way it is reserved. The first servers asked, per request, for a contiguous chunk of HBM the size of the maximum possible context. Since almost no request reaches that maximum, the result was catastrophic: 60-80% of the HBM wasted on fragmentation. PagedAttention applies to the KV the oldest and most battle-tested idea in operating systems, paging: split the KV into fixed-size blocks (16 tokens by default), store them in non-contiguous HBM wherever there is room, and keep a block table that translates each sequence’s logical block into its physical one. Waste falls to ~4% (only the last block, half-filled). And since each block can be identified by the hash of its content, two requests that share a prefix point at the same physical block and share memory, with copy-on-write when one of them diverges: that is the engine of prefix caching. This post explains fragmentation with numbers, the block manager, the block table, COW, the block-size trade-off, the 10 knobs and the trap of confusing “fragmentation solved” with “zero waste”. On the generic 4×H100 SXM cluster.

Where you are: the pantry, underneath the scheduler

Go back to the kitchen of the previous post. The front-of-house manager assembles trays, but behind him there is a pantry where the ingredients each table accumulates over its meal are stored, its KV cache. The question in this post is how that pantry is organised.

The naive way: each table gets a whole contiguous shelf, sized for the greediest imaginable customer. The problem is obvious: a table that orders little leaves almost all of its shelf empty, but that shelf is already reserved and nobody else can use it. With many tables, the pantry fills up with half-empty shelves and no new table fits, even though the gaps added together leave plenty of room.

The PagedAttention way: the pantry is divided into small, identical pigeonholes. Each table is given the pigeonholes it needs, one at a time, wherever there is room, and they do not have to be next to each other. A map book records which physical pigeonholes each table has and in what order. When a table leaves, its pigeonholes go back to the pile. There are no half-empty shelves: the only waste is the last pigeonhole of each table, the half-filled one. That is, almost literally, the virtual memory of an operating system applied to the KV cache.

Why contiguous memory fragmented

Reserving contiguously and up front produces three distinct kinds of waste:

Reservation fragmentation. You set aside space for max_model_len (8192 tokens, say) even though the request will use 800. Reserved and never used.
Internal fragmentation. Within what you reserved, everything above what you actually use at any given moment.
External fragmentation. Gaps between contiguous reservations that are too small for a new request, even though added together they would be plenty.

The original vLLM paper measured that earlier systems wasted 60% to 80% of the KV memory through these three routes (Kwon et al., SOSP 2023). That is: on a GPU with room for 100 real requests, only 20-40 fitted. Paging attacks all three at once. It eliminates reservation waste (on-demand allocation) and external waste (blocks need not be contiguous), and leaves only a remainder of the internal kind: the last partial block.

The mechanism: blocks, block table and the kernel gather

The KV of a sequence is chopped into logical blocks of $b$ tokens (by default $b = 16$). Each logical block is mapped, via the block table, to a physical block somewhere in the HBM. The block table is the “map book”: a per-request list of which physical block corresponds to each logical one (vLLM implementation).

The key point is that the attention kernel knows how to read like that. Instead of assuming a contiguous KV tensor, the PagedAttention kernel receives the block table and does a gather: for each sequence it walks its physical blocks in logical order and reads K and V as if they were together. That is why PagedAttention is not just a data structure: it is a kernel that knows how to attend over paged memory. And that is why the attention backend and the block manager are tied together, the second deciding where the KV lives and the first knowing how to read it from there.

The block manager: the pantry librarian

The block manager (in V1, the KVCacheManager) is the one that keeps the map book. Its responsibilities:

Maintain a pool of free physical blocks (a queue of available blocks).
Allocate blocks to a sequence as it grows (a new block every $b$ tokens).
Free the blocks when the sequence finishes or is preempted.
Maintain the block tables (logical→physical) of each request.
Manage prefix caching: detect blocks with identical content and share them.
When free blocks run out, tell the scheduler to preempt (see the scheduler post).

When the block manager says “no blocks left”, the scheduler has to take somebody off the train. That is why the two budgets, tokens and blocks, are the two hands of the same engine.

Here is the elegant part. If two requests start with the same prefix, the same system prompt, the same context document, the first KV blocks of both are identical byte for byte. Why compute and store them twice?

vLLM gives each block a hash summarising its content (the tokens that formed it, plus the hash of the previous block, so that the hash captures position). It keeps a global table of blocks by hash. When a new request produces a block whose hash already exists, it allocates no new memory: it points its block table at the physical block that was already there (automatic prefix caching, vLLM).

Copy-on-write is the safeguard: while A and B share a block, neither can modify it. The moment one of the two needs to write something different into that block (because their sequences diverge, or in parallel sampling / beam search where several branches share a prefix), the block manager copies the block for that branch and only then writes (details, vLLM). It is the same COW that fork() uses in an OS: share until somebody writes.

The saving is direct: if 50 requests share a system prompt of 1,000 tokens, instead of 50 copies of that prefix’s KV there is one. How to maximise that saving in practice is the subject of the prefix cache hit rate post.

The maths that matter: how much KV, how many blocks

KV bytes per token. For a transformer block with $L$ layers, $h_{kv}$ KV heads (GQA), per-head dimension $d$ and $s$ bytes per element (2 in FP16):

$$\text{KV/token} = 2 \cdot L \cdot h_{kv} \cdot d \cdot s$$

For a Llama-70B ($L=80$, $h_{kv}=8$, $d=128$, FP16):

$$\text{KV/token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327680 \text{ bytes} \approx 320 \text{ KB}$$

A block of 16 tokens takes up $16 \times 320\,\text{KB} = 5.12$ MB.

How many requests fit. If after loading the weights ~120 GB of the node’s 320 are left for KV:

$$\text{KV tokens} = \frac{120 \cdot 10^9}{327680} \approx 366000 \text{ tokens} \approx 22900 \text{ blocks}$$

With average contexts of 4,000 tokens (250 blocks each), that is ~90 concurrent requests. That number, not max_num_seqs, is the real concurrency ceiling, and it is exactly the “block budget” of the scheduler.

The waste that remains. PagedAttention does not get to zero: each sequence wastes, on average, half a block (the last one, half-filled). With 16-token blocks and sequences of 4,000, that is $8 / 4000 = 0.2\%$ per sequence, and the famous aggregate “~4%” from the paper includes other overheads. The lesson: the waste does not disappear, it is bounded to the size of one block.

The block-size trade-off

The block_size (16 by default) is a trade-off, not a magic constant:

Block	Advantage	Drawback
Small (8)	less internal waste; finer-grained prefix sharing	more block table entries; more management and gather overhead
Large (32)	fewer metadata; more efficient gather	more waste in the last block; prefix caching shares at a coarser grain (fewer hits)

A large block shares worse: prefix caching can only reuse complete and identical blocks, so with 32-token blocks two prompts that agree on 20 tokens share nothing (they do not fill a common block), whereas with 8-token blocks they share two blocks. The default of 16 is the point vLLM found reasonable for most workloads; it is worth testing if your workload has short, heavily repeated prefixes.

The 10 knobs

#	Knob	What it controls	Cost if you overdo it
1	`block_size`	tokens per block	waste / overhead (see table)
2	`enable_prefix_caching`	sharing blocks by hash	almost none; usually on
3	`gpu_memory_utilization`	how many physical blocks there are	OOM if too high
4	`kv_cache_dtype` (FP8)	bytes per KV element	quality (measure, do not assume)
5	`swap_space`	blocks that fit on the host (SWAP)	PCIe traffic on preemption
6	`max_model_len`	maximum length per request	fewer requests if too high
7	eviction policy	who gets blocks taken away	prefix cache hit rate
8	sliding window	discarding old KV	quality on long contexts
9	TP / KV sharding	splitting the KV across GPUs	NVLink traffic
10	num_gpu_blocks (override)	forcing the block count	OOM or underuse

How it connects to the rest of the stack

With the scheduler. The “block budget” of the scheduler is administered by this block manager. When it says there are no blocks, the scheduler preempts (RECOMPUTE by default).

With the KV cache. The KV cache post explains what each token stores; this one, how it is placed in memory without fragmenting.

With prefix caching. COW and block hashes are the mechanism; hit rate engineering is how to squeeze it (prompt structure, prefix-aware routing).

With KV quantisation. Moving the KV to FP8 halves the bytes/token: the same node holds twice the tokens. It is the most direct lever on concurrency.

With the attention backend. The FlashAttention/FlashInfer kernel has to know how to attend over paged blocks; the block manager decides where they live, the kernel knows how to read them.

With disaggregated serving. Moving a request from a prefill pool to a decode pool in disaggregated serving is, at bottom, transferring its KV blocks between engines, over NVLink or over the network.

With multi-LoRA. In multi-LoRA serving, the base shares prefix KV between requests from different adapters as long as the prefix is identical.

Traps and things that are not what they seem

“PagedAttention eliminates waste.” It bounds it, it does not eliminate it. What remains is the last partial block per sequence (~half a block) plus the block table metadata. It is ~4% instead of 60-80%, but it is not zero. Sizing as if it were zero leaves you without a cushion.

“Bigger blocks always perform better.” The gather is somewhat more efficient, yes, but you lose sharing granularity: prefix caching shares worse and the waste in the last block grows. On workloads with many short repeated prefixes, small blocks can win.

“Prefix caching shares KV between users, that is a privacy problem.” It shares only blocks that are identical token by token (same system prompt, same document). It does not expose one user’s content to another: if the tokens do not match, there is no common block. What is worth watching is information leaking through timing side channels (a hit is faster than a miss), relevant only in very adversarial multi-tenant scenarios.

“FP8 in the KV is free: twice the concurrency.” It does double the tokens that fit, yes, but KV in FP8 degrades quality measurably on long contexts. It is a real lever, not a free lunch: quality has to be measured (FP8 end-to-end), not assumed.

“Going back to contiguous memory would be simpler and almost as good.” That is nostalgia for the contiguous tensor. The “simple” option reintroduces 60-80% fragmentation: on a GPU, that is the difference between 30 and 90 concurrent requests. The complexity of the block table pays for itself many times over.

SWAP versus RECOMPUTE when preempting. Configuring a lot of swap_space “so as not to lose KV” puts gigabyte transfers over PCIe on the critical path. In V1, RECOMPUTE is usually better; swap is for specific cases.

Conclusion

The bottleneck in serving an LLM was never only how much memory you have, but how you hand it out. The first servers treated the KV cache as a contiguous shelf per client and threw two thirds of the HBM in the bin without it showing up on any dashboard. PagedAttention stole from the operating system its best idea of fifty years ago, paging, and applied it at the exact place where it hurt: small pigeonholes, a map book, on-demand allocation and, as a bonus, the possibility that two requests starting the same way share the same pigeonholes until they stop resembling each other. The result is not magic: the waste is still there, but bounded to the size of one block instead of to the size of the worst imaginable case. And that difference, from 70% to 4%, is what turned a GPU serving thirty clients into one serving ninety, without touching the hardware. The pantry did not get bigger; it got better organised.

References

W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): https://arxiv.org/pdf/2309.06180.
vLLM, Automatic Prefix Caching (design, block hashing): https://docs.vllm.ai/en/v0.8.1/design/automatic_prefix_caching.html.
vLLM, Automatic Prefix Caching — Implementation (block table, COW): https://docs.vllm.ai/en/v0.6.1/automatic_prefix_caching/details.html.
H. Elshafie, Paged Attention from First Principles: A View Inside vLLM: https://hamzaelshafie.bearblog.dev/paged-attention-from-first-principles-a-view-inside-vllm/.
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention (alternative, critical context): https://arxiv.org/pdf/2405.04437.

The pass: the maître d' who builds every round — vLLM's scheduler step

Mon, 08 Jun 2026 05:00:00 +0200

Part of the under the engine series. Earlier posts looked at the silicon that runs the kernels (SMs and CUDA graphs) and at loading the weights (from disk to HBM). This one goes up a floor: who decides what runs in each forward. Before the GPU launches a single kernel, someone has had to build this round’s order. That someone is the scheduler, and it is the heart of continuous batching.

TL;DR

An LLM server does not serve one whole request and then the next: it advances every live request at once, a little on each engine iteration. The piece that decides how far each one advances in each step is the scheduler, and its output is deceptively simple: a dictionary {req_id: number of tokens} that the model runner turns into a single forward on the GPU. vLLM V1’s most important decision was to erase the distinction between prefill and decode: for the scheduler, a prompt token and a freshly generated token are the same thing, tokens that need processing, and that is why it can put a 4000-token prompt in the same batch as 200 one-token decodes. There are four pieces: the token budget (max_num_batched_tokens), a fixed-size tray filled every round; chunked prefill, which splits a huge prompt into chunks so it does not hog the tray and blow up everyone else’s latency; the two queues (waiting, in arrival order, and running); and preemption, because when the KV cache runs out somebody has to get off the train. This post explains the loop, the arithmetic of the budget and of concurrency, the 10 knobs and the headline trap: raising the budget improves throughput but worsens ITL (inter-token latency), and almost nobody measures both at once. On the generic 4×H100 SXM cluster.

Where you are: the pass, before the kitchen starts

Picture the pass of a restaurant with a busy dining room. There is no one cook per customer; there is a shared kitchen and a maître d’ who, every few seconds, looks at all the open orders and builds a tray to send to the stoves. That tray holds, say, 8000 “units of work”. The maître d’ decides what goes in: tables already eating (requests in decode) get one more dish each; new tables that have just ordered (requests in prefill, with their whole prompt still to process) get as much of their order as fits in whatever is left of the tray. The tray goes out, the kitchen executes it in one go, and it starts again. Hundreds of times per second.

That maître d’ is the scheduler. The kitchen is the GPU running a forward pass. And the house rule is that the kitchen never stops to wait for a single table: if a new order is enormous (a 30,000-token prompt), it is not sent whole in one go blocking everyone else, it is sent in chunks. That is, in one sentence, the whole mechanics of vLLM’s scheduler.

The engine loop: one dictionary per iteration

The inference engine is a very short loop. On each pass:

The scheduler looks at the queues and produces a decision.
The model runner runs a forward with that batch on the GPU.
The sampler draws one new token for each active sequence.
State is updated (KV cache, positions, finished requests) and it goes back to 1.

The surprising part is the shape of the decision in step 1. In vLLM V1 it is not a complex structure with phases: it is literally a dictionary

$$\text{schedule} = \{\, \text{req id} \rightarrow n_\text{tokens} \,\}$$

that says, for each request entering this round, how many tokens of it get processed. For a request that is generating text, n_tokens = 1 (one autoregressive step). For a new request, n_tokens can be the entire length of its prompt. And it can be anything in between, a chunk of prompt, under chunked prefill, prefix caching or speculative decoding (vLLM V1 docs).

That the decision fits in a {id: number} dictionary is not a neat implementation detail: it is why continuous batching works. Because the scheduler does not think in “phases” but in “how many tokens for each”, it can mix requests at any point of their life in the same forward. The GPU receives a single heterogeneous token tensor and processes it in one go.

The death of the prefill/decode distinction

This is the idea that costs the most effort and matters the most. In the first LLM server architectures, a request lived in two separate phases: first prefill (process the whole prompt and fill the KV cache), then decode (generate token by token). The scheduler had to choreograph the move from one phase to the other, and mixing them was hard.

vLLM V1 removed the distinction (V1 design). The scheduler treats prompt tokens and generated tokens uniformly: they are all tokens the model has to process in a forward. The practical consequence is enormous. A 4000-token prompt and a sequence that has generated 800 tokens and needs one more are, to the scheduler, “4000 tokens of request A” and “1 token of request B”. They fit together on the same tray. There is no phase choreography, just a budget to hand out.

This unlocks the pattern that really pays: mixing prefill and decode in every step. Prefill is compute-bound work (a lot of matrix to multiply); decode is memory-bound (little compute, a lot of KV to move). Mixing them in the same batch fills the gaps: while the GPU is busy with the heavy prefill, it advances the light decodes “for free”. It is the same efficiency principle as continuous batching, taken to its cleanest form.

The token budget: the fixed-size tray

The tray has a size: max_num_batched_tokens. It is the maximum number of tokens the scheduler can put in a single step (vLLM optimisation). The policy, with chunked prefill on (it always is in V1), is clear:

Decodes first. Room is reserved for one token per request in the running queue. They are cheap, and they are diners already eating: they are not left waiting.
What is left goes to prefills. With the remaining budget, prompt tokens from waiting requests are added, in arrival order (FCFS), split into chunks if needed.

An example with numbers. Budget max_num_batched_tokens = 8192, and at this instant there are 200 requests generating:

$$\text{decode} = 200 \times 1 = 200 \text{ tokens}$$ $$\text{free budget} = 8192 - 200 = 7992 \text{ tokens for prefill}$$

If a new request arrives with a 4000-token prompt, it fits whole in this step (4000 < 7992) and 3992 are left for another. If one arrives with 30,000 tokens, it does not fit: the scheduler sends it a 7992-token chunk this step, and the remaining 22,008 in later steps. That is chunked prefill.

The budget matters because it fixes how many forwards are needed to swallow a prompt. A 30,000-token prompt with an 8192 tray takes 4 steps in prefill alone before it emits its first token. With a 2048 tray it takes 15 steps, but each of those steps leaves more room for other people’s decodes, so the other diners feel the jam less.

The two queues and preemption: when someone gets off the train

The scheduler handles two queues. The waiting queue holds requests that have not started yet (their prompt has not been processed), served in arrival order, FCFS by default, though there is a priority policy. The running queue holds those already alive and generating (vLLM scheduling).

There is a second budget, harder than the token one: the KV cache. Every live token occupies KV blocks in HBM, and they are finite (gpu_memory_utilization fixes them). When the scheduler wants to advance the running requests but there are no free blocks for the next token’s KV, someone has to get off: that is preemption.

vLLM V1 preempts by RECOMPUTE by default, not by SWAP (V1 guide). The difference:

SWAP: copy the victim’s KV to host RAM and bring it back later. It moves gigabytes across PCIe (see the PCIe and P2P post).
RECOMPUTE: throw the victim’s KV away and, when there is room again, redo its prefill from scratch. It sounds expensive, but in the V1 architecture it works out cheaper than swapping, because prefill is work the GPU does very fast and you save the round trip across the bus.

The victim is usually the newest request in the running queue (so as not to penalise whoever has been waiting longest for their answer). The danger is thrashing: admit too many requests at once and the system enters a preempt-recompute-preempt cycle that drives throughput into the floor. That is why the second ceiling exists.

The arithmetic that matters: concurrency and the budget trade-off

How many requests fit at once. The real concurrency limit is not max_num_seqs (the nominal ceiling on simultaneous sequences); it is usually the KV cache. If a node has $B$ free KV blocks, each block holds $b$ tokens (16 by default), and each request occupies $L$ context tokens on average, the maximum sustainable concurrency is:

$$N_\text{max} \approx \frac{B \cdot b}{L}$$

Take a 70B model in FP16 on 4×H100 SXM (320 GB), with the bulk of HBM in weights and, say, ~120 GB free for KV. With a KV of ~0.3 MB/token (an order-of-magnitude figure, it depends on layers and heads), that is ~400,000 KV tokens. With average contexts of 4000 tokens:

$$N_\text{max} \approx \frac{400000}{4000} = 100 \text{ concurrent requests}$$

Raising max_num_seqs to 400 does not give you 400 concurrent: it gives you preemption and thrashing as soon as contexts grow. The KV rules.

The budget trade-off. Raising max_num_batched_tokens puts more work in each forward, so fewer forwards for the same total work, and more throughput. But a large budget lets a huge prefill occupy almost the whole tray in one step, and that step takes longer, so everyone else’s decodes wait for that entire step, and ITL (inter-token latency) rises for everybody. The rule of thumb (vLLM optimisation):

Budget	Effect	At the cost of
Low (e.g. 2048)	more interleaving, stable ITL	less peak throughput
High (e.g. 16384)	maximum throughput	ITL spikes when a large prefill arrives

There is no “correct” value: there is a point for your load. And you only find it by measuring throughput and ITL at the same time, which is exactly what almost nobody does.

The scheduler’s 10 knobs

#	Knob	What it controls	Cost if you overdo it
1	`max_num_batched_tokens`	tray size per step	high ITL if too large
2	`max_num_seqs`	nominal concurrency ceiling	preemption if the KV falls short
3	`gpu_memory_utilization`	KV blocks available	OOM if too high
4	chunked prefill (threshold)	prompt chunk size	chunking overhead if too fine
5	policy (`fcfs`/`priority`)	who gets served first	starvation of low priority
6	preemption mode	RECOMPUTE vs SWAP	PCIe traffic / recompute
7	`enable_prefix_caching`	reuse prefix KV	little; almost always on
8	`max_model_len`	maximum context per request	pessimistic KV reservation
9	CUDA graph sizes	align batch with buckets	padding / capture (see below)
10	speculative tokens	extra tokens per step	wasted work if acceptance drops

How it connects to the rest of the stack

With continuous batching. The scheduler is continuous batching turned into code. The batching post explains the what (advance every request at once); this one explains the how (the per-step token dictionary).

With the KV cache and the block manager. The second budget, the blocks, is managed by the PagedAttention block manager. The scheduler asks for blocks; if there are none, it preempts. The two pieces are coupled through memory.

With CUDA graphs. CUDA graphs are captured for specific batch sizes (buckets). The scheduler should produce batches whose size lands in those buckets to avoid padding; otherwise part of the graph’s benefit is lost.

With chunked prefill and the prefix cache. Chunking a prompt interacts with prefix caching: chunks that match an already cached prefix skip the compute, and the scheduler reflects that by lowering that request’s n_tokens.

With speculative decoding. Speculative decoding makes a step verify several tokens at once; the scheduler models it as n_tokens > 1 for a request in decode.

With disaggregated serving. In disaggregated prefill/decode there are two schedulers, one per pool, each with its own budget; the phase distinction that V1 erased inside an engine reappears between engines.

With autoscaling. The metrics that drive autoscaling with KEDA, waiting-queue length and preempted requests, come straight out of the scheduler’s state.

Traps and things that are not what they look like

“Raising max_num_batched_tokens always helps.” It improves throughput and worsens ITL. If you only look at tokens/s in a large-batch benchmark, you “confirm” that more is better; in interactive production, your users feel the stutter. Measure both metrics or you are not measuring.

“The engine does all the prefills first and then the decodes.” That is the old architecture’s intuition. In V1 there are no phases: each step mixes prefill and decode according to the budget. Reasoning with the phase model leads to wrong conclusions about why latency rises.

“More max_num_seqs = more throughput.” Only until the KV cache runs out. Past that, more nominal concurrency produces preemption, and cascading preemption (thrashing) lowers throughput. The real ceiling is the KV, not the parameter.

“RECOMPUTE is a waste, SWAP is better.” In V1, RECOMPUTE usually wins: prefill is extremely fast on GPU and SWAP puts a gigabyte-scale round trip across PCIe on the critical path. Switching to SWAP “to avoid recomputing” can make latency worse.

“The scheduler is the bottleneck.” Almost never. The decision is a dictionary built in microseconds; the cost of the round is the forward on the GPU, three orders of magnitude above. If your scheduling CPU shows up in the profiler, the problem is usually host-thread jitter (see NUMA and CPU isolation), not the scheduler logic.

Chunked prefill that is too fine. Tiny chunks make a large prompt take many steps and add fixed per-step overhead. Chunking is there to bound the ITL impact, not to pulverise the prompt.

Conclusion

All the power of a modern LLM server, swallowing hundreds of requests at once without any of them blocking the rest, rests on a decision that fits in a {request: how many tokens} dictionary, taken hundreds of times per second. The idea that made it possible was not a faster kernel or a bigger GPU: it was to stop thinking in phases. When a prompt token and a generated token are the same thing, the scheduler can fill every tray by mixing the heavy and the light, and the kitchen never stops. The rest is two budgets, tokens and KV blocks, and a rule for when the second runs out. The maître d’ does not cook; he only decides what goes to the stoves in each round. But that decision, repeated without pause, is what separates an idle GPU waiting for orders from a kitchen running flat out. And the uncomfortable lesson for whoever is tuning: throughput and latency meet in the budget, and optimising one blindly means worsening the other without noticing.

References

vLLM, vLLM V1: A Major Upgrade to vLLM’s Core Architecture: https://openlm.ai/vllm-v1/.
vLLM, vLLM V1 User Guide (chunked prefill por defecto, preemption RECOMPUTE): https://docs.vllm.ai/en/v0.9.2/usage/v1_guide.html.
vLLM, Optimization and Tuning (max_num_batched_tokens, presupuesto y trade-off): https://docs.vllm.ai/en/stable/configuration/optimization/.
A. Wong, Understanding vLLM Scheduling: Token Budgets, Chunked Prefill, and Policies: https://audreywongkg.medium.com/understanding-vllm-scheduling-token-budgets-chunked-prefill-and-policies-2c879e3980e3.
W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): https://arxiv.org/pdf/2309.06180.

The chef who calls out every order: SMs, CUDA streams and CUDA graphs, or why the GPU gets bored generating tokens

Sun, 07 Jun 2026 09:00:00 +0200

This closes the “outside the API” pair. The previous post moved the weights from disk into HBM; here we look at what happens once they are inside, in the silicon that executes them. It is the floor below the kernel launch that the NUMA post mentioned without opening: who launches those kernels, how, and why in decode the GPU spends more time waiting for orders than computing.

TL;DR

An H100 has ~132 streaming multiprocessors (SMs), the “hobs” that execute the compute, and occupancy measures how many warps (groups of 32 threads) it has active to hide latency. But the decode bottleneck is rarely the power of those SMs. Each decode step launches hundreds of tiny kernels (several projections per layer × ~80 layers), and each kernel launch costs 5-10 µs of serial CPU. Since decode kernels are small (small batch, a single token), the GPU finishes them before the CPU calls out the next one: bubbles appear and the GPU sits bored waiting for orders. That regime is called launch-bound, and it is the deep reason, not power, not memory, why --enforce-eager delivers 54 tok/s where with optimisations you reach 89-140. The solution is CUDA graphs: record the entire kernel sequence once and replay it as a single submission, removing the per-launch overhead (~28% of the latency per iteration). vLLM captures ~102 graphs at startup and pads the real batch to the nearest bucket so it can replay a graph with fixed shapes. This post explains SM, occupancy, streams, launch overhead with the maths, CUDA graphs, the 10 knobs, and the trap that this capture is the second half of the cold start from the previous post. With scepticism about what actually moves the needle. On the generic 4×H100 SXM cluster.

Where you are: the silicon, below the kernel launch

The analogy: the chef who calls out every order

Last scene in the restaurant of the series. The kitchen is set up, the pantry is stocked (the previous post). Now the food has to be plated. The hobs are the SMs: 132 stations cooking in parallel. The head chef is the CPU: he calls out the orders, each kernel launch is a shout of “one matrix multiplication, coming up!”. The cooks (the SMs) execute what the chef calls out.

In prefill, processing the whole prompt, each order is a huge dish: a giant matmul over hundreds of tokens at once. The chef calls out an order and the hobs take a good while to produce it. The chef has plenty of time to call out the next one. The hobs are flat out: compute-bound.

In decode, generating one token at a time, each order is minuscule: a matmul over a single token. The hob finishes it in an instant… and sits staring at the chef waiting for the next one. But the chef can only call out one order every 5-10 µs, and there are hundreds of orders per token. The hobs, lightning fast, get bored between shouts. The restaurant is not slow because the cooks are bad: it is slow because the chef does not call out fast enough. That is the launch-bound regime.

The solution is not more hobs or faster cooks. It is to stop calling out order by order. If the chef prints the whole night’s sequence on a single sheet and hands it to the line (“do this, in this order, without waiting for me”), the hobs run without pauses. That is a CUDA graph: record the kernel sequence once and replay it in one go, without the CPU calling out each one. And --enforce-eager is exactly the opposite: forcing the chef to call out order by order, all night long.

The mechanism: SM, warps and occupancy

An H100 SXM has ~132 SMs. Each SM executes threads in groups of 32 called warps, and can have several warps “in flight” at once. Occupancy is the fraction of active warps against the maximum the SM supports. What is the point of having many active warps? To hide latency: while one warp waits for data from HBM (hundreds of cycles), the SM executes another ready warp. With few warps, the SM runs out of anyone to give a turn to and stalls.

But, and this is key, occupancy is a necessary, not sufficient condition, and it only matters if the SM has work to do. In decode, the typical problem is not low occupancy inside a kernel: it is that between kernels the SM has nothing, because the CPU has not launched the next one yet. Raising the occupancy of a kernel that lasts 8 µs does not help if the GPU spends 6 µs waiting for it to be launched.

The mechanism: streams, the order queue

A CUDA stream is a queue of operations that the GPU executes in order. Operations in the same stream are sequential; operations in different streams can overlap. That is what allows, for instance, copying data H2D in one stream while another stream computes, the compute/copy overlap. vLLM uses streams to overlap work, but a stream on its own does not remove the cost of launching each kernel: it only decides ordering and parallelism. The launch cost is still there, order by order, until the graphs come in.

The maths that matters: when the GPU is left waiting

The number that governs everything: a kernel launch costs 5-10 µs of CPU, serially. Take a Llama-70B with ~80 layers. Each layer, without fusion, launches on the order of ~10 kernels (Q/K/V projections, attention, output projection, the two or three MLP matmuls, the normalisations, RoPE and so on). That gives:

$$ N_{\text{kernels}} \approx 80 \text{ layers} \times 10 \approx 800 \text{ launches per token} $$

At 5 µs per launch, serially:

$$ T_{\text{launch}} \approx 800 \times 5\,\mu s = 4{.}0 \text{ ms per token} $$

Those 4 ms are CPU calling out orders only, not counting how long the SMs take to cook. If the GPU could compute instantaneously, the launch ceiling would be ~250 tok/s, and with synchronisation points between kernels, worse. Now compare with the memory ceiling of decode: each token reads the 140 GB of weights once from HBM:

$$ T_{\text{mem}} = \frac{140 \text{ GB}}{3{.}35 \text{ TB/s}} \approx 42 \text{ ms} \;\Rightarrow\; \approx 24 \text{ tok/s (one sequence, no batch)} $$

Here is the subtlety almost nobody has in mind. For a single sequence, decode is memory-bound at ~24 tok/s, and the 4 ms of launch fit inside the 42 ms of reading: the launch hides. But batching changes everything. When serving a batch of B sequences, the weights are read once and serve all B, so the memory cost per token amortises and falls. The GPU stops being memory-bound… and what was underneath emerges: the launch cost, which does not amortise with batch because you have to launch the same kernel sequence regardless. The result: the better you batch, the more launch-bound you become, and the more CUDA graphs pay off. That is why the raw measurement confirms it, --enforce-eager gives 54 tok/s where graphs give 89, and up to 8× in configurations where decode is very small and launch dominates completely.

vLLM’s CUDA graphs, specifically

vLLM does not capture a single graph: it captures ~102 at startup, on the order of 51 piecewise (for the mixed prefill+decode steps) and 51 full (for pure decode). Each one is recorded for a fixed batch size (a bucket: 1, 2, 4, 8… up to a maximum). In service the real batch almost never falls exactly on a bucket, so vLLM pads with zeros up to the next bucket up, replays that graph, and trims the output to the real size. That is the price of graphs: they need static shapes, and padding is what makes them static.

This has two consequences that show up in the knobs and the traps:

Capture costs time and memory. Recording 102 graphs at startup adds seconds to the cold start, the second half of the startup that the previous post left pending, and consumes HBM (each graph retains its buffers). The FULL_AND_PIECEWISE mode (the default) is the fastest in service but the one that asks for the most memory and the longest capture; FULL_DECODE_ONLY saves both in exchange for not accelerating the mixed steps.

Padding wastes some compute. Padding a batch of 33 up to the bucket of 64 computes 31 phantom sequences. It is a small cost against what removing the launch overhead saves, but it exists, and it grows if the buckets are badly chosen.

The 10 knobs worth touching

Knob 1 — Measure whether decode is launch-bound

Before touching anything: is the GPU computing or waiting? With nsys (Nsight Systems) you can see the gaps between kernels on the timeline; if there are gaps in decode, it is launch-bound and graphs will help. If the GPU is at 100% with no gaps, the bottleneck is something else (memory or compute) and graphs will not work miracles. nvidia-smi dmon showing low utilisation in decode but poor TPS is the cheap signal.

Knob 2 — Do not use `--enforce-eager` in production

--enforce-eager disables CUDA graphs. It is a debugging tool (to isolate which kernel fails), not a production one. Leaving it on “because it used to start up” throws away 26-50% of decode throughput. If it is in your production command, take it out and measure.

Knob 3 — Capture buckets (`cudagraph_capture_sizes`)

Which batch sizes to capture. Buckets too far apart make padding expensive; too many of them make capture slow and eat HBM. Tuning them to the real distribution of batch sizes you see in production is the fine adjustment, but only after measuring that distribution.

Knob 4 — CUDA graph mode

FULL_AND_PIECEWISE (default, fastest, more memory/capture), FULL_DECODE_ONLY (saves memory and capture, ideal for pure decode pods in disaggregated serving), PIECEWISE, or NONE (= eager). The right mode depends on whether the pod does pure or mixed decode.

Knob 5 — `torch.compile`

vLLM leans on torch.compile to fuse and optimise kernels before capturing them in graphs. Fewer kernels (fusion) = fewer launches = less dependence on the graph and better decode even in eager. The compilation level is a knob, with its startup time cost.

Knob 6 — Batch size: fill the hobs

Memory-bound decode amortises with batching (as we saw in continuous batching): read the weights once for B sequences. More batch = more SM occupancy and more memory amortisation. The limit is set by the HBM available for the KV cache. It is the knob that moves aggregate throughput most.

Knob 7 — Do not break stream overlap

vLLM overlaps compute and copy with streams. Patching the code to “simplify” can serialise what was overlapped. If you do not know why there are several streams, do not collapse them.

Knob 8 — Persistence mode + locked clocks

nvidia-smi -pm 1 keeps the driver resident (avoiding reinitialisations that add launch latency). Locking clocks to the boost frequency stops the GPU dropping P-state between tiny decode kernels and paying the ramp-up latency. It is the same anti-jitter spirit as the NUMA post, applied to the GPU.

Knob 9 — Fused kernels (FlashAttention, FP8 kernels)

Fewer kernels = fewer orders to call out. FlashAttention fuses attention into one kernel instead of several; fused FP8 kernels reduce the count. Fusion attacks the problem at the root: it does not speed up the launch, it removes launches.

Knob 10 — Accept the capture cost in the cold start

Graph capture adds seconds to startup. In a pod that lives for hours, it amortises easily. In one that scales up and down every minute, that cost is paid over and over; there, FULL_DECODE_ONLY (shorter capture) or accepting slightly less throughput can be worth it. It is the same warm-vs-elastic tension as in the cold start.

Summary table

#	Knob	What it attacks	Risk / cost
1	`nsys` / `dmon`	knowing whether it is launch-bound	none; do it first
2	remove `--enforce-eager`	graphs disabled	it was for debugging; reactivate if a bug returns
3	capture buckets	expensive padding / slow capture	requires measuring the real distribution
4	graph mode	memory and capture	less coverage on mixed steps
5	`torch.compile`	unfused kernels	startup time
6	batch size	occupancy + memory	HBM for KV cache
7	streams	broken overlap	do not touch if not understood
8	persistence + clocks	jitter / P-states	electrical consumption
9	fused kernels	number of launches	kernel compatibility
10	capture vs cold start	slower startup	less throughput if trimmed

How this connects with the rest of the stack

With the cold start. CUDA graph capture is the second half of the startup that the previous post opened: loading weights + capturing graphs = the complete cold start.

With continuous batching. Continuous batching is what makes decode launch-bound (it amortises memory and leaves the launch exposed), and that is why graphs and batching reinforce each other.

With the KV cache. The KV cache decides how much batch fits in HBM, and the batch decides SM occupancy and how much launch overhead matters. Everything is coupled through memory.

With the interconnect. In TP, between the compute kernels there are all-reduces (NVLink/NCCL) that also get launched and synchronised. vLLM’s custom all-reduce is integrated into the same graph so the sequence is not broken by a CPU synchronisation.

With NUMA. Who launches the kernels is the CPU from the host post; if that thread suffers jitter or lands on the wrong socket, launch overhead gets worse. Graphs reduce the dependence on that thread, which is another reason they help.

With disaggregated serving. The pure decode pods of disaggregated serving are the ideal case for FULL_DECODE_ONLY: they maximise the graph benefit exactly in the most launch-bound phase.

Traps and things that are not what they look like

“Raising occupancy will fix slow decode.” Not if the problem is launch-bound. Occupancy matters inside a kernel that has work; if the GPU is idle between kernels waiting for the CPU, more occupancy does not touch that bubble. Measure before optimising what is not the bottleneck.

“CUDA graphs always speed things up.” They speed things up when decode is launch-bound. If the GPU is already at 100% (compute-bound in prefill, or memory saturated with a huge batch), graphs add little. Their territory is decode with small kernels.

"--enforce-eager gives more stable results." It gives slower results. The stability it appears to give is that it avoids graph capture bugs on new hardware (for example a newly supported architecture). It is a temporary patch, not a production configuration.

Capturing too many buckets “just in case”. Each bucket adds capture time and HBM. Capturing 30 sizes when in production you only see 4 means paying cold start and memory for graphs that are never replayed. Tune to the real distribution.

Confusing utilisation with efficiency. nvidia-smi at 100% “utilisation” only says there is a kernel running, not that the SM is full of useful work. A low-occupancy kernel keeps “utilisation” high while wasting the SM. nvidia-smi utilisation is a coarse thermometer; to know whether the silicon is performing you need nsys/DCGM and to look at real occupancy and gaps.

Optimising the silicon before memory. If decode is limited by HBM bandwidth (large batch, large model), fighting with graphs and occupancy is polishing what is not the bottleneck. The right order: measure the regime (memory / compute / launch) and attack the one in charge.

Conclusion

Intuition says that a GPU generating slow tokens is “working hard”. Almost never: in decode it is waiting for orders. The 132 SMs cook a tiny token in an instant and then sit staring at the CPU, which can only call out one order every 5-10 µs and has hundreds to call out per token. That bottleneck, neither power nor memory but launch, is invisible on any dashboard that looks at “GPU utilisation”, and it is the real reason why --enforce-eager performs at half speed. CUDA graphs solve it with a simple idea: stop calling out order by order and hand over the whole sheet for the night, so the silicon runs without pauses. And there is an uncomfortable truth that reorders the optimisation priority: the better you batch, the more launch-bound you become, because batching kills the memory bottleneck and exposes the launch one. That is why graphs and batching are not separate optimisations: they are the same lever seen from two sides. The chef who learns not to call out every dish is what finally makes the kitchen run as fast as the hobs always could.

References

vLLM, CUDA Graphs (diseño, modos FULL/PIECEWISE, captura): https://docs.vllm.ai/en/stable/design/cuda_graphs/.
vLLM, Inside vLLM: Anatomy of a High-Throughput LLM Inference System: https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html.
NVIDIA, Getting Started with CUDA Graphs: https://developer.nvidia.com/blog/cuda-graphs/.
NVIDIA, Achieved Occupancy (ocupación de SM): https://archive.docs.nvidia.com/gameworks/content/developertools/desktop/analysis/report/cudaexperiments/kernellevel/achievedoccupancy.htm.
PyTorch, torch.compile y CUDA Graphs para inferencia LLM: https://docs.vllm.ai/en/stable/design/cuda_graphs/.
Understanding the Overheads of Launching CUDA Kernels (ICPP 2019): https://www.hpcs.cs.tsukuba.ac.jp/icpp2019/data/posters/Poster17-abst.pdf.

The pantry lift: from disk to HBM, or why the kitchen opens late

Sun, 07 Jun 2026 08:30:00 +0200

This is a trip down to the basement. The under the engine series optimised the hot path, what happens with every token once you are already serving. This post looks at the journey that comes before serving: how the weights climb from disk to HBM. It is the first of a pair on the two things that happen outside the API and that almost nobody times: model loading (this one) and execution on the silicon (the next one).

TL;DR

Before an inference pod generates its first token, it has to push the whole model into HBM. A Llama-70B in FP16 is 140 GB travelling along a path nobody draws: disk → page cache → host buffer → PCIe → HBM. Intuition fails here: HBM is not the bottleneck, it moves 3.35 TB/s and swallows 140 GB in 42 ms; the bottleneck is the supply chain. The Gen5 NVMe disk reads at ~14 GB/s (10 s for 140 GB); PCIe Gen5 copies host→GPU at ~50 GB/s (2.8 s); and the default safetensors loader, which deserialises tensor by tensor and bounces every byte through a CPU buffer, inflates all of that up to 30-60 s. That time is the cold start, and it is the hidden tax paid by autoscaling (scale-from-zero), by canary/blue-green and by disaggregated serving every time a pod is born. There are three families of solution, GPUDirect Storage (direct DMA disk→HBM, no bounce through the CPU), fastsafetensors (4.8-7.5× over the default loader) and the Run:ai Model Streamer (concurrent reads that saturate the disk), plus the simplest lever of all: move fewer bytes (FP8 is half of FP16). This post explains the path, the maths, the 10 knobs, and the cruellest trap: “the second time it loaded fast” is not your loader being good, it is the page cache lying to you. On the generic 4×H100 SXM cluster.

Where you are: the basement, before opening

The analogy: the pantry lift

Let us stay in the restaurant of the series. The shared table was the NVSwitch, the floor next door was NUMA, the maître was the kubelet. All of that describes the restaurant running, with diners seated. But there is a moment no post looked at: before opening, somebody has to bring the entire pantry up from the basement store to the kitchen.

The model weights are the ingredients. They live in the basement store (the disk). The kitchen, the hot line where tokens are plated, is the GPU’s HBM. And between one and the other there is a service lift: the path disk → host → PCIe → HBM. The kitchen cannot serve the first dish until the pantry is upstairs and in place. That restocking time is the cold start.

The intuition trap: the kitchen (HBM) is enormous and blazingly fast, it places ingredients at 3.35 TB/s. So we blame the kitchen when the restaurant opens late. But the kitchen is standing idle waiting for the lift. The bottleneck is never the hot line: it is the lift and the store. And worse still, there is a porter (the default loader) who, instead of carrying whole crates, takes the ingredients out one by one, writes each one down in a notebook and packs them again before sending them up. That porter, not the lift, is half the problem.

The mechanism: what really happens when a model loads

When vLLM starts up with a model in safetensors, the 140 GB of the Llama-70B FP16 make this journey:

Disk → page cache. The kernel reads the .safetensors files from NVMe into the page cache (host RAM). If this is the first time after a reboot, it is a physical read from disk (~14 GB/s on Gen5). If the files are already in page cache from an earlier start-up, this is almost free, and here is where the trap we will see is born.
Deserialise. The default Hugging Face loader mmaps the file and builds the tensors one by one, copying them into a CPU tensor before moving them. This is single-threaded CPU work that saturates neither the disk nor PCIe: most of the “slow” loading time goes here, not on moving bytes.
Host → HBM (H2D). Each tensor is copied from the host buffer into HBM over PCIe Gen5 x16 (~50 GB/s in practice). For the DMA to be efficient, the host buffer should be pinned, which connects directly with the hugepages and pinned memory of the NUMA post.
Place in HBM. HBM receives the 140 GB. At 3.35 TB/s, this takes 42 ms. It is never the bottleneck.

The path has a shortcut: GPUDirect Storage (GDS). Instead of bouncing through the CPU buffer (steps 2-3), a DMA engine close to the NVMe controller writes straight from disk to HBM, without involving the CPU. It is the same principle as the GPUDirect RDMA of the network: get the CPU out of the way. fastsafetensors uses GDS and reaches 26.4 GB/s reading a Llama-70B from NVMe across 4 GPUs.

Why the problem exists: the economics of bytes

The size of the model in bytes is decided by quantisation, and that sets the floor of the cold start, because all those bytes have to move before the first token. For a 70B parameter model:

Format	Bytes/param	Size 70B	Read 1 NVMe @14 GB/s	H2D PCIe @50 GB/s	HBM @3.35 TB/s
FP16 / BF16	2	140 GB	10.0 s	2.8 s	42 ms
FP8	1	70 GB	5.0 s	1.4 s	21 ms
INT4 (GPTQ/AWQ)	0.5	~35 GB	2.5 s	0.7 s	10 ms

Three readings of this table:

HBM never shows up as a problem. The last column is always milliseconds. Anyone saying “the GPU is slow to load” is blaming the wrong place.

Quantising is the most underrated cold start lever. Going from FP16 to FP8 does not only double inference throughput (less HBM bandwidth per token, as we saw in quantization): it also halves the cold start, because there are half as many bytes to push up. It is a two-for-one that sizing usually ignores.

The disk is the bottleneck on bytes; the loader is the bottleneck on time. The figures in the table are the theoretical floor, just moving bytes. The default loader adds single-threaded deserialisation on top, which is the difference between the theoretical 10 s and the real 30-60 s. That is why the solutions attack on two fronts: fewer bytes (quantisation) and a better porter (concurrent loaders / GDS).

The maths that matter: cold start as a tax on autoscaling

Cold start is not paid once. It is paid every time a pod is born. And on an elastic platform, pods are born continuously.

Take an autoscaling setup with KEDA that scales from 2 to 6 replicas when the queue grows. The 4 new replicas take this long to be ready:

$$ T_{\text{ready}} = T_{\text{schedule}} + T_{\text{pull-image}} + T_{\text{load-weights}} + T_{\text{cuda-graphs}} $$

With a Llama-70B FP16 and the default loader, $T_{\text{load-weights}}$ dominates: it can be 40 s of the ~60 s total. During those 40 s, the queue that triggered the autoscaling keeps growing, since the new replicas absorb no traffic until they have loaded. The real number in the formula is not “how many replicas”, it is how long each one takes to start serving, and that number is written by the loading path.

This has a harsh operational consequence: scale-to-zero is unworkable for workloads with a latency SLO if the cold start is 40 s. Nobody waits 40 s for the first token. The real elasticity of an inference platform is not limited by GPU availability, it is limited by how long that GPU takes to have the model inside it. Bringing the cold start down from 40 s to 8 s (with streamer + FP8) is what turns “theoretical scale-to-zero” into “usable scale-to-zero”.

The published numbers give the order of magnitude of the improvement: fastsafetensors cuts start-up from 12.39 s to 4.74 s on a Llama-2-13B over 4×L40S, and from 16.04 s to 6.88 s on 1×A100, 4.8-7.5× over the default deserialiser. The Run:ai Model Streamer loads in 4.88 s from S3 at concurrency 32 and 7.53 s from SSD IO2 at concurrency 8; integrated into vLLM, total time to ready drops to ~23 s from S3. There is no magic here: it is getting the CPU out of the loop (GDS) and reading in parallel (concurrency) to saturate the disk instead of leaving it half idle while one thread deserialises.

The 10 knobs worth touching

Knob 1 — Measure where the time goes (read vs deserialise vs H2D)

Before touching anything: time it. Is the time going on reading from disk, on deserialising, or on the H2D? iostat -x 1 during the load tells you whether the NVMe is saturated (disk bottleneck) or nearly idle (loader/CPU bottleneck). If the disk is at 20%, the problem is not the disk: it is the porter. Changing disk would fix nothing; changing loader would.

Knob 2 — `--load-format`: choosing the porter

vLLM exposes several loaders via --load-format: safetensors (default), runai_streamer, fastsafetensors, tensorizer. The default is the slowest. Changing one flag can be the cheapest 4-7× there is.

# Run:ai Model Streamer (concurrent reads, saturates the disk)
vllm serve meta-llama/Llama-3.1-70B --load-format runai_streamer
# fastsafetensors (GPUDirect Storage, direct DMA disk→HBM)
vllm serve meta-llama/Llama-3.1-70B --load-format fastsafetensors

Knob 3 — Streamer concurrency

The Run:ai Model Streamer splits the read across N threads according to the size of each tensor in order to saturate the storage bandwidth. Concurrency is the key parameter: 16 is usually enough for local NVMe; 32 (sometimes 64) for high-throughput network storage. One thread does not saturate a Gen5 NVMe; 32 do.

vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

Knob 4 — GPUDirect Storage (GDS) + fastsafetensors

If there is an nvidia-fs driver, a supported filesystem and local NVMe, GDS writes straight disk→HBM with no bounce through the CPU. It is the difference between the two paths in the diagram. But: it requires the stack to be in place, and it only wins if the bottleneck is the CPU bounce, not the disk itself. Check with gdscheck.

Knob 5 — Local NVMe for the weights, not network

Serving the weights from Ceph RGW / NFS is convenient (one shared place) but it puts the network into the loading path. For cold start, weights on the node’s local NVMe (or a local cache). Network storage is for the model repository; the inference node should have a hot local copy.

Knob 6 — Pre-pull / local model cache

An initContainer that downloads the model onto a local or hostPath NVMe volume before starting vLLM turns a cold start “from the network” into one “from local NVMe”. Combined with a per-node cache DaemonSet, new pods on an already warm node read from local disk, not from the network.

Knob 7 — Quantisation: move fewer bytes

Weights already in FP8 or INT4 on disk = half or a quarter of the cold start. It is the knob from the table above. And it composes with all the others: FP8 + streamer + GDS is multiplicative.

Knob 8 — Parallel loading across GPUs (vLLM V1)

vLLM’s V1 engine (the default since 0.19) loads the weight shards in parallel across the GPUs of a TP group, instead of sequentially. At TP=4, each GPU loads its quarter at the same time. Check it is active; in old versions loading was serial and the cold start of TP=4 was almost 4× that of TP=1.

Knob 9 — NUMA locality of the NVMe

The NVMe hangs off a PCIe root under a specific socket, exactly the same NUMA map as the host post. If the load’s host buffer lands on the wrong socket, the H2D crosses the UPI. The fifth list to align, alongside isolcpus, reserved-cpus and the NIC IRQs: which socket is local to the NVMe and to the destination GPU. nvidia-smi topo -m shows it.

Knob 10 — Do not pay the cold start: keep pods warm

Sometimes the answer is not to load faster, but not to unload. A floor of always-live replicas (no scale-to-zero), or a pool of preloaded warm standbys, swaps “wait 40 s” for “0 s”. It is the cost of idle GPU in exchange for start-up latency: a capacity planning decision, not a technical one.

Summary table

#	Knob	What it attacks	Risk / cost
1	`iostat` while loading	knowing whether the bottleneck is disk or loader	none; always do it first
2	`--load-format`	single-threaded deserialisation	format compatibility
3	streamer concurrency	underused disk	host RAM for buffers
4	GPUDirect Storage	bounce through the CPU	requires nvidia-fs + supported FS
5	local NVMe vs network	the network in the path	duplicating weights per node
6	pre-pull / node cache	network on every start-up	local disk space
7	FP8/INT4 quantisation	bytes to move	quality (measure, do not assume)
8	V1 parallel loading	serial loading across GPUs	none if V1 is active
9	NUMA of the NVMe	H2D crossing the UPI	align with the rest of the NUMA lists
10	warm pods / no zero	the whole cold start	idle GPU paid for

How it connects with the rest of the stack

With autoscaling. The whole of scale-from-zero with KEDA rests on this: real elasticity is limited by the cold start, not by GPU availability. An autoscaler with 40 s of loading reacts late to every peak.

With disaggregated serving. In disaggregated prefill/decode, bringing up a decode pool on demand pays the cold start of loading the model into each new pod. The elasticity of the pattern depends on how fast those pods start.

With canary/blue-green. Every canary deployment loads a new version of the model alongside the old one. The validation time of a canary includes its cold start; large models make deployments slower and more expensive.

With NUMA and hugepages. The load’s host buffer wants to be pinned and NUMA-local, the same thing the host post asked for on the hot path. The loading path is another client of the same NUMA map.

With quantisation. FP8/INT4 is not only inference throughput: it is the direct lever on the bytes of the cold start.

With capacity planning. Sizing that ignores cold start underestimates how many replicas are needed to absorb a peak: if they take 40 s to start, you need a bigger permanent cushion.

Traps and things that are not what they seem

“The second time it loaded really fast.” This is the star trap. The first load filled the page cache (host RAM); the second reads from RAM, not from disk, and flies. But in production pods are ephemeral and are born on different nodes: the start-up that counts is the cold one, on a node where those files are not in page cache. Benchmarking the second load is measuring a situation that almost never happens at the moment that matters (the peak that triggers the autoscaler).

“GDS always speeds things up.” No. GDS removes the bounce through the CPU; if your bottleneck is the disk itself (saturated NVMe) or the deserialisation, GDS does not touch that part. Measure first (knob 1). It also demands nvidia-fs, a supported filesystem, and sometimes it does not work over whatever network storage you have.

“mmap makes loading instant.” mmap maps the file but reads nothing yet: the cost is deferred to the first access to each page. The time does not disappear, it moves, and the first token pays the page faults that start-up did not. You have moved the cold start into the latency of the first request, which is probably a worse place to have it.

Weights on network storage “because it is cleaner”. Sharing a model repository on Ceph RGW is fine for storing them; serving the cold start from there puts the network (and its latency and its contention) into the critical path. Local NVMe cache on the inference node.

Loading FP16 and quantising at start-up. Quantising on the fly during loading (for example FP16→FP8 on the GPU) can be slower than having the weights already quantised on disk: you move twice the bytes and on top of that you do conversion work. If you are going to serve in FP8, store the weights in FP8.

Optimising the load and ignoring T_cuda-graphs. Bringing weight loading down to 8 s and forgetting that CUDA graph capture adds several more seconds leaves the cold start half done. That second half of start-up is the subject of the next post.

Conclusion

The whole series optimised what happens with every token: the cable, the host, the network, the silicon. But before the first token there is a journey almost nobody times and that decides whether an inference pod opens in 10 s or in 60 s: pushing the model from disk to HBM. Intuition blames the GPU, and the GPU is innocent: HBM swallows 140 GB in 42 ms. The bottleneck is the supply chain: a disk reading at 14 GB/s, a PCIe copying at 50, and above all a default loader that deserialises tensor by tensor with a single thread and turns 10 s of bytes into 60 s of waiting. The solutions attack the two right fronts, fewer bytes (quantisation) and better transport (GDS, concurrent streamers, local NVMe), and give 4-7× almost for free. And above the technique, an idea that reorders the priorities: on an elastic platform, cold start is not a start-up detail, it is the ceiling on elasticity. The fastest GPU in the world does not scale if it takes 40 s to have the model inside. The pantry lift, the one nobody timed, is what decides what time the kitchen really opens.

References

NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/.
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/.
vLLM, Loading model weights with fastsafetensors: https://docs.vllm.ai/en/stable/models/extensions/fastsafetensor/.
foundation-model-stack, fastsafetensors (high-performance loader, GDS): https://github.com/foundation-model-stack/fastsafetensors.
Speeding up Model Loading with fastsafetensors (arXiv 2505.23072): https://arxiv.org/html/2505.23072v1.
NVIDIA, GPUDirect Storage: A Direct Path Between Storage and GPU Memory: https://developer.nvidia.com/blog/gpudirect-storage/.
NVIDIA, Magnum IO GPUDirect Storage (benchmarking and configuration): https://developer.nvidia.com/gpudirect-storage.

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

Sat, 06 Jun 2026 12:00:00 +0200

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

Disaggregated serving. The KV-cache that travels between the prefill pool and the decode pool moves over RDMA. These are large transfers, sensitive to latency and bandwidth, which in multi-node setups go out through the NIC.
Multi-node NCCL collectives. When tensor/pipeline parallelism crosses the node boundary, the all-reduce/all-gather operations no longer go over NVLink but over GPUDirect RDMA on the NIC.

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Three kernel facts the analogy compresses:

The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 piece	What it does	What Cilium eBPF puts in
`kube-proxy` (iptables/IPVS)	Service balancing	LB in eBPF; with `kubeProxyReplacement=true`, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)	VXLAN overlay + NetworkPolicy	native datapath (`routingMode=native`), L3/L4 and L7 NetworkPolicy in eBPF
veth per pod	namespace interface pair	netkit (kernel ≥6.8): namespace overhead ~0
host iptables traversal	netfilter hooks	eBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus = 2-31,34-63 # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33 # kubelet housekeeping (post 3)
IRQ affinity = 0-1,32-33 # NIC IRQs → housekeeping ONLY (this post)
 # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
 --namespace kube-system \
 --set routingMode=native \
 --set bpf.datapathMode=netkit \  # namespace overhead ~0 (kernel >=6.8)
 --set bpf.masquerade=true \
 --set kubeProxyReplacement=true \  # replaces RKE2's kube-proxy
 --set enableIPv4BIGTCP=true \  # 192k super-packets (mlx5/ice NIC)
 --set enableIPv6BIGTCP=true \
 --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
 --set bpf.mapDynamicSizeRatio=0.08 \
 --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
 name: gpu-nic-numa-aligned
spec:
 spec:
 devices:
 requests:
 - name: rdma-nic
 deviceClassName: dra.net  # NICs published by DRANET
 constraints:
 - requests: ["rdma-nic"]
 matchAttribute: "dra.net/pcieRoot" # same PCIe root as the GPU
 # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

References

Cilium, Tuning Guide (netkit, host-routing, BIG TCP, XDP, pin the IRQs and kill irqbalance): https://docs.cilium.io/en/stable/operations/performance/tuning/.
Cilium 1.19 (February 2026), Cilium at Ten Years — hardening of encryption, policies and observability: https://www.infoq.com/news/2026/02/cilium-119/.
Isovalent, Cilium 1.18 (IPv6, encrypted overlay, ingress bandwidth, policy perf): https://isovalent.com/blog/post/cilium-1-18/.
RKE2, Network Options (Canal by default; Cilium with kube-proxy replacement): https://docs.rke2.io/networking/basic_network_options.
Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/.
Kubernetes blog, v1.36: More Drivers, New Features, and the Next Era of DRA (May 2026): https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/.
DRANET (kubernetes-sigs), the network DRA driver and the paper The Kubernetes Network Driver Model (+59.6% all_gather / +58.1% all_reduce): https://github.com/kubernetes-sigs/dranet.
AKS Engineering, Optimizing RDMA performance for AI workloads on AKS with DRANET (April 2026): https://blog.aks.azure.com/2026/04/01/dranet-rdma-optimization-for-ai-on-aks.
Linux network tuning — IRQ affinity, RSS/RPS/RFS and NUMA softirq: https://andreaskaris.github.io/blog/networking/rss-irq-affinity-and-rps/.

Choosing the switchboard: which OSS gateway to put in front, and why the licence decides before the features

Sat, 06 Jun 2026 08:30:00 +0200

This post is the decision companion to The LLM inference router. That one explained what an inference router is and why it exists (catalogue, traffic splitting, cross-cutting policy, prefix-aware routing). This one answers the question that actually blocks a deployment: which one to choose, with verified licences and an order of criteria that is not the one most people use.

TL;DR

The gateway is the piece 100 % of the traffic passes through and the most expensive to replace later: the client SDK, the auth and rate-limit policies, the tracing and the model catalogue all couple to it. That is why the order of the criteria matters, and it is not the usual one. First the licence: not “is it open source?” but “are the features I need under a permissive licence or behind an Enterprise wall?”. Second the architectural fit: is it a citizen of your Kubernetes (Gateway API) or a separate process you have to operate on the side? Third, maturity and documentation. Fourth, the features. With data verified as of June 2026: LiteLLM is MIT in the core, but SSO, audit logs, fine-grained RBAC and several guardrails are Enterprise; Kong has an Apache 2.0 core but the AI plugins that matter (semantic cache, prompt guard, AI proxy advanced) are gated; Envoy AI Gateway, Gateway API Inference Extension (GIE), Higress, APISIX and Bifrost are Apache 2.0 end to end. For an RKE2 + vLLM stack with a K8s-native priority (the case we assume here), the recommendation is to adopt the Gateway API Inference Extension model, the Endpoint Picker that routes with awareness of prefix cache, KV and LoRA, which is exactly what multiplies the hit rate. Implement it with Envoy AI Gateway if the AI-native trajectory weighs more, or with Higress if today’s maturity weighs more; and put LiteLLM (MIT) behind it as an optional multi-provider control plane. With explicit scepticism about v0.x releases, about “OpenAI-compatible ≠ inference-aware” and about phone-home telemetry.

The principle that reorders the criteria: the gateway is a marriage, not a date

Some pieces of the stack can be swapped on a Sunday afternoon: the embedding model, the reranker, even the inference engine behind an OpenAI-compatible interface. The gateway cannot. It is the piece everything else couples to: every client SDK points at its URL, the security policies live in it, the tracing is born in it, it defines the model catalogue. Ripping it out two years later means touching every consumer at once. It is the infrastructure decision with the highest cost of reversal in the whole serving layer.

When a decision is expensive to reverse, the dominant criterion is not “what does it do today?” but “can I own it and operate it for years without surprises?”. And that puts the licence ahead of the features. A brilliant tool whose SSO, audit log or RBAC live behind an Enterprise contract is a tool that, the day your ENS deployment needs those controls, forces you to pay or to migrate, which is exactly the scenario the choice was supposed to avoid. That is why this post’s filter order is licence → fit → maturity/docs → features, and not the other way round.

The field of candidates (June 2026)

The real OSS pieces anyone would put in front of an on-premise vLLM fleet:

Candidate	What it is	Language
LiteLLM Proxy	OpenAI-compatible gateway, 100+ providers, virtual keys, spend	Python (FastAPI)
Envoy AI Gateway	AI layer on top of Envoy Gateway; integrates GIE/EPP, InferencePool	Go (Envoy)
Gateway API Inference Extension (GIE)	K8s-SIG extension: inference-aware Endpoint Picker	Go
Higress	API gateway + AI plugins, Envoy/Istio, CNCF	Go/C++
Apache APISIX	Mature API gateway with AI plugins	Lua/Nginx
Kong (+ AI plugins)	API gateway; Apache 2.0 core, Enterprise AI plugins	Lua/Nginx
Bifrost	High-performance AI-first gateway	Go
vLLM Production Stack / Semantic Router	vLLM-specific router, KV/prefix/intent-aware	Go/Python

Filter 1 — Licence: where the small print is

This is where half the options die, and where “open source” misleads if you do not look at the detail. What has been verified:

Candidate	Core licence	What is gated (paid)
LiteLLM	MIT	SSO (Okta/Azure AD), audit logs, JWT auth, fine-grained RBAC, several guardrails (llmguard, llamaguard, prompt-injection) → Enterprise (~250 USD/month and rising)
Envoy AI Gateway	Apache 2.0	— (full AI layer is OSS)
GIE (Inference Extension)	Apache 2.0	— (K8s-SIG project)
Higress	Apache 2.0	— (AI plugins included)
APISIX	Apache 2.0	— (more built-in AI than Kong OSS)
Kong	Apache 2.0 (core)	AI Semantic Cache, AI Prompt Guard, AI Proxy Advanced, AI RAG Injector → Enterprise
Bifrost	Apache 2.0	—

Two conclusions that settle the field:

Kong falls out for an OSS AI deployment. Its core is Apache 2.0, but precisely the AI plugins that would justify choosing it (semantic cache, prompt guard, proxy advanced) are Enterprise, with contracts that the sector’s own comparisons put above five figures a year. For AI on an OSS budget, APISIX offers more built-in at no cost. Kong remains excellent as a classic API gateway; as an OSS AI gateway, no.
LiteLLM really is MIT in the core, and that is genuine: you can fork it, modify it and use it commercially. But SSO, audit logs, fine-grained RBAC and several guardrails are Enterprise. Under the criterion we set (“a permissive core is enough, governance is solved with OIDC and external auditing in the stack”) LiteLLM stays in play; under an “all OSS or nothing” criterion it would be wounded. It pays to know exactly what falls on which side of the line before committing.

LiteLLM: what is gated and its OSS equivalent

The nuance that decides whether LiteLLM’s gating is a real problem or not: “paid built-in feature” is not the same as “capability impossible in OSS”. In almost every case the capability is achieved by changing the how, wiring OSS through LiteLLM’s open hook, or solving it in the layer next door (the gateway, the observability). The breakdown, verified against the docs:

Capability	In LiteLLM Enterprise	OSS equivalent
Guardrails	prebuilt integrations (`llmguard`, `llamaguard`, `lakera`, `aporia`, `hide_secrets`)	the framework + custom guardrail hook + Presidio are OSS (MIT core); you invoke LLM Guard, Llama Guard or Presidio from the hook yourself
Audit log	turnkey UI with retention policies	request/response logging + custom callbacks + OTel/Langfuse exporters (official OSS integration) → you build the trail and you own it
RBAC	fine-grained (`enforce_rbac`, org/team/user roles)	coarse (virtual keys per team/budget/model) is OSS; fine-grained is done at the edge in the gateway (`ext_authz` + OPA)
SSO	Admin UI SSO (Okta/Azure/Google/OIDC)	user/API SSO is solved by fronting with OSS OIDC (Keycloak + oauth2-proxy) or in the gateway itself; JWT auth + JWT→virtual-key mapping are already in the core

The conclusion that closes the decision: in a K8s-native design with a gateway in front, OIDC and authorisation live in the gateway (Envoy/Higress + OPA) and the audit trail in Langfuse/OTel, both Apache 2.0 and already in the stack, so LiteLLM placed behind barely needs its Enterprise tier: the governance pieces sit where they architecturally belong and happen to be OSS. The only real cost is the DIY integration and maintenance, and, for ENS, designing the guarantees of the audit trail (retention, immutability/WORM) is on you, it does not come out of the box.

The rest, Envoy AI Gateway, GIE, Higress, APISIX and Bifrost, pass the licence filter clean: Apache 2.0 end to end, with no critical feature behind a wall.

Filter 2 — Fit: Kubernetes citizen or tenant?

With a K8s-native priority (RKE2), the question is whether the gateway is modelled as cluster resources, versioned with GitOps, observed with the same tools and integrated with the scheduler, or whether it is a separate process you have to operate on the side. This is where the structural novelty of 2025-2026 appears: the Gateway API Inference Extension (GIE).

GIE is an extension of the standard Kubernetes Gateway API, from SIG-Network, which adds two pieces: the InferencePool (a pool of replicas of a model as a native resource) and the Endpoint Picker (EPP), a scheduler that decides which replica each request goes to according to the inference state: queue length, available LoRA adapters, and, the key piece, the prefix cache state of each replica. It is exactly the prefix-aware routing that the router post explained as the reason hit rate goes from 5-15 % to 60-85 %, now as a community standard instead of a proprietary feature of each product.

Why this settles the decision when the priority is K8s-native: GIE turns inference-aware routing into a standard interface that several implementations satisfy (Envoy AI Gateway, Higress, kgateway, Istio, GKE Inference Gateway). You pick an implementation today and, if you change it tomorrow, the resource model (InferencePool, HTTPRoute) is preserved. That is precisely the property you wanted from a decision that is expensive to reverse: the coupling is to a standard, not to a product. A standalone proxy like LiteLLM, however good it is, lives outside this model. It is one more Deployment, with its own config, its own catalogue format and its own management plane.

Filter 3 — Maturity and documentation: the uncomfortable honesty

Here comes the tension no marketing comparison admits: what is K8s-native-correct and what is battle-tested-today do not quite coincide in June 2026.

Envoy AI Gateway: the AI layer is at v0.5, pre-1.0 and young. But it runs on Envoy and Envoy Gateway, which are among the most mature things that exist in proxies. It has the most complete GIE/EPP integration, model virtualisation and OpenInference tracing. Risk: the AI layer still moves fast (breaking changes are possible). Docs are growing, and good.
GIE: the project is reaching GA (one of the first standardised inference-aware extensions). Standard maturity is high and rising; the maturity of each implementation varies.
Higress: the most mature today among the K8s-native Apache-2.0 options with AI. Production at Alibaba scale, CNCF, Envoy/Istio base, AI plugins included, Gateway API support. If “mature” weighs more than “cutting-edge AI-native”, it is the safe bet.
APISIX: a very mature classic gateway, with growing AI plugins; less specialised in inference-aware routing (prefix/KV) than the GIE model. Battle-tested at massive scale.
LiteLLM: the one with the best documentation and the widest adoption in the field, by a distance. Mature as a proxy. Its ceiling is the language (Python/FastAPI: ~250-300 RPS per instance, scaling out by replicas) and the governance gating already covered.
Bifrost: Apache 2.0, Go, the highest performer (~11 µs overhead at 5,000 RPS), with semantic caching and governance built in. Younger and less proven over years, but technically strong.

Decision matrix

Weighting the four filters for the case of RKE2 + vLLM, K8s-native priority, permissive core acceptable:

Candidate	Licence	K8s-native	Maturity	Docs	Inference-aware	Verdict
Envoy AI GW + GIE/EPP	✅ Apache 2.0	✅✅ standard	⚠️ v0.5 (mature Envoy)	✅ good	✅✅ prefix/KV/LoRA	Primary (trajectory)
Higress (+ GIE)	✅ Apache 2.0	✅✅	✅✅ production	✅ good	✅ via GIE/plugins	Primary (maturity today)
APISIX	✅ Apache 2.0	✅	✅✅	✅✅	⚠️ less specialised	Solid alternative
LiteLLM	⚠️ MIT core, gov. gated	❌ separate proxy	✅✅	✅✅✅	⚠️ basic	Control plane behind
Bifrost	✅ Apache 2.0	⚠️	⚠️ young	✅	✅	One to watch (perf)
Kong	⚠️ AI plugins Enterprise	✅	✅✅	✅✅	❌ gated	Ruled out (OSS AI)

The recommendation

For the case we fixed, RKE2 + vLLM, K8s-native priority, permissive licence with a sufficient core, the recommendation has two layers, not one:

Data layer (what sits in front of the replicas): adopt the Gateway API Inference Extension model. It is the decision that ages well because it couples you to an Apache 2.0 standard, not to a product, and because it brings the prefix/KV/LoRA-aware routing that really moves the hit rate. For the implementation:

If the AI-native trajectory weighs more and you accept operating a v0.x layer on top of a mature Envoy: Envoy AI Gateway + GIE/EPP. That is where the cutting edge and the most complete integration are.
If today’s maturity outweighs everything else (ENS, critical production, zero appetite for v0.x in the path of 100 % of the traffic): Higress with GIE. CNCF, proven at scale, Apache 2.0, and migratable to the standard EPP.

Control layer (optional, behind): LiteLLM (MIT) if you need a single multi-provider point with virtual keys, spend tracking and fallbacks to external models. It goes behind the data gateway, not in its place, and you accept that SSO, audit and RBAC are solved with the cluster’s own OIDC and auditing (as we set out in the criterion). If your front end is only on-premise vLLM with no external providers, this layer is probably unnecessary.

What I would not recommend here: Kong (its AI features are gated, which contradicts the licence criterion); and starting with LiteLLM as the main data gateway in a stack that will be multi-replica and K8s-native, since its Python ceiling and its position outside the Gateway API model turn it into debt the day you scale.

Applied to our infrastructure: RKE2 + vLLM + 4×H100

The concrete deployment on the generic reference cluster:

# The InferencePool groups the replicas of a model (GIE resource)
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata: { name: llama-70b-pool }
spec:
 selector: { app: vllm-llama70b }
 extensionRef: { name: llama-70b-epp }  # the Endpoint Picker
---
# The standard Gateway API route points at the pool
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: llama-70b }
spec:
 parentRefs: [ { name: ai-gateway } ]
 rules:
 - backendRefs:
 - group: inference.networking.x-k8s.io
 kind: InferencePool
 name: llama-70b-pool

Three notes on how this fits the blog series:

It lives in layer 1 of the seven-layer stack, in front of the replicas pinned by the RKE2 resource managers. The gateway routes; the kubelet pins; both are resources of the same GitOps cluster.
The EPP reads the prefix cache state of each replica, which connects directly with the engineering of prefix cache hit rate: the gateway is what turns that affinity into actual routing.
The gateway’s gen_ai.* tracing feeds the OTel observability pipeline and lands in Langfuse, closing the loop of the Observe layer.

Traps and things that are not what they seem

“It is open source” without reading what falls under Enterprise. The mistake this post tries to avoid: Kong and LiteLLM are OSS, but the features that justify choosing them for AI (AI plugins in Kong; SSO/audit/RBAC/guardrails in LiteLLM) are partly or wholly gated. “Open source” is the wrong question; “is what I need permissive?” is the right one.

“OpenAI-compatible” ≠ “inference-aware”. Almost all of them expose the OpenAI API. That does not mean they understand the KV cache state, the queue of each replica or the LoRA adapters. API compatibility is table stakes; intelligent routing (EPP) is a different league. Do not confuse “speaks OpenAI” with “routes well”.

Choosing by the RPS benchmark. Bifrost wins on raw overhead, but the bottleneck of an on-premise vLLM fleet is rarely the proxy: it is the GPUs. Optimising the gateway for 5,000 RPS when your replicas serve 300 req/s is solving the problem you do not have. The latency the gateway adds matters; its peak throughput almost never does, in this context.

Phone-home telemetry in a sovereign deployment. Several gateways send telemetry home by default. In an ENS or sovereignty context, that is an audit finding, not a detail. Verify and disable the phone-home before putting the gateway in the path of 100 % of the traffic; document it for the ENS/42001/EU AI Act controls file.

Betting on a v0.x in the critical path with no rollback plan. Envoy AI Gateway v0.5 is promising, but it sits in the path of 100 % of the traffic. If you choose it, have the rollback rehearsed and follow the changelog: v0.x releases break things. The GIE standard mitigates this (you can change implementation), but the specific implementation has to be operated with a safety net.

Confusing the data gateway with the control plane. LiteLLM in front of everything seems to simplify, but it mixes two roles: routing data traffic (where you want K8s-native + inference-aware) and managing multi-provider keys and spend (where LiteLLM shines). Separate them: a GIE-native data gateway, with LiteLLM as a control plane behind it if needed.

Conclusion

The gateway is the piece with the highest cost of reversal in the serving layer, and that forces you to invert the intuitive order of criteria. First the licence, not “is it OSS?” but “is what I need permissive or is it gated?”, which already rules Kong out for OSS AI and puts an asterisk on LiteLLM for its Enterprise governance. Then the fit, where the Gateway API Inference Extension turns inference-aware routing into an Apache 2.0 standard you can couple to without marrying a product. And only after that maturity, docs and features, where honesty forces you to admit that the cutting-edge AI-native option (Envoy AI Gateway v0.5) and the battle-tested-today one (Higress) are still not the same thing. For an RKE2 + vLLM stack with a K8s-native priority, the answer is not a product but a model, GIE/EPP, implemented with Envoy AI Gateway if you look at the trajectory or with Higress if you look at maturity, and LiteLLM as an optional control plane behind. The decision that ages well is the one that couples you to a standard, not to a vendor; in gateways, in June 2026, that standard finally has a name.

References

Kubernetes, Introducing Gateway API Inference Extension (blog oficial, jun 2025): https://kubernetes.io/blog/2025/06/05/introducing-gateway-api-inference-extension/.
Gateway API Inference Extension, doc del proyecto (Endpoint Picker, InferencePool): https://gateway-api-inference-extension.sigs.k8s.io/.
Envoy AI Gateway, sitio y release notes (v0.x, GIE integration): https://aigateway.envoyproxy.io/.
LiteLLM, Enterprise Features (qué está gated): https://docs.litellm.ai/docs/proxy/enterprise · licencia MIT: https://github.com/BerriAI/litellm/blob/main/LICENSE.
Kong, Announcing Kong’s Open Source AI Gateway y matriz de plugins Enterprise: https://konghq.com/blog/product-releases/announcing-kong-ai-gateway.
Higress (Apache 2.0, CNCF, AI gateway): https://higress.cn/en/.
Apache APISIX, APISIX vs Kong (cobertura de IA OSS): https://apisix.apache.org/learning-center/apisix-vs-kong/.
vLLM, Semantic Router v0.1 Iris y Production Stack (routing específico de vLLM): https://blog.vllm.ai/2026/01/05/vllm-sr-iris.html.

The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

Sat, 06 Jun 2026 08:00:00 +0200

The close of the “under the engine” series. We saw the cable between GPUs and the host: NUMA, hugepages and CPU isolation done by hand. Here is the piece that makes it declarative and at scale: how the RKE2 kubelet pins each vLLM pod to the right NUMA node without a single script.

TL;DR

Pinning NUMA with numactl/isolcpus/taskset, the subject of the previous post, does not scale to a cluster where pods come and go and there are dozens of nodes. The kubelet automates it with three components that work as Hint Providers for a central coordinator, the Topology Manager: the CPU Manager (assigns exclusive CPUs to containers of Guaranteed pods with whole CPUs), the Memory Manager (NUMA-local memory and hugepages) and the Device Manager/GPU plugin (which knows which GPU is on which NUMA node). With the single-numa-node policy, the Topology Manager only admits the pod if its CPUs, its memory and its GPU fit in the same NUMA domain; if they do not fit, it rejects the pod, strict admission, like the maître d’ who will not seat a party of eight unless there is a table for eight. On RKE2 all of this is configured with kubelet-arg in /etc/rancher/rke2/config.yaml. This post explains the mechanism, gives the 10 knobs and takes apart the gotchas that break pinning silently: the cpu_manager_state file you have to delete when changing policy, the QoS that has to be exactly Guaranteed, and the reserved-cpus that must match the host’s isolcpus. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the orchestration that materialises the host

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

A restaurant has tables of different sizes and, house rule, tables are never pushed together. A party of eight arrives. The maître d’ looks for a single table where all eight fit. If there is one, he seats them; if only tables for four are left, he does not accept them. He would rather turn the booking away than seat the group split across two separate tables, because he knows a split dinner goes badly.

That maître d’ is the Topology Manager under the single-numa-node policy. The “party” is an inference pod asking for CPUs, memory and a GPU. The “table” is a NUMA node. The maître d’ asks three assistants (are there free CPUs on any node? the CPU Manager; is there free memory? the Memory Manager; is there a free GPU? the Device Manager) and only admits the pod if all three resources fit on the same node. If not, he rejects it (the pod ends up Failed with TopologyAffinityError), and the scheduler will try another node.

The difference from the previous post: there you were the maître d’, seating each process by hand with numactl. Here the maître d’ is the kubelet, and it does it for every pod, on every node, automatically, rejecting whatever does not fit. That is what turns artisanal pinning into a declarative property of the cluster.

The mechanism: Hint Providers and the coordinator

The Topology Manager does not assign resources; it coordinates the ones that do. The flow, when a Guaranteed pod arrives at a node:

The three managers are Hint Providers: each one tells the Topology Manager on which NUMA node(s) it could satisfy its part. The Topology Manager computes the intersection and, depending on the policy, decides:

none (default): no coordination; each manager does its own thing without aligning. No NUMA guarantee.
best-effort: it tries to align on one node; if it cannot, it admits anyway (on whichever node). Better than nothing, no guarantee.
restricted: if it fails to align, it rejects the pod. Strict, but it allows multi-node affinity if the intersection provides it.
single-numa-node: it requires everything to fit on a single NUMA node, or it rejects. The strictest one, and the one that genuinely guarantees the locality of the previous post.

And two preconditions without which none of this kicks in:

The pod has to be QoS Guaranteed: requests == limits on CPU and memory, and whole CPUs (not 500m). Only then does the CPU Manager assign exclusive CPUs.
The CPU Manager has to be on the static policy (not none).

Without those two, the Topology Manager has nothing to align and pinning does not happen, even with the policy in place. That is gotcha no. 1.

How it is configured on RKE2

RKE2 passes arguments to the kubelet with the kubelet-arg key in /etc/rancher/rke2/config.yaml. The reference configuration for GPU inference nodes:

# /etc/rancher/rke2/config.yaml (on every agent node with GPUs)
kubelet-arg:
 - "cpu-manager-policy=static"
 - "topology-manager-policy=single-numa-node"
 - "topology-manager-scope=pod"
 - "memory-manager-policy=Static"
 - "reserved-cpus=0-1,64-65" # housekeeping; must match the host
 - "system-reserved=memory=8Gi"
 - "kube-reserved=memory=4Gi"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi" # required by Memory Manager Static
node-label:
 - "fibercli.local/numa-pinned=true"

After deploying it: systemctl restart rke2-agent. Critical gotcha: if the node has already run with cpu-manager-policy=none, there is a state file /var/lib/kubelet/cpu_manager_state that pins the old policy; changing the arg without deleting that file makes the kubelet fail to start or ignore the new policy. You have to stop the agent, rm /var/lib/kubelet/cpu_manager_state, and start again. (The same applies to memory_manager_state.)

And the vLLM pod, to be eligible, Guaranteed with whole CPUs:

resources:
 requests:
 cpu: "16" # whole, not a weird fractional "16000m"
 memory: "200Gi"
 nvidia.com/gpu: "2" # TP=2 → 2 GPUs on the same NUMA node
 hugepages-1Gi: "16Gi"
 limits:
 cpu: "16" # == requests → Guaranteed QoS
 memory: "200Gi"
 nvidia.com/gpu: "2"
 hugepages-1Gi: "16Gi"

With this, on a node with the config above, the kubelet assigns 16 exclusive CPUs from the NUMA node where the 2 requested GPUs live, its local memory and the hugepages, or it rejects the pod if they do not fit together. The artisanal pinning of the previous post, now declarative.

The 10 knobs to reach for

Ordered by dependency (the first ones are a precondition for the later ones). The canonical reference is the Kubernetes Topology Manager documentation and the RKE2 configuration.

Knob 1 — `cpu-manager-policy=static`: the foundation

Without this there are no exclusive CPUs and nothing else kicks in.

kubelet-arg: [ "cpu-manager-policy=static" ]

Gotcha: changing it requires deleting /var/lib/kubelet/cpu_manager_state and restarting the kubelet, or startup fails. It is cause no. 1 of “I set the policy and it does not pin”.

Knob 2 — `Guaranteed` QoS + whole CPUs: the pod’s precondition

This is not node config, it is pod config, but without it knob 1 does nothing for that pod. requests == limits on CPU and memory, and whole CPUs. A cpu: 500m or a requests != limits downgrades the pod to Burstable and it loses the pinning. Plenty of people set the node policy and forget the pod’s QoS.

Knob 3 — `topology-manager-policy=single-numa-node`: strict admission

kubelet-arg: [ "topology-manager-policy=single-numa-node" ]

The strict maître d’. For GPU inference it is the right policy: it guarantees that CPU+memory+GPU share a node. best-effort guarantees nothing (it admits misaligned); restricted allows multi-node affinity. Start with single-numa-node and drop to restricted only if you have admission problems.

Knob 4 — `topology-manager-scope=pod`: group the whole pod

kubelet-arg: [ "topology-manager-scope=pod" ]

With container scope (the default), each container is aligned separately; with pod scope, the whole pod goes to the same node. For a vLLM pod with sidecars (metrics, proxy), pod scope stops the sidecar from dragging the main container onto another node. Recommended for inference.

Knob 5 — `memory-manager-policy=Static` + `reserved-memory`: NUMA-local memory

kubelet-arg:
 - "memory-manager-policy=Static"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi"

Memory Manager Static forces cpuset.mems so that the pod’s memory comes out of the right node (and so do the hugepages). It requires declaring reserved-memory per node, or the kubelet does not start. It is the declarative equivalent of the previous post’s --membind.

Knob 6 — `reserved-cpus`: the housekeeping cores (must match `isolcpus`)

kubelet-arg: [ "reserved-cpus=0-1,64-65" ]

It reserves cores for the system and the daemons; the rest are left for exclusive pods. Key point of the series: these reserved-cpus must be the same cores you left out of isolcpus on the host (previous post). If the host isolates 2-31 but RKE2 reserves 0-3, there is a mismatch: isolated cores that the kubelet assigns to pods without them actually being quiet. Coordinate the two layers.

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

The Device Manager can only give a correct NUMA hint if the GPU plugin exposes which node each GPU is on. The NVIDIA device plugin / GPU Operator does this, but you have to verify that the topology information arrives (in some versions it requires flags). Without a GPU hint, the Topology Manager aligns CPU and memory but not the GPU, and GPU locality is precisely the one that matters most.

Knob 8 — hugepages as a pod resource

resources:
 limits:
 hugepages-1Gi: "16Gi" # the node must have them pre-reserved (post 2, knob 4)

The hugepages you reserved at host boot (previous post) are requested as a resource. The Memory Manager assigns them NUMA-local. If you request them without having reserved them on the node, the pod does not get scheduled.

Knob 9 — `system-reserved` / `kube-reserved`: do not oversubscribe

kubelet-arg:
 - "system-reserved=cpu=500m,memory=8Gi"
 - "kube-reserved=cpu=500m,memory=4Gi"

It reserves resources for the system and the K8s components so the node does not run out of air under load. Badly calibrated, either the node chokes (too little reserved) or you waste capacity (too much). It must be consistent with reserved-cpus.

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

# GPU node: taint to repel anything that does not need a GPU
node-taint: [ "nvidia.com/gpu=present:NoSchedule" ]
node-label: [ "fibercli.local/numa-pinned=true" ]

Keep the NUMA-pinned nodes for inference and push out whatever does not need it (databases, the Langfuse backend, runners). A ClickHouse stealing memory bandwidth from a carefully pinned vLLM pod throws away all the work of the previous nine knobs. Workload isolation is the closing move.

Summary table

#	Knob	Where	Function
1	`cpu-manager-policy=static`	kubelet-arg	exclusive CPUs (foundation)
2	`Guaranteed` QoS + whole CPUs	pod spec	pinning precondition
3	`topology-manager-policy=single-numa-node`	kubelet-arg	strict NUMA admission
4	`topology-manager-scope=pod`	kubelet-arg	group the whole pod
5	`memory-manager-policy=Static`	kubelet-arg	NUMA-local memory/hugepages
6	`reserved-cpus`	kubelet-arg	housekeeping (match isolcpus)
7	GPU plugin with topology	GPU Operator	NUMA hint for the GPU
8	`hugepages-1Gi`	pod spec	hugepages as a resource
9	`system/kube-reserved`	kubelet-arg	do not oversubscribe
10	taints + labels	node config	isolate GPU workloads

Verifying that the pinning really happened

Do not trust that the config “is in place”. Check:

# Is the active policy the one you set?
cat /var/lib/kubelet/cpu_manager_state | jq .policyName # "static"

# Which exclusive CPUs does the container have?
kubectl exec <pod> -- cat /sys/fs/cgroup/cpuset.cpus.effective

# Inside the pod: is the assigned GPU local to those cores?
kubectl exec <pod> -- nvidia-smi topo -m

# Were there topology rejections?
kubectl describe pod <pod> | grep -i TopologyAffinityError

A pod in Failed with TopologyAffinityError is not a bug: it is the maître d’ doing his job. That node did not have a table where CPU+memory+GPU fitted together. The answer is to review the sizing of the pod or the node, not to relax the policy casually.

How it connects with the rest of the stack

With the host (previous post). This post is the declarative automation of that one. cpu-manager-policy=static materialises the taskset; memory-manager-policy=Static materialises the --membind; reserved-cpus must match isolcpus. The two layers are one single decision seen from two places: the host executes it, the kubelet declares it. Uncoordinating them (isolcpus 2-31 vs reserved-cpus 0-3) breaks both.

With the interconnect (post 1). The Topology Manager pins the right GPU to the pod, but if you ask for 2 GPUs for TP=2, you will want those two to share NVLink. The NUMA policy guarantees they are on the same socket; that they are NVLink-connected is guaranteed by the baseboard hardware (post 1, knob 1). The two things together are what makes TP=2 perform.

With autoscaling. When KEDA scales vLLM pods, each new replica goes through Topology Manager admission. If the node has no free “table”, the pod stays pending. Pod autoscaling and node autoscaling (cluster-autoscaler) have to account for NUMA granularity, not just aggregate CPU/memory.

With capacity planning. Sizing changes: it is not “128 vCPU per machine”, it is “128 minus the reserved-cpus, in blocks that fit per NUMA node”. A machine with 2 sockets × 64 cores does not serve a pod asking for 80 cores under single-numa-node: they do not fit at one table. Planning has to reason per NUMA node, not per machine.

With service coexistence. The knob 10 taint is what keeps Langfuse, databases and runners off the inference nodes. Without that boundary, all the fine-grained pinning gets eaten by a noisy neighbour. Observability goes on its own nodes; inference, pinned, on its own.

Traps and things that are not what they seem

Policy in place, QoS forgotten. The most common mistake: cpu-manager-policy=static on the node but the pod is Burstable (requests != limits or fractional CPU). Pinning does not happen and nobody warns you. Guaranteed QoS with whole CPUs is a necessary condition.

Fossilised cpu_manager_state. Changing policy without deleting /var/lib/kubelet/cpu_manager_state (and memory_manager_state) makes the kubelet fail or ignore the change. Stop agent → delete file → start.

reserved-cpus ≠ isolcpus. If the host isolates some cores and RKE2 reserves others, the managers assign cores to pods that are not really quiet, or they leave isolated cores idle. The two lists have to be consistent. It is the coordination failure between the previous post and this one.

GPU plugin without NUMA topology. If the device plugin does not expose each GPU’s NUMA node, the Topology Manager aligns CPU and memory but leaves the GPU to chance, and GPU locality is the one that weighs most. Verify that the GPU Operator publishes the topology.

single-numa-node that rejects too much. If the pods ask for more resources than fit on one node (for example, more cores than a socket has), rejection is constant. The answer is not to drop to best-effort (which silences the problem by serving misaligned), but to size the pod so it fits at one table, or to accept restricted with full knowledge of what you are doing.

Believing best-effort “is nearly the same”. best-effort admits the pod even if it fails to align: it gives you a false sense of NUMA-awareness while you serve from the wrong socket. For inference with a tail SLO, single-numa-node or restricted; best-effort only if the alternative is scheduling nothing at all.

Conclusion

The previous post pinned by hand; this one does it at scale and with a guarantee that artisanal numactl did not give: strict admission. The kubelet, via CPU Manager, Memory Manager and Topology Manager, acts as a maître d’ who only seats the pod if its CPUs, its memory and its GPU fit at the same NUMA table, and who rejects what does not fit instead of serving a split dinner. Of the ten knobs, the first two, cpu-manager-policy=static and Guaranteed QoS with whole CPUs, are the precondition without which the other eight do nothing, and they are exactly the ones most often forgotten; the rest tune the policy, the memory, the hugepages and the coexistence. The thread that closes the series: the inference performance that looked like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a cable problem (NVLink not used), a host problem (remote NUMA, jitter) or an orchestration problem (pinning that never happened because the QoS was wrong). Going down a level is not infrastructure snobbery: it is where the root causes live that no application-layer dashboard is ever going to point out for you.

References

Kubernetes, Control Topology Management Policies on a node: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/.
Kubernetes, Control CPU Management Policies on the Node: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/.
Kubernetes, Control Memory Management Policies on a Node: https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/.
RKE2, Configuration Options (kubelet-arg in config.yaml): https://docs.rke2.io/install/configuration.
RKE2, Advanced Options and Configuration: https://docs.rke2.io/advanced.
rancher/rke2, discussion #3034 CPU Management Policies for RKE2 (the cpu_manager_state gotcha): https://github.com/rancher/rke2/discussions/3034.

The floor next door: NUMA, hugepages and CPU isolation, or why your GPU waits on the kernel

Sat, 06 Jun 2026 07:30:00 +0200

Second post in the “below the engine” series. The first opened up the cable between GPUs (NVLink/NCCL). This one goes down to the host: the cores, the memory and the kernel that surround those GPUs and that, badly configured, leave them waiting. The third will explain how Kubernetes automates all of this; here is the raw layer, the one you have to understand before delegating it.

TL;DR

A node with 4×H100 SXM is, physically, a two-socket server = two NUMA domains. Each socket has its own cores, its own memory channels and PCIe lanes towards half of the GPUs and NICs. Inference is not only GPU: the host does work on the hot path of every token, launching CUDA kernels, sampling the next token, tokenising, moving pinned buffers between host and GPU, running the NCCL threads. If those threads and their memory land on the socket that is not local to the GPU, every access crosses the inter-socket link (UPI/Infinity Fabric): 2-3× slower and with p99 spikes. There are three kernel levers that decide the latency tail and that almost nobody touches: locality (NUMA affinity: CPU, memory, GPU and NIC on the same “floor”), page tables (hugepages: a few large pages instead of millions of small ones, plus pinned memory for DMA), and jitter (CPU isolation with isolcpus/nohz_full/rcu_nocbs plus IRQ affinity, so that the kernel does not interrupt the thread launching the next decode kernel). This post explains the mechanism, gives the 10 real knobs, and connects with the interconnect and latency-bound decode. With scepticism about which knobs move the needle in inference and which are cargo cult inherited from low-latency trading.

Where you are: the host below the cable

The analogy: the two-floor office

Picture a consultancy in a building with two floors. On each floor there are desks (CPU cores), an archive with the files (the memory of that socket) and a loading bay connecting to the outside (the PCIe lanes towards the GPUs and NICs of that socket). An analyst works fast while everything they need is on their floor: they reach out and take the file from the archive next to them.

The problem starts when the analyst is on floor 1 but their file is in the archive on floor 2. Every time they need it, they take the lift. The work “functions”, but every lookup costs a trip. And if on top of that the loading bay through which their materials arrive (their GPU) is on the other floor, every delivery crosses the building. This is NUMA: local access (same floor) is fast; remote access (another floor, via the inter-socket link) is 2-3× slower.

And there are two more ways to ruin that analyst even when they are on the right floor:

Interrupting them constantly. Every few minutes, a tannoy announcement, a colleague with a question, a fire alarm test. Every interruption breaks their concentration right when they were about to deliver. This is kernel jitter: the scheduler tick, device IRQs, RCU callbacks, interrupting the host thread right when it was about to launch the next GPU kernel. CPU isolation is putting them in an office with a “do not disturb” sign.
Giving them an index of a thousand tiny tabs. If to find each file they have to search an index with a million minuscule entries, they lose time in the search. If the index has a few large entries, they find it instantly. These are hugepages: pages of 2 MB or 1 GB instead of 4 KB reduce the pressure on the TLB (the cache of the page index).

The thesis: the GPU is expensive and fast, but it spends a surprising fraction of decode waiting on the host. If the host is on the wrong floor, interrupted, and searching a giant index, the GPU, the 30,000 € resource, waits. The three levers in this post exist so that it does not wait.

The mechanism: what the host does on the token path

It is tempting to think that in inference “the GPU does everything”. It is not true. For every token, the host (CPU) does, at a minimum:

Launch the CUDA kernels for each operation. The GPU does not decide what to execute; the host keeps putting kernels in its queue. In decode, where each kernel is short, the host has to run ahead feeding the queue; if the host thread stalls, the GPU runs out of work: a bubble.
Sample the next token (argmax/top-p/top-k over the logits), which comes back from device to host.
Tokenise the input and detokenise the output.
Move pinned buffers (page-locked) between host and GPU by DMA: prompts, logits, and in offload configurations, part of the KV cache.
Run the NCCL threads that coordinate the collectives of the previous post.

All of that is CPU and host memory work. And all of it suffers if:

The process runs on the socket that is not local to its GPU → every DMA and every memory access crosses the inter-socket link.
The kernel interrupts the threads → bubbles in the GPU queue.
The memory is not pinned or uses small pages → page faults, TLB misses, and worse: if the memory of the DMA transfer is not pinned, the driver makes an intermediate copy.

The map: `nvidia-smi topo -m`

Everything starts by looking at the map. nvidia-smi topo -m shows, for each GPU, which NUMA node and which cores it is local to, and through what kind of path it talks to each NIC and to each other GPU:

 GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA Affinity
GPU0 X NV18 NV18 NV18 PIX 0-31,64-95 0
GPU1 NV18 X NV18 NV18 SYS 0-31,64-95 0
GPU2 NV18 NV18 X NV18 SYS 32-63,96-127 1
GPU3 NV18 NV18 NV18 X SYS 32-63,96-127 1

Read it like this: GPU0 and GPU1 are local to NUMA node 0 (cores 0-31, 64-95); GPU2 and GPU3 to NUMA node 1. NV18 between GPUs = 18 NVLink links (the good stuff, from the previous post). In the NIC column: PIX = a single PCIe switch in between (optimal for GPUDirect RDMA); SYS = the path crosses the inter-socket link (the worst). The rule: the process serving on GPU0/1 must be pinned to cores 0-31/64-95 and to the memory of node 0; and if it also uses RDMA, you want the NIC that is on PIX with its GPU.

The maths that matter: a kernel tick is a GPU bubble

The figure worth internalising is not the bandwidth one, it is the jitter one. In decode, the host launches many short kernels per token. If the host thread that launches them is preempted by the scheduler (a timer tick, an IRQ, an RCU callback) for $t_{\text{stall}}$, and the GPU drains its queue in that time, a bubble appears: the GPU stops.

Put numbers on it. A typical timer tick or the handling of an IRQ costs of the order of tens of microseconds of diversion. If a decode kernel lasts around 50-100 µs and the queue carries 2-3 kernels in flight, a host stall of 50-100 µs drains the queue and the GPU sits idle until the host resumes. Multiply by the interrupt frequency of a non-isolated kernel (the default tick is 250-1000 Hz, plus network and disk IRQs): the p99/p999 tail of TTFT and of inter-token time fills up with these episodes.

$$ \text{jitter}_{p99} \approx f_{\text{interrupts}} \times t_{\text{stall}} \times \mathbb{1}[\text{GPU queue drained}] $$

The intuition: in average throughput it is barely noticeable (the bubbles average out), but in the tail, which is what an SLO measures, kernel jitter is a first-order contributor. That is why CPU isolation, which was born in low-latency trading, makes sense in LLM decode: it is the same physics, a critical thread that cannot afford the kernel stopping it.

And the NUMA cost, in parallel: a remote memory access (another floor) has latency around 1.5-2× the local one and half the bandwidth. For the pinned buffers moved by DMA on every step, and for the vLLM scheduler structures that live on the host, that penalty is paid token by token.

The three levers, one by one

Locality (NUMA): getting everything on the same floor

The goal is that the inference process using GPU0/1 has its cores, its memory and (if applicable) its NIC on NUMA node 0. Raw, without Kubernetes:

# Pin process to node 0 (cores and memory) to serve on GPU0/1
numactl --cpunodebind=0 --membind=0 \
 vllm serve meta-llama/Llama-3-70B --tensor-parallel-size 2

--membind=0 is the key: it forces all the process memory to be allocated on node 0. Without --membind, the kernel can place pages on node 1 under pressure, and you start paying for the lift without knowing it.

Page tables (hugepages): a few large pages and pinned memory

Two different things under the same umbrella. First, hugepages reduce TLB pressure for the large host buffers (pinned, KV offload). Second, pinned memory (page-locked) is what allows direct DMA with no intermediate copy. The silent trap is transparent hugepages (THP): their background compaction causes latency spikes, exactly what you do not want.

Jitter (CPU isolation): the office with “do not disturb”

Three kernel boot parameters, coordinated:

isolcpus=2-31,66-95 # take these cores out of the scheduler's balancing
nohz_full=2-31,66-95 # tickless: no timer tick if there is only 1 runnable thread
rcu_nocbs=2-31,66-95 # offload RCU callbacks to housekeeping cores

isolcpus sets the cores aside; you have to pin the inference threads there (the non-isolated cores, 0-1, are left for the system). nohz_full removes the periodic tick (it only works if there is a single runnable thread on the core). rcu_nocbs takes RCU work off those cores. And separately, IRQ affinity: moving device interrupts away from the inference cores.

The 10 knobs worth touching

Ordered by impact/frequency. Almost all of them are sysctls, kernel boot parameters or numactl. The low-level reference is Rigtorp’s low-latency guide and Red Hat’s real-time docs.

Knob 1 — `nvidia-smi topo -m`: look at the map before touching anything

Just as in the interconnect post: the map first. Which GPU is local to which NUMA node and to which cores, and what path (PIX/PHB/SYS) there is to each NIC. Without this, any pinning is blind. Half of the “inference has latency spikes” problems are processes running on the wrong socket without anyone having looked.

Knob 2 — `numactl --cpunodebind --membind`: pin to the local node

numactl --cpunodebind=0 --membind=0 <process> # cores AND memory on node 0
numactl --hardware # see nodes, distances, free memory

The --membind is what really matters: without it, the memory scatters. It is the highest-impact knob on sustained throughput.

Knob 3 — `kernel.numa_balancing=0`: turn off automatic migration

sysctl -w kernel.numa_balancing=0

The kernel’s automatic NUMA balancing migrates pages between nodes trying to “bring them closer”, but that background work causes jitter and, with the memory already pinned by knob 2, it adds nothing. On dedicated inference nodes, turn it off.

Knob 4 — Explicit hugepages (1 GB) for host buffers

# Kernel boot, for KV offload / large pinned buffers
default_hugepagesz=1G hugepagesz=1G hugepages=32

grep Huge /proc/meminfo # verify the reservation

Useful when there is host memory on the hot path (vLLM with --cpu-offload-gb, or large pinned buffers). If your deployment does not touch host memory in the hot path, explicit hugepages add little, so do not set them out of cargo cult.

Knob 5 — THP on `madvise` or `never`: avoid the compaction spikes

echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Transparent hugepages on always save TLB but their compaction triggers unpredictable latency. For tail-sensitive workloads, madvise (only where the app asks for it) or never is what is recommended. It is one of the few knobs with clear consensus: THP always is bad for latency.

Knob 6 — `isolcpus`: set the inference cores aside from the scheduler

isolcpus=2-31,66-95

Takes those cores out of the scheduler’s load balancing; the system (kernel threads, daemons) stays on the non-isolated ones. You have to pin the inference threads explicitly to the isolated cores (via numactl/taskset or, on K8s, the CPU Manager of the next post). Isolating without pinning is useless.

Knob 7 — `nohz_full` + `rcu_nocbs`: tickless and no RCU on the critical cores

nohz_full=2-31,66-95 rcu_nocbs=2-31,66-95

Removes the periodic timer tick and the RCU callbacks from the inference cores. Two warnings from practice: nohz_full only eliminates the tick if there is a single runnable thread on the core (if you pin two threads there, the tick comes back); and nohz_full is not compatible with the intel_pstate driver in some configurations, which has to be validated, not assumed.

Knob 8 — IRQ affinity: interrupts, away from the inference cores

systemctl stop irqbalance # or configure it to respect isolcpus
# move a device's IRQs to the housekeeping cores (0-1)
echo 3 > /proc/irq/<N>/smp_affinity # mask for cores 0-1

A network or disk IRQ landing on an inference core is a direct interruption of the thread feeding the GPU. Move them to the housekeeping cores. (irqbalance can respect isolcpus automatically if it is configured to.)

Knob 9 — CPU governor `performance` + C-states

cpupower frequency-set -g performance
# stop idle cores entering deep C-states (wakeup latency)
cpupower idle-set -D 0 # or limit the C-state depth

With the powersave/ondemand governor, a core that was idle takes time to ramp up its frequency: wakeup latency exactly when work arrives. performance keeps it flat out. On servers dedicated to inference, the energy saving does not compensate for the latency tail.

Knob 10 — Memory locking + `swappiness=0`

sysctl -w vm.swappiness=0 # do not evict inference pages to swap
# and in the app / container: ulimit -l unlimited (memlock) for pinned memory

An inference page that the kernel decides to swap to disk is a page fault of milliseconds on the hot path. swappiness=0 and adequate memlock limits (so the driver can pin memory) close that door. On K8s, this translates into Guaranteed QoS and memory limits, the bridge to the next post.

Summary table

#	Knob	Mechanism	What it attacks
1	`nvidia-smi topo -m`	diagnosis	see GPU–NUMA–NIC affinity
2	`numactl --cpunodebind --membind`	pinning	locality (the biggest lever)
3	`kernel.numa_balancing=0`	sysctl	jitter from page migration
4	explicit 1G hugepages	boot param	TLB on host buffers
5	THP `madvise`/`never`	sysfs	compaction spikes
6	`isolcpus`	boot param	scheduler off the critical cores
7	`nohz_full`+`rcu_nocbs`	boot param	tick + RCU jitter
8	IRQ affinity	`/proc/irq`	device interrupts
9	`performance` governor	cpupower	frequency wakeup latency
10	`swappiness=0` + memlock	sysctl/ulimit	page faults on the hot path

How it connects with the rest of the stack

With the interconnect (previous post). The NCCL host threads want cores local to their GPU; and for multi-node, the RDMA NIC must be on the PIX path with its GPU (knob 1). A GPUDirect RDMA with the NIC under the other socket loses half its advantage. NUMA and NVLink are the same story seen from the host and from the cable.

With vLLM and decode. Decode is latency-bound: the host thread feeding the kernel queue is exactly the one CPU isolation protects. And vLLM’s --cpu-offload-gb puts host memory on the hot path, where NUMA locality plus hugepages (knobs 2, 4) go from “nice” to “critical”. Continuous batching helps here too: more tokens per iteration amortise both the collective latency and the fixed cost of the host launches.

With Kubernetes (next post). Everything in this post is done by hand (numactl, taskset, boot parameters). In production it is not done by hand: the kubelet automates it with CPU Manager, Memory Manager and Topology Manager. The next post is exactly how this is declared so that every vLLM pod is born pinned to the right NUMA node, with no scripts.

With observability. The p99 spikes from jitter or from remote access are visible: in DCGM, low GPU utilisation with a full queue (bubbles); in system metrics, inter-socket traffic and CPU migrations. GPU observability with DCGM is where you diagnose a “the GPU is at 60 % and I do not know why” that many times is the host waiting.

With capacity planning. Reserving cores for the system (housekeeping) and dedicating the rest to inference changes the calculation of how many pods/replicas fit per node. Capacity planning has to count those reserved cores, not assume that all 128 vCPUs are available for serving.

Traps and things that are not what they look like

Cargo cult from low-latency trading. Many isolcpus/nohz_full guides come from HFT, where the last microsecond is squeezed out. In LLM inference, CPU isolation does help on the decode tail, but do not expect a miracle: if your bottleneck is HBM bandwidth or the interconnect, isolating cores does not move the needle. Measure first; apply it where the host is the limit.

Isolating without pinning. isolcpus takes the cores out of the scheduler, but if you do not pin the inference threads there, those cores stay empty and inference runs on the housekeeping ones, worse than before. Isolating and pinning always go together.

Forgotten --membind. Pinning cores but not memory (--cpunodebind without --membind) lets the pages scatter to the other node under pressure. Memory pinning is the half that gets forgotten most and the one that pays off most.

THP always “because it saves TLB”. It saves TLB and throws in latency spikes from compaction. For workloads with a tail SLO, it is a bad trade. madvise/never.

nohz_full with two threads on the core. Tickless only works with a single runnable thread per core. If you pin two inference threads to the same isolated core, the tick comes back and you have complicated the kernel boot for nothing.

Assuming the topology instead of reading it. Different servers wire GPUs and NICs to different sockets. nvidia-smi topo -m and numactl --hardware are the truth; the manufacturer’s diagram is indicative. Read it on every node model.

Conclusion

The GPU is the expensive resource, but in decode it spends a surprising part of its time waiting on the host: waiting for it to launch the next kernel, to sample, to move a buffer. If that host is on the wrong floor (remote NUMA), interrupted (kernel jitter) or searching a giant index (4 KB pages), the GPU is left with an empty queue and p99 blows up, without any API dashboard saying why. Of the ten knobs, the first (reading the map with nvidia-smi topo -m) and the second (pinning cores and memory to the local node with --membind) solve most of it; CPU isolation (isolcpus/nohz_full/IRQ affinity) is the second layer, the one that trims the decode tail, and it makes sense where the host is the limit, not as a ritual. The idea that reorders the intuition: inference is not “all GPU”; it is a dance between GPU and host, and the host dances better close by, uninterrupted, and with few large pages. The next post shows how Kubernetes choreographs that dance for every pod without a single hand-written script.

References

Erik Rigtorp, Low Latency Tuning Guide (isolcpus, nohz_full, IRQ affinity, THP): https://rigtorp.se/low-latency-guide/.
Red Hat, Usage, constraints and implications of isolcpus=, nohz_full= and rcu_nocbs=: https://access.redhat.com/articles/3720611.
NVIDIA, nvidia-smi topo -m and the GPU–NUMA–NIC affinity matrix (CUDA docs / Developer Forums): https://forums.developer.nvidia.com/t/nvidia-smi-topo-m-revisited/216584.
Chaim Rand, The Crucial Role of NUMA Awareness in High-Performance Deep Learning: https://chaimrand.medium.com/the-crucial-role-of-numa-awareness-in-high-performance-deep-learning-99ae3e8eb49a.
SUSE Labs, CPU Isolation – nohz_full (part 3): https://www.suse.com/c/cpu-isolation-nohz_full-part-3/.
Linux kernel, Automatic NUMA Balancing and Transparent Hugepage Support (Documentation/admin-guide): https://docs.kernel.org/admin-guide/mm/transhuge.html.

The shared bench: NVLink, NVSwitch and NCCL, the cable every token crosses in tensor parallel

Sat, 06 Jun 2026 07:00:00 +0200

This post goes one floor below the engine. In the seven-layer inference stack and in one big replica or many small ones the decision was how many GPUs and how to split the model; here we explain the cable that makes that split work, or that strangles it. It is the first of a mini-series “below the engine”: interconnect (this one) → kernel and NUMA → Kubernetes resource managers.

TL;DR

Tensor parallelism (TP) does not split a model into four chunks that run on their own. It spreads each layer across the GPUs, but after attention and after the MLP the GPUs have to add up their partial results with an all-reduce. On a Llama-70B with 80 layers, that is around 160 all-reduces per generated token. That all-reduce travels over the interconnect, so the interconnect sits on the critical path of every token, not in the background plumbing. On an HGX H100 baseboard, the 8 GPUs all talk to each other at 900 GB/s bidirectional via four NVSwitch chips; without NVSwitch/NVLink, that same traffic falls back to the CPU over PCIe and loses an order of magnitude. NCCL is the library that decides how each collective is done (ring, tree, or NVLS = NVLink-SHARP, which offloads the summation to the switch itself). And there is an asymmetry almost nobody keeps in mind: decode is latency-bound (tiny messages, 16 KB) and prefill is bandwidth-bound (huge batched activations), which is why “more NVLink bandwidth” speeds up prefill but barely touches token-by-token decode. This post explains the mechanism, gives the 10 real NCCL/driver knobs worth touching, and connects with vLLM’s custom all-reduce, disaggregated serving and GPU observability. With scepticism about which levers move the needle.

Where you are: the floor below the engine

The analogy: four mechanics and a single bench

Four mechanics assemble one and the same car engine. It is not that each builds their own engine in parallel, that would be four cars (four replicas of the model, a different strategy). Here they build a single one, at the same time, splitting the parts: one does the pistons, another the cylinder head, another the crankshaft. The problem is that the parts fit into each other: before going on, all four have to put together what they have and check that it matches. That “put together and check” happens dozens of times during the build.

There are two ways to organise the workshop:

A single large bench, everyone around it (NVSwitch). Each mechanic reaches out and hands their part directly to any other, all at once, without standing up. It is instant and simultaneous. This is NVLink + NVSwitch: the GPUs form an all-to-all where anyone talks to anyone at 900 GB/s at the same time.
Four separate workshops with a courier (PCIe via CPU). Every part a mechanic wants to hand to another goes into a box, goes down to reception (host memory, via CPU), and from there up to the destination workshop. Slower, and serialised by reception. This is what happens when there is no NVLink: inter-GPU traffic falls back to PCIe and bounces through the CPU, around 14× slower than NVLink.

The thesis of the post follows on its own: tensor parallelism only makes sense if the mechanics share the bench. As soon as the “put together and check” (the all-reduce) has to go through reception, splitting the work costs more than it saves. That is why, on a serious platform, TP does not cross the NVLink boundary: TP=4 or 8 inside the baseboard where there is NVSwitch, and above that you replicate or use pipeline, never stretch TP over PCIe or over the network. When each option makes sense is covered in one big replica or many small ones; here we explain why the cable dictates that decision.

The mechanism: what an all-reduce really is and why there are 160 per token

Tensor parallelism splits the weight matrices by columns/rows across the $N$ GPUs. Each GPU computes a portion of the layer output. But the next operation needs the complete output, so it has to be recombined. That recombination is a collective operation: an all-reduce, which adds the partial tensors of all the GPUs element by element and leaves the result identical on all of them.

In a standard transformer block there are two synchronisation points per layer:

After the output projection of attention (the o_proj that recombines the split heads).
After the second matrix of the MLP (the down_proj that recombines the split feed-forward).

$$ \text{all-reduces per token} = 2 \times L_{\text{layers}} $$

For a Llama-70B ($L = 80$): $2 \times 80 = 160$ all-reduces per generated token. Not per request, not per sequence: per token. Multiply by decode throughput and you understand why the interconnect is not background infrastructure but a hot path.

How the all-reduce is done: ring, tree, NVLS

NCCL does not have a single way of doing an all-reduce; it picks an algorithm according to topology and message size:

Ring. The GPUs form a ring; each one passes a chunk to its neighbour, adds, and rotates. It takes $2(N-1)$ steps. It is bandwidth-optimal for large messages: the cost of moving the data is $\frac{2(N-1)}{N} \times M$ bytes over the link, almost independent of $N$. The downside: $2(N-1)$ latency hops, bad for small messages.
Tree. Tree reduction: $\log N$ levels. Better latency for small messages and many nodes, worse bandwidth utilisation.
NVLS (NVLink SHARP). The Hopper trick: the summation is not done by the GPUs, it is done by the NVSwitch. The switch has reduction units; the GPUs send their tensors, the switch adds them in flight and returns the result. It takes work away from the GPUs (frees up SMs) and reduces hops. Available only with a 3rd-generation NVSwitch (NVLink4) plus Hopper or newer.

The mental rule: decode (tiny messages) wants latency → tree/LL or vLLM’s custom kernel; prefill (huge messages) wants bandwidth → ring/NVLS. That is why there is no global “optimal NCCL_ALGO”; it depends on which phase you are looking at.

The maths that matter: why decode and prefill stress the cable the other way round

Here is the asymmetry almost everyone skips. The size of the tensor that gets all-reduced in each layer is, approximately:

$$ M \approx B \times S \times h \times 2\ \text{bytes (BF16)} $$

where $B$ = batch, $S$ = tokens processed in this forward pass, $h$ = hidden size.

In decode, you generate 1 token per sequence per iteration. For a single sequence ($B \times S = 1$) and $h = 8192$ (Llama-70B):

$$ M_{\text{decode}} \approx 1 \times 8192 \times 2 = 16\ \text{KB per all-reduce} $$

16 KB is tiny. At 900 GB/s, moving 16 KB takes around 18 nanoseconds of pure transfer, but the real cost is dominated by the launch latency of the collective (synchronisation, kernel launch), of the order of single-digit microseconds. With 160 all-reduces per token:

$$ t_{\text{comms/token}} \approx 160 \times (5\text{–}10\,\mu s) \approx 0.8\text{–}1.6\ \text{ms} $$

That is the communication floor per token, independent of bandwidth. An uncomfortable and counter-intuitive implication: buying more NVLink bandwidth does not speed up token-by-token decode of a single sequence. What helps in decode is lowering the latency per collective (LL protocol, vLLM’s custom all-reduce, NVLS to remove hops) and batching (raising $B$ amortises the fixed latency over more tokens, the deep reason continuous batching exists, covered in continuous batching).

In prefill, you process the whole prompt at once: $S$ can be thousands of tokens, and with batching $B \times S$ reaches tens of thousands. There:

$$ M_{\text{prefill}} \approx 8000 \times 8192 \times 2 \approx 131\ \text{MB per all-reduce} $$

131 MB does stress the bandwidth. At 900 GB/s (NVSwitch) the ring all-reduce moves $\frac{2 \cdot 3}{4} \times 131 \approx 196$ MB effective in around 0.22 ms; over PCIe (around 64 GB/s aggregate, bouncing through the CPU) it would be around 3 ms and serialised. Here the cable is the bottleneck and NVLS/bandwidth rule.

Summary in one line: prefill is bandwidth-bound, decode is latency-bound. Any interconnect tuning that does not say which phase it helps is noise.

The hardware: NVLink 4 and NVSwitch on the HGX baseboard

On the generic reference cluster, 4×H100 SXM inside an HGX baseboard, the concrete figures:

Each H100 SXM5 has 18 NVLink 4 links, each one 50 GB/s bidirectional ⇒ 900 GB/s bidirectional aggregate per GPU. That is >14× the bandwidth of a PCIe Gen4 x16 (around 64 GB/s bidir).
On an 8-GPU HGX H100 baseboard, the 18 links of each GPU are spread against four 3rd-generation NVSwitch chips (grouped 5+4+4+5). The result is all-to-all: any GPU talks to any other at 900 GB/s simultaneously, without going through CPU or PCIe.
A 4-GPU baseboard is half a board: same principle, via NVSwitch. Design key: if your 4 H100s are connected by NVSwitch, you have real all-to-all; if they are on different boards connected by PCIe (some “4×PCIe” configurations), you do not have NVLink between all of them and TP=4 suffers. Verify it, do not assume it.

The 10 knobs worth touching

Almost all of them are NCCL environment variables (injected into the inference engine process) or driver settings. Ordered by impact/frequency in an on-premise deployment. The canonical detail is in the NCCL env vars docs.

Knob 1 — `NCCL_DEBUG` + topology dump: see what is happening before touching anything

Do not optimise blind: first confirm which topology and algorithms NCCL picked. This tells you whether it really is using NVLink or whether, silently, it fell back to PCIe/SHM, the number one failure and the most expensive one.

NCCL_DEBUG=INFO # prints topology, rings/trees built, transport chosen
NCCL_DEBUG_SUBSYS=GRAPH,TUNING,NET # narrows it down to what matters
# look in the log for: "via NVLink" / "via P2P" (good) vs "via SHM" / "via PCI" (bad)

If you see via SHM or via PCI between GPUs that should have NVLink, you have a topology problem (PCIe ACS enabled, IOMMU, GPUs on different boards) and no other knob fixes it. This is knob 1 for a reason: half of the “NVLink is slow” cases are “NVLink is not being used”.

Knob 2 — `NCCL_ALGO`: ring vs tree vs NVLS

Forces or excludes algorithms. By default NCCL picks according to size, and it usually gets it right; touch it only with a measurement in front of you.

NCCL_ALGO=NVLS,Tree,Ring # order of preference
NCCL_ALGO=^Ring # exclude Ring (^ prefix)

Rule: prefill/training (bandwidth) ⇒ Ring/NVLS; decode (latency) ⇒ Tree or, better, vLLM’s custom kernel (knob 10/stack). In most inference workloads, leaving it on auto and validating with knob 1 is the right thing; forcing it “just in case” usually makes things worse.

Knob 3 — `NCCL_PROTO`: LL / LL128 / Simple

The protocol controls the latency/bandwidth trade-off at low level:

NCCL_PROTO=Simple # maximum bandwidth, more latency (large messages)
NCCL_PROTO=LL # low-latency, half-bandwidth (tiny messages: decode)
NCCL_PROTO=LL128 # compromise, default on platforms that support it

LL (low-latency) uses flags instead of barriers and wins on the 16 KB messages of decode; Simple wins on the 131 MB of prefill. The default LL,LL128,Simple lets NCCL choose by size, which again is normally the best option.

Knob 4 — `NCCL_NVLS_ENABLE`: offload the summation to the NVSwitch

NVLink SHARP (NVLS) makes the switch do the reduction, freeing up GPU SMs:

NCCL_NVLS_ENABLE=1 # default: ON where there is an NVLink4+ NVSwitch (Hopper)

An important sceptical caveat: NVLS requires NVSwitch (3rd gen, NVLink4). On a node with NVLink through direct GPU-to-GPU bridges (no switch) or on 4×PCIe, NVLS is not available and this knob does nothing. Before “enabling it”, confirm with knob 1 that your topology has a switch. Where it does apply, its biggest advantage is freeing SMs for compute, which is relevant when comms and kernels compete (knob 5).

Knob 5 — `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`: how many SMs communication steals

Each NCCL “channel” consumes GPU SMs to move data. More channels = more collective bandwidth, but fewer SMs for the inference kernel. It is a split of a fixed resource.

NCCL_MIN_NCHANNELS=4
NCCL_MAX_NCHANNELS=16 # raising it helps prefill (bandwidth); steals SMs from decode

In decode, where the GPU is under-used on compute but tied to latency, trimming channels rarely hurts and sometimes helps; in prefill, more channels squeeze out the bandwidth. A measurement knob, not a faith knob.

Knob 6 — `NCCL_BUFFSIZE`: the buffer size per channel

NCCL_BUFFSIZE=8388608 # 8 MB (default 4 MB); larger buffers → better BW on large messages

Raising it helps bandwidth-bound prefill at the cost of memory per channel. For workloads dominated by small messages (pure decode), the default is plenty.

Knob 7 — `NCCL_P2P_LEVEL` / `NCCL_P2P_DISABLE`: guarantee P2P over NVLink

P2P is what lets one GPU read another’s memory directly over NVLink without going through the host. If it is disabled or degraded, traffic falls back to SHM/PCIe.

NCCL_P2P_LEVEL=NVL # use P2P up to the NVLink level
# NCCL_P2P_DISABLE=1 ← only as a workaround if P2P HANGS (multi-NUMA PCIe, certain Blackwell)

Watch out for the trap: NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce are recommended as a patch when vLLM hangs on PCIe-only multi-NUMA topologies. It is a robustness patch that sacrifices performance: use it if it hangs, never “by default”.

Knob 8 — GPUDirect RDMA for multi-node: `NCCL_NET_GDR_LEVEL`

When TP fits in one node, this does not apply. When you have to cross nodes (huge model, pipeline parallel between baseboards), GPUDirect RDMA lets the GPU talk to the NIC without bouncing through host memory:

NCCL_NET_GDR_LEVEL=PHB # enables GDR according to GPU–NIC proximity on the PCIe bus

Without GDR, every inter-node hop adds a host copy. With InfiniBand/RoCE plus GDR, the KV or the activations travel GPU→NIC→network→NIC→GPU. It is the basis of serious multi-node work and of mixed environments.

Knob 9 — `NCCL_IB_HCA` / `NCCL_SOCKET_IFNAME`: pin the right NIC

The most common and silent multi-node mistake: NCCL picks the management NIC (1 GbE) instead of the fabric one (InfiniBand/100 GbE). Result: collectives at a snail’s pace with no visible error.

NCCL_SOCKET_IFNAME=eth0 # control interface (bootstrap)
NCCL_IB_HCA=mlx5_0,mlx5_1 # the real InfiniBand HCAs of the fabric
NCCL_IB_GID_INDEX=3 # correct GID for RoCE v2

Pin them explicitly. “Auto” gets it right on clean clusters and fails as soon as there is more than one NIC.

Knob 10 — Driver: persistence mode, clocks and NVLink error counters

Below NCCL, the driver has levers and, above all, telemetry you have to look at:

nvidia-smi -pm 1 # persistence mode: avoids driver re-init (latency/jitter)
nvidia-smi nvlink --status # are all 18 links active and at full speed?
nvidia-smi nvlink -e # error/CRC counters per link
nvidia-smi -q -d ECC # memory errors that degrade silently

An NVLink link that negotiates at half speed or accumulates CRC errors degrades the all-reduce without raising any error: the system “works”, it is just slower. These counters are the difference between diagnosing in five minutes and chasing a ghost for days. They are integrated in DCGM (knob/stack: observability).

Summary table

#	Knob	Variable / command	Phase it helps
1	Topology diagnosis	`NCCL_DEBUG=INFO` + `SUBSYS=GRAPH`	always, first
2	Collective algorithm	`NCCL_ALGO` (NVLS/Tree/Ring)	depends on phase; auto usually wins
3	Protocol	`NCCL_PROTO` (LL/LL128/Simple)	LL=decode, Simple=prefill
4	NVLink SHARP	`NCCL_NVLS_ENABLE=1`	prefill; frees SMs (requires NVSwitch)
5	Channels (SMs)	`NCCL_MIN/MAX_NCHANNELS`	+prefill bandwidth / −SM theft in decode
6	Buffer	`NCCL_BUFFSIZE`	bandwidth-bound prefill
7	NVLink P2P	`NCCL_P2P_LEVEL=NVL`	critical; disable only if it hangs
8	GPUDirect RDMA	`NCCL_NET_GDR_LEVEL`	multi-node
9	Fabric NIC	`NCCL_IB_HCA`/`SOCKET_IFNAME`	multi-node (avoids mgmt NIC)
10	Driver + telemetry	`nvidia-smi -pm 1` / `nvlink -e`	jitter + silent diagnosis

How it connects with the rest of the stack

The interconnect is not an island; it touches almost every layer above.

With vLLM, the custom all-reduce. vLLM does not always use NCCL: for the tiny messages of decode (world_size==2 or a fully-connected NVLink topology, below a certain max_size) it uses its own all-reduce kernel that beats NCCL on latency, exactly the decode bottleneck we saw in the maths. It falls back to NCCL for large messages and for topologies without NVLink (where its custom kernel “adds little over NCCL”). The flag --disable-custom-all-reduce / VLLM_DISABLE_CUSTOM_ALL_REDUCE turns it off; it is the patch for hangs on multi-NUMA PCIe. Translation: the most effective decode latency knob is sometimes not an NCCL one, it is choosing well between vLLM’s custom kernel and NCCL.

With TP vs replicas. Everything in one big replica or many small ones rests on this: high TP is only viable inside the NVLink domain. The boundary of “TP=4 or 4 replicas at TP=1?” is drawn by the cable: crossing NVLink with TP means paying for the all-reduce at PCIe prices.

With disaggregated serving. In disaggregated prefill/decode, the KV cache generated in the prefill pool has to travel to the decode pool. That transfer is another consumer of the interconnect (NVLink intra-node, GPUDirect RDMA inter-node) and competes with the all-reduces. Designing the disaggregation without counting the cost of transferring the KV is the classic trap.

With MoE. Mixture-of-Experts models add expert parallelism: an all-to-all (not an all-reduce) that routes each token to its expert, possibly on another GPU. It is a different communication pattern and heavier on bandwidth; MoE in inference lives or dies by the same cable, with an even more demanding collective.

With GPU observability. The NVLink counters (nvidia-smi nvlink -e, TX/RX bytes per link, CRC errors) and NVSwitch utilisation are exposed via DCGM and land in Prometheus/Grafana. The question “is the interconnect healthy and saturated?” is answered there, alongside the rest of GPU observability with DCGM. A slow all-reduce shows up sooner in an NVLink error counter than in API latency.

With capacity planning. The inference sizing that assumes “TP=4 scales almost linearly” only holds inside NVLink. Outside it, scaling efficiency collapses and the capacity plan lies. The cable is a parameter of the capacity model, not a detail.

Traps and things that are not what they look like

“More NVLink bandwidth = faster decode.” False for a single sequence. Decode is latency-bound; bandwidth is barely touched by 16 KB messages. What speeds up decode is batching (amortising the fixed latency) and lowering latency per collective (LL, custom kernel, NVLS). Bandwidth rules in prefill.

“I have 4 H100s, so I have NVLink between all four.” Not necessarily. There are configurations where the GPUs are on different boards joined by PCIe, or with NVLink bridges only in pairs. Confirm it with nvidia-smi nvlink --status and knob 1 before planning TP=4. A TP=4 over P2P-over-PCIe performs far worse than the brochure says.

Forcing NCCL_ALGO/NCCL_PROTO “to go faster”. NCCL picks well by size in most cases. Forcing an algorithm without measuring usually makes one of the two phases worse. The correct sequence is: knob 1 (see what it is doing) → measure → touch only if there is evidence.

Disabling P2P/custom all-reduce by default. They are robustness patches for broken topologies (multi-NUMA PCIe, certain Blackwell). Leaving them on “for stability” on a node with healthy NVLink throws performance in the bin.

Stretching TP over the network. TP=8 crossing two nodes over InfiniBand because “there is bandwidth” ignores that the per-layer all-reduce now pays network latency ×160 per token. To cross nodes, pipeline parallel (which communicates once per micro-batch, not per layer) almost always wins. The communication pattern, not just the bandwidth, decides.

Ignoring the NVLink error counters. A degraded link does not raise an exception: the system works, it is just slow. Without watching nvlink -e and ECC, you chase a performance ghost that a counter would have pointed out in five minutes.

Conclusion

Tensor parallelism sells a simple promise, split the model, multiply the VRAM, serve models that do not fit in one GPU, but the small print is that every layer forces the GPUs to come together and add up, twice, dozens of times per token. That all-reduce is the real hidden protagonist of performance, and it lives in the cable: NVLink does it on the shared bench of the NVSwitch at 900 GB/s, or PCIe drags it through the CPU’s reception desk 14× slower. Of the ten knobs, the first one, looking with NCCL_DEBUG at what is really happening, solves half the problems, because half of the “NVLink is slow” cases are “NVLink is not being used”. The rest are refinements that only mean something if you know which phase you are in: prefill wants bandwidth (NVLS, Simple, channels, buffer), decode wants latency (LL, vLLM’s custom kernel, batching). And above all, an idea that reorders the intuition: in on-premise inference, the interconnect is not plumbing you install and forget, it is a hot path, a capacity parameter and, when it degrades silently, the root cause that no API dashboard will point out to you unless you look at the counters of the cable itself.

References

NVIDIA, NVIDIA Hopper Architecture In-Depth (NVLink 4, 900 GB/s): https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/.
NVIDIA, Introducing NVIDIA HGX H100 (4× NVSwitch, all-to-all): https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/.
NVIDIA, NCCL Environment Variables (all the knobs in this post): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html.
NVIDIA, The NVLink-Network Switch (Hot Chips 2022, NVLink SHARP): https://hc34.hotchips.org/.
vLLM, Why does vLLM use a custom all-reduce method? (discussion #6159) and custom_all_reduce.py: https://github.com/vllm-project/vllm/discussions/6159.
NVIDIA, NCCL Multi-Node NVLink Tuning Guide: https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/nccl.html.

Langfuse from the inside: the sorting centre that must not become the bottleneck it came to observe

Sat, 06 Jun 2026 06:00:00 +0200

This post closes a trilogy of the Observe layer: in LLM tracing with OpenTelemetry GenAI the SDK → Collector → backend pipeline was set up and Langfuse was treated as a black box that receives spans; in Prompt versioning with Langfuse and MLflow its prompt management layer was used. Here we open the box: what is inside Langfuse, why v3 stopped being a monolith on Postgres, and how it is operated so it holds the traffic of an inference cluster without becoming the problem.

TL;DR

Langfuse v3 (stable since December 2024) is not an application, it is six services: two containers of its own (Web and Worker) and four stateful dependencies (Postgres, ClickHouse, Redis/Valkey and an S3-compatible blob store). The key architectural change compared with v2, which was a Next.js monolith on Postgres, is the asynchronous ingestion pipeline: traces are received in batches, written immediately to S3, only a reference is queued in Redis, and a Worker ingests them into ClickHouse in the background. This decouples reception speed (limited only by Redis write latency, ~1-5 ms) from the cost of persisting and merging into the analytical database. The result: the Web container sustains hundreds of events per second without a spike blocking the client that serves the inference. But that design only performs with the right settings. This post covers the architecture, its interaction with the rest of the on-premise stack, and ten backend knobs, from batching to ClickHouse to queue sharding, from the FINAL modifier to system log table hygiene, that decide the real throughput and the storage cost. And it marks where the async design hides data loss windows worth knowing about before promising “full traceability”.

You are here: OBSERVE (and the layer that holds up the rest)

The analogy: the postal sorting centre

Picture the central mail sorting office of a large city at rush hour. Lorries loaded with sacks (batches of letters) arrive at a rate that does not stop. If the counter clerk had to open every sack, read every letter, decide its destination and file it before accepting the next lorry, the queue of lorries would go round the block in ten minutes. No serious sorting centre works like that.

What they do is decouple reception from processing:

The reception counter accepts the sack, stamps a receipt on it, leaves it in a pigeonhole in the warehouse and drops a ticket on a conveyor belt. Time per sack: seconds. The counter never blocks.
Further back, in the sorting room, a team of operators picks tickets off the belt, retrieves the sack from its pigeonhole, opens it, sorts the letters and files them in the permanent archive, ordered, indexed, searchable.

Langfuse v3 is exactly this sorting centre:

Postal centre	Langfuse v3	Function
Reception counter	Web container (ingestion endpoint)	Accepts event batches, gives an immediate receipt (HTTP 207)
Pigeonhole warehouse	S3 / Blob store (MinIO on-prem)	Stores the raw sack (the full event)
Ticket on the belt	Redis / Valkey (BullMQ queue)	Only the reference to the object in S3, not the content
Sorting room	Worker container	Takes tickets, reads S3, transforms and files
Indexed permanent archive	ClickHouse (OLAP)	Traces, observations and scores, queryable by project+time
Administrative register	Postgres (OLTP)	Users, projects, API keys, prompts, datasets, config

The thesis of the whole post follows from this analogy: the value of Langfuse lies in the counter never blocking the client that serves the inference. An observability tool that adds latency or outages to the token-serving path is worse than having no observability, because it degrades precisely the system it set out to look after. The whole of the v3 design, and every knob in this post, exists to keep that promise under load.

The mechanism itself: six services, two planes

Langfuse v3 separates two planes that in v2 were fused together:

Ingestion and query plane (the two containers of its own, stateless, horizontally scalable): Web and Worker.
State plane (four dependencies, each with its own load profile): Postgres (transactional OLTP), ClickHouse (analytical OLAP), Redis/Valkey (queue + cache), Blob store (raw objects).

What to take away from this diagram:

Web and Worker are interchangeable and stateless. They store nothing locally. You can run 1 or 20 replicas of each; the state lives in the four dependencies. This is what allows scaling by load without choreography.
Redis never carries the event content, only the reference to the object in S3. That is why Redis holds the spike: a Redis write is ~1-5 ms and moves bytes, not kilobytes. The bottleneck of the Web container is, literally, Redis write speed.
Postgres and ClickHouse have opposite profiles. Postgres is OLTP: many small transactional reads and writes (is this API key valid? which version does the production label point to?). ClickHouse is OLAP: few enormous batch writes and analytical queries over billions of rows (give me the p95 TTFT of project X over the last 7 days). Putting traces in Postgres, which is what v2 did, works until it does not: at production volume, Postgres drowns in a workload it was not designed for. That was the reason for the redesign.

The ingestion flow step by step (and the maths of the decoupling)

The heart of the design is the ingestion route. Seen in detail, a POST /api/public/ingestion request with a batch of events does this:

The mathematical point is the early ACK. The latency the client perceives when sending traces is:

$$ t_{\text{client}} = t_{\text{S3 write}} + t_{\text{Redis enqueue}} \approx 10\text{–}40\,\text{ms} $$

whereas the real cost of persisting, reading S3, transforming, merging against the previous version, inserting into ClickHouse, letting the background merges compact, happens off that path, in the Worker, and can take hundreds of ms or seconds without the client caring. The decoupling turns a system whose throughput would be limited by ClickHouse speed into one limited by Redis speed. And Redis, on modest hardware, sustains on the order of 50,000 operations/second.

This has an important sizing consequence. If your inference load generates $E$ events/second (a chat with RAG + 2 tool calls easily produces 6-10 spans = events per request), the Web container absorbs them while $E \ll 50{,}000$. The Worker, by contrast, scales with the cost of processing: that is the component to watch and replicate, and the first knob of the post.

Honest scepticism. The early ACK has a flip side: between the HTTP 207 and persistence in ClickHouse there is a potential loss window. If the event is in S3 and the reference in Redis, and Redis goes down without persistence (AOF/RDB) before the Worker processes it, the reference is lost; the data is still in S3 but nobody claims it any more. More subtly: the Worker buffers writes to ClickHouse in memory and flushes them in batches; a Worker crash with a full buffer loses that batch. There is a reported bug where the ClickhouseWriter discards rows after exhausting flush retries with no dead-letter queue. For observability this is usually tolerable (losing 0.01 % of the traces breaks nothing). For regulatory auditing, where the trace is evidence, it is not, and Langfuse is better treated as “best-effort” and not as a ledger. We will come back to this at the close.

Interaction with the rest of the stack: Langfuse in the example 4×H100 cluster

Langfuse does not live in isolation. In the seven-layer stack it occupies the LLM-aware observability layer, and it relates to almost every other one. On the generic reference cluster we use throughout the blog, 4×H100 SXM 80 GB (320 GB aggregate VRAM), NVLink, 640 GB of system RAM, NVMe-oF, 25/100 GbE network, the telemetry flow looks like this:

Three ideas from this topology:

Langfuse receives from the OTel Collector, not from the application directly (in the recommended pattern). The app SDK or vLLM emit spans with the gen_ai.* semantic conventions; the Collector does batch, tail-sampling (preserving 100 % of errors and high latencies, sampling the rest) and enriches with its own attributes (tenant_id, priority_tier); and it distributes: LLM traces go to Langfuse, infrastructure spans to Tempo, metrics (GPU DCGM, vLLM metrics) to Prometheus. Langfuse is one more exporter, not the only destination. This is covered in detail in the OTel tracing post.
Langfuse runs off the GPUs. It is a consumer of CPU, RAM, disk and network (ClickHouse wants memory, MinIO wants disk, Redis wants CPU for networking) but it does not touch VRAM. On the 4×H100 cluster, Langfuse lives on a CPU node (or on the GPU nodes but with nodeSelector/taints keeping it away from the vLLM pods). Mixing ClickHouse with vLLM on the same node without resource limits is asking for an ingestion spike to steal memory bandwidth from inference. Isolation by design.
The telemetry route is “cold” and the data route is “hot”. The data plane (left) serves tokens with a millisecond latency budget; the telemetry plane (right) tolerates seconds. The early ingestion ACK is what keeps these two clocks apart: the app does not wait for Langfuse to file anything before returning the response to the user.

The 10 backend knobs that move the needle most

These are, in approximate order of impact/frequency, the settings that decide whether your self-hosted Langfuse ingests 50 events/s or 5,000, and whether your disk grows sustainably or explodes in three weeks. They are all environment variables or config injected into the Web and Worker containers (except the ClickHouse ones, which go in its server-side config). The canonical detail is in the Langfuse scaling doc.

Knob 1 — Scale the Worker by load (the first lever, always)

The Worker is the component that saturates first, because it does the expensive work: reading S3, transforming, merging, inserting into ClickHouse. The Langfuse operational rule is simple: a 2-CPU Worker container above 50 % CPU use is saturated; add replicas. Better than CPU, the Worker publishes via statsd the metric langfuse.queue.ingestion.length (ingestion queue length), which is the direct signal for autoscaling: if the queue grows without draining, there are not enough Workers.

# The ideal autoscaler watches queue depth, not just CPU.
# (KEDA ScaledObject over the statsd metric → Prometheus)
triggers:
 - type: prometheus
 metadata:
 query: langfuse_queue_ingestion_length
 threshold: "10000" # if the queue goes past 10k refs, scale

In AWS deployments there is ENABLE_AWS_CLOUDWATCH_METRIC_PUBLISHING=true to push these metrics to CloudWatch. On-premise, the path is statsd → Prometheus → KEDA, slotting into the autoscaling on Kubernetes with KEDA we already covered for vLLM. Always start here: most “Langfuse is slow” problems are simply insufficient Workers, not fine tuning.

Knob 2 — Separate the ingestion deployment from the UI one

When ingestion is heavily loaded, UI queries and the public API become slow because they share the same Web container. The solution is to split langfuse-web into two identical deployments and route by path: everything under /api/public/ingestion*, /api/public/media* and /api/public/otel* goes to the ingestion deployment; the rest (UI, read API) to the interface one.

# Ingress / gateway rule
location ~ ^/api/public/(ingestion|media|otel) {
proxy_pass http://langfuse-web-ingest; # replicas dedicated to writing
}
location / {
proxy_pass http://langfuse-web-ui; # replicas dedicated to reading
}

It is the same idea as the read/write separation of any system with mixed loads: a storm of writes should not starve whoever is trying to look at the dashboard right in the middle of the incident, which is precisely when they need it most.

Knob 3 — Batching writes to ClickHouse (interval + batch size)

ClickHouse hates small, frequent inserts: every INSERT creates a part on disk that then has to be merged, and thousands of tiny inserts generate thousands of parts and a storm of background merges that saturates the disk. The defence is to accumulate in an in-memory buffer in the Worker and flush in large batches:

# Worker: fewer flushes, larger batches → fewer parts, fewer merges
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=1000 # raise e.g. to 2000-5000
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_BATCH_SIZE=10000 # raise if there is throughput

Raising the interval and the batch size reduces the flush frequency and improves sustained throughput. The trade-off is direct and has to be understood: larger, less frequent batches mean more data in the Worker’s volatile buffer, that is, a bigger loss window if the Worker goes down (a knob coupled to the scepticism at the close). Langfuse also uses ClickHouse async_insert, which accumulates server-side before confirming; it adds another layer of buffering to keep in mind.

Knob 4 — Skip the ClickHouse read-before-write in ingestion

By default, when ingesting an event the Worker reads the existing event from ClickHouse and merges it with the incoming one (necessary when legacy SDKs send partial events: a start, then an end, then an update of the same observation). That per-event read loads ClickHouse on the write path and limits total throughput.

If your projects are not migrated from an old version, because the full history already lives in S3, you can disable that read:

# Date earlier than the creation of your first project
LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE=2025-01-01

With modern Langfuse SDKs or with ingestion via OpenTelemetry, this does not affect you negatively and removes one read per event. A warning from the doc itself: if you combine this with aggressive deletion (lifecycle) rules in S3 plus late event updates, you can generate duplicates in the history. Know that before enabling it.

Knob 5 — Write concurrency to S3/Blob storage

In high-throughput scenarios, the S3 client can exhaust its sockets and start queueing and throttling writes. The symptom is unmistakable in the logs of the Web container that handles ingestion:

@smithy/node-http-handler:WARN - socket usage at capacity=150
and 387 additional requests are enqueued.

…accompanied by a rise in memory in that container (queued requests pile up in RAM). The cure is to raise the concurrent write limit from its default of 50:

LANGFUSE_S3_CONCURRENT_WRITES=100 # raise gradually from 50

Each additional socket has a small memory cost, so the official advice is to raise it gradually while observing the behaviour, not to jump to 1000 in one go.

Knob 6 — Redis queue sharding + concurrency per shard

If Redis goes past 90 % CPU, first the obvious: an instance with at least 4 CPUs (so Redis spreads networking and background tasks across different cores) and Redis Cluster mode enabled. If CPU still does not come down, you can shard the queues Langfuse uses:

# Advanced: only if Redis is genuinely drowning and you already did the above
LANGFUSE_INGESTION_QUEUE_SHARD_COUNT=6 # ~2-3× the number of Redis cluster shards
LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT=6
# Concurrency counts PER SHARD; target ~20 per worker
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=3 # 6 shards × ~3 ≈ 18
LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY=3

Two traps the doc underlines and that are worth tattooing: once you shard, do not reduce the number of shards (it breaks the distribution); and concurrency counts per shard, not globally, so if you have 10 shards and want concurrency 20 per worker, set 2, not 20. It is an advanced knob: most on-premise deployments never need it.

Knob 7 — The `FINAL` modifier for OTel-only projects

Langfuse stores observations in a ClickHouse ReplacingMergeTree and, by default, adds the FINAL modifier to API queries so the last version of each row wins at read time. FINAL is necessary when ingestion produces several versions of the same observation (legacy SDKs with their start/end/update events), but it adds merge work to every read and slows it down.

Projects that ingest exclusively over OpenTelemetry write each observation as a single immutable row, so FINAL is redundant for them:

# Recommended in mixed deployments: per-project, flagged in Redis with a 24h TTL
LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true

# Only if ALL projects are OTel-only: global, no Redis lookup
LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL=true

Since on the example cluster the instrumentation is 100 % OTel (gen_ai.* via Collector), this knob is free money in dashboard read latency. Careful with the global version: do not enable it if some project is still using legacy ingestion, or reads may return duplicate or stale rows.

Knob 8 — Separate analytical reads from the write path (compute-compute)

Heavy dashboard queries (percentiles over millions of spans) compete with ingestion inserts and with background merges on the same ClickHouse. If your deployment supports compute-compute separation (ClickHouse Cloud or BYOC), you can route reads to a read-only compute group:

CLICKHOUSE_URL=http://clickhouse-primary:8123 # writes, migrations, ingestion
CLICKHOUSE_READ_ONLY_URL=http://clickhouse-reader:8123 # UI + public API reads

Critical nuance for on-premise, and here it is time to be sceptical about the usefulness of this knob in our context: on a single-node ClickHouse or on a self-managed cluster without compute separation, this variable adds nothing, because the read endpoint would be the same as the write one. It is a knob for cloud architectures with storage separated from compute. On an on-premise 4×H100 cluster with ClickHouse on one node, the real alternative is to scale ClickHouse vertically (the doc recommends ≥16 GiB of RAM for large deployments; ClickHouse scales vertically well) and to make sure every query filters by projectId and time, which is how the tables are indexed. Without a time filter, even the fattest ClickHouse suffers.

Knob 9 — Data retention: TTL in ClickHouse + lifecycle in S3

Disk is the cost that grows on its own. LLM traces carry whole inputs and outputs (sometimes prompts of tens of KB), and ClickHouse also accumulates its own system tables. The first-order lever is a retention policy that nightly deletes traces, observations, scores and media older than N days, coordinating ClickHouse and blob storage. Where the retention feature is not available, it is done by hand:

-- ClickHouse: TTL over the tracing tables
ALTER TABLE traces MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE observations MODIFY TTL toDateTime(start_time) + INTERVAL 90 DAY;
ALTER TABLE scores MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE event_log MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY;

# S3/MinIO: lifecycle rule, e.g. 30 days for the raw event bucket
# WATCH OUT! Do NOT apply retention to the MEDIA bucket:
# - it breaks the files referenced in traces
# - it breaks future uploads (state is tracked by hash in Postgres)

Two operational parameters that avoid nasty surprises on large deletions:

LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS=600000 # default 10 min; raise it if deletions time out
# ClickHouse 25.7+: less mutation pressure on bulk deletions
CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE=lightweight_update
CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE=true

The mental rule: short retention for raw events (S3, 30 days is usually enough, they are recoverable/recomputable), retention by business value for the ClickHouse tables (90 days, 180, whatever compliance asks for), and never touch the media bucket with blind lifecycle rules.

Knob 10 — Hygiene of the ClickHouse system log tables (the silent disk killer)

This is the knob nobody configures and that fills the disk without showing up in any Langfuse metric, because it is not Langfuse data: they are the system tables of ClickHouse itself (trace_log, text_log, opentelemetry_span_log, asynchronous_metric_log, metric_log, latency_log). By default they have no TTL, and the query profiler writes to system.trace_log continuously. On a ClickHouse with traffic, these tables can dominate disk usage while you look for the problem in your traces. Langfuse does not read from them, so they can be trimmed without fear. Two options:

<!-- Option A — disable the ones Langfuse never reads
 (file in /etc/clickhouse-server/config.d/) -->
<clickhouse>
 <trace_log remove="1"/>
 <text_log remove="1"/>
 <opentelemetry_span_log remove="1"/>
 <asynchronous_metric_log remove="1"/>
 <metric_log remove="1"/>
 <latency_log remove="1"/>
</clickhouse>
<!-- Keep query_log, part_log and error_log: useful for debugging and small -->

-- Option B — aggressive TTL + turn off the profiler, if you want to keep them for debugging
-- (in config: query_profiler_real_time_period_ns = 0)
SET max_table_size_to_drop = 0;
TRUNCATE TABLE system.trace_log;
ALTER TABLE system.trace_log MODIFY TTL event_date + INTERVAL 7 DAY;
-- repeat for each log table to cap

To identify which table is eating the disk, the golden query:

SELECT table, formatReadableSize(sum(bytes)) AS size, sum(rows) AS rows
FROM system.parts WHERE active GROUP BY table ORDER BY sum(bytes) DESC;

If you take only one knob from this post to your first real deployment, make it this one: the difference between a ClickHouse that grows 2 GB/day of useful data and one that grows 20 GB/day of system logs nobody looks at.

Summary table of the 10 knobs

#	Knob	Variable / action	When
1	Scale Worker	replicas by CPU>50 % / `langfuse.queue.ingestion.length`	always, first
2	Separate ingestion/UI	route `/ingestion`,`/media`,`/otel*` to a dedicated replica	slow UI under load
3	Batching to ClickHouse	`LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS` / `_BATCH_SIZE`	high throughput
4	Skip CH read-before-write	`LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE`	non-migrated projects
5	S3 concurrency	`LANGFUSE_S3_CONCURRENT_WRITES` (def. 50)	“socket usage at capacity”
6	Redis queue sharding	`LANGFUSE__QUEUE_SHARD_COUNT` + `_CONCURRENCY` (per shard)	Redis CPU >90 %
7	Remove `FINAL` (OTel)	`LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true`	100 % OTel instrumentation
8	CH read/write split	`CLICKHOUSE_READ_ONLY_URL` (cloud/BYOC only)	compute-compute available
9	Retention + TTL	TTL in CH + S3 lifecycle + `LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS`	always (disk cost)
10	CH system log hygiene	`<trace_log remove="1"/>` or aggressive TTL	always (hidden disk)

How to maximise Langfuse on the example 4×H100 cluster

With the architecture and the knobs clear, this is a concrete sizing to get the most out of Langfuse on the generic reference cluster (4×H100 SXM, 320 GB VRAM, 640 GB RAM, NVMe-oF, 25/100 GbE), without stealing a single GB of VRAM from inference.

Component distribution

Langfuse is 100 % CPU/RAM/disk/network load, so its natural place is off the GPU nodes or, if it cohabits, with taints/nodeSelector confining it away from the vLLM pods. Suggested distribution:

nodo-cpu-01 (control + observability, no GPU)
├── langfuse-web-ingest ×3 (2 CPU / 4 GiB each) ← ingestion, scales with load
├── langfuse-web-ui ×2 (2 CPU / 4 GiB each) ← dashboard/read API
├── langfuse-worker ×4 (2 CPU / 4 GiB each) ← the one that scales most
├── redis/valkey ×1 (4 CPU / 4 GiB, cluster mode)
└── postgres ×1 (2 CPU / 8 GiB, replica for HA)
nodo-storage-01 (heavy state, local NVMe)
├── clickhouse ×1 (8 CPU / 32 GiB / NVMe) ← ≥16 GiB is the minimum; 32 is comfortable
└── minio (S3) ×1 (4 CPU / 8 GiB / HDD+NVMe cache)
nodo-gpu-01..02 (4×H100 SXM each) → inference ONLY
└── vLLM, embeddings, reranker, guardrails (emit spans, do not host Langfuse)

Sizing by real load

Let us put numbers on an example load. Suppose the cluster serving 300 requests/second of chat-with-RAG, where each request generates on the order of 8 spans (request, retrieval, rerank, 2× tool, guardrail in, llm, guardrail out):

$$ E = 300\,\tfrac{\text{req}}{\text{s}} \times 8\,\tfrac{\text{spans}}{\text{req}} = 2,400\ \text{events/s} $$

Against the Redis ceiling (~50,000 ops/s), $E = 2{,}400$ leaves the reception counter at ~5 % of its capacity: enormous headroom. The component to watch is the Worker. With a target of ~20 concurrency per Worker and batches of 10,000 events every ~1-2 s, 4 Workers drain 2,400 ev/s with margin; the langfuse.queue.ingestion.length metric should stay flat near zero. If it grows, knob 1 (more Workers) is the answer before any fine tuning.

Tail-sampling is the multiplier that changes the economics. If the Collector preserves 100 % of errors/high latencies but samples normal traffic at, say, 10 %, the 2,400 ev/s you store in ClickHouse drop to ~240-300 effective ev/s without losing the signal that matters. The rule: sample in the Collector, not in Langfuse. Langfuse should receive, already filtered, what deserves to be persisted. This is developed in the OTel tracing post; here it is enough to note that upstream sampling is, in effect, the knob 0 that multiplies all the others.

Storage estimate

An LLM observation with full input+output weighs, compressed in ClickHouse, on the order of 1-3 KB (ClickHouse compresses text very well, 5-10×). With sampling at 10 % over 2,400 ev/s:

$$ 240\,\tfrac{\text{ev}}{\text{s}} \times 2\,\text{KB} \times 86,400\,\tfrac{\text{s}}{\text{day}} \approx 41\ \text{GB/day (raw)} \;\xrightarrow{\text{compression}}\; \sim 5\text{–}8\ \text{GB/day in CH} $$

At 90 days of retention (knob 9), the permanent archive settles at around 500-700 GB in ClickHouse, comfortable on the NVMe of the storage node, plus the raw events in MinIO with a 30-day lifecycle. Without the system log hygiene (knob 10), add easily as much again in rubbish nobody queries. The two disk knobs together are the difference between planning storage once a year and fighting a full disk every month.

“Maximum use” checklist

Sampling in the Collector (tail: 100 % errors + N % normal), before touching anything in Langfuse.
Workers scaled by queue length via KEDA (knob 1), not fixed.
Ingestion separated from UI (knob 2) so the dashboard responds during incidents.
SKIP_FINAL_FOR_OTEL_PROJECTS enabled (knob 7) because the instrumentation is 100 % OTel.
Generous CH batching (knob 3) tuned to the throughput, accepting the loss window.
Retention + TTL + system log hygiene (knobs 9 and 10) configured on day 1, not when the disk screams.
ClickHouse with ≥16 GiB and every query filtering by projectId+time (knob 8 in its on-premise version: vertical scaling).
Langfuse isolated from the GPUs by taints/nodeSelector: not a MB of VRAM, no memory bandwidth contention with vLLM.

Traps and things that are not what they look like

“Langfuse guarantees me full traceability.” No: the design is high-performance best-effort, not a ledger. Between the HTTP 207 and the row in ClickHouse there are volatile buffers (Redis without hard persistence, the Worker’s in-memory buffer, ClickHouse’s server-side async_insert). There is a known bug where the writer discards rows with no dead-letter queue after exhausting retries. For operational observability, losing 0.01 % of spans is irrelevant. For ENS/EU AI Act audit evidence, where the trace is the proof, Langfuse should not be the only record; the regulatory audit log needs durability guarantees this pipeline does not promise. A distinction covered in the technical controls for ENS/42001/EU AI Act.

Raising ClickHouse batching “to go faster” and nothing else. Knob 3 improves throughput at the cost of enlarging the loss window and the latency of data appearing in the dashboard. Batches of 50,000 every 10 s perform wonderfully… until the Worker restarts with 50,000 events in the buffer. Tune with awareness of the trade-off, not by blind maximisation.

Putting ClickHouse on the same node as vLLM with no limits. ClickHouse is voracious with memory bandwidth during merges. Sharing a node with vLLM without resources.limits or NUMA isolation means an ingestion spike can degrade inference TTFT, exactly the original sin this whole architecture wanted to avoid. Isolate.

Forgetting the time filter in your own queries. The ClickHouse tables are indexed by projectId and time. A custom dashboard or an API query without a time filter scans the whole history and takes performance down for everyone. It is not Langfuse “being slow”: it is a badly written query.

Applying lifecycle to the media bucket. It breaks the files referenced in traces and blocks future uploads (state is tracked by hash in Postgres). The media bucket is managed only with the Langfuse retention feature, never with blind S3 rules.

Treating queue sharding as a routine optimisation. It is an advanced knob for a genuinely drowning Redis, irreversible (do not reduce shards) and with per-shard concurrency semantics that are easy to misread. In the vast majority of on-premise deployments it is not needed; if you enable it “just in case”, you complicate your life without gaining anything.

Conclusion

Langfuse v3 solved the structural problem of LLM observability, that the observer should not suffocate the observed, by moving from a monolith on Postgres to a six-service sorting centre with asynchronous ingestion. That design is what allows a cluster serving thousands of tokens per second to be instrumented end to end without the app ever waiting for a trace to be filed. But the design is a necessary condition, not a sufficient one: it performs if the right levers are tuned. Of the ten knobs, three decide almost everything in a typical on-premise deployment, scaling Workers by queue length (1), retention + TTL (9), and system log hygiene (10); the rest are refinements that show up when the load bites. And above them all lives knob 0, which is not a Langfuse one: sampling in the Collector, which decides how much reaches the pipeline before any internal setting matters. Maximising Langfuse on the 4×H100 cluster is not about squeezing its peak throughput: it is about putting it off the GPUs, feeding it already-sampled traffic, sizing the Worker by the queue, and configuring retention on day one, so that the tool that came to tell the story does not end up being the chapter about the incident.

References

Langfuse, Scaling Langfuse Deployments (doc oficial de sizing y todos los env vars de este post): https://langfuse.com/self-hosting/configuration/scaling.
Langfuse, Self-host Langfuse y Configuration via Environment Variables: https://langfuse.com/self-hosting · https://langfuse.com/self-hosting/configuration.
Langfuse, ClickHouse (self-hosted): https://langfuse.com/self-hosting/deployment/infrastructure/clickhouse.
Langfuse, From Zero to Scale: Langfuse’s Infrastructure Evolution (el porqué del rediseño v2→v3): https://langfuse.com/blog/2024-12-langfuse-v3-infrastructure-evolution.
Langfuse, Migrate v2 to v3 (self-hosted): https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v2-to-v3.
ClickHouse, Langfuse and ClickHouse: A new data stack for modern LLM applications: https://clickhouse.com/blog/langfuse-and-clickhouse-a-new-data-stack-for-modern-llm-applications.
Langfuse, issue #13468 — ClickhouseWriter drops rows after max flush attempts with no DLQ (la ventana de pérdida documentada): https://github.com/langfuse/langfuse/issues/13468.
ClickHouse, TTL for tables and columns: https://clickhouse.com/docs/guides/developer/ttl.
OpenTelemetry, Semantic Conventions for Generative AI (gen_ai.*): https://opentelemetry.io/docs/specs/semconv/gen-ai/.

Mixed NVIDIA + Intel environments for LLM inference: from the central H100 cluster to the NUC in the branch office

Tue, 02 Jun 2026 04:30:00 +0200

This post complements those on Capacity planning for on-premise LLM inference (which assumed a pure NVIDIA cluster), Seven layers of the stack (which also did not get into hardware heterogeneity) and The LLM inference router (where capability routing makes full sense once there is mixed hardware). It is the missing piece for talking about “hardware sovereignty” without reducing it to “which vendor do I pick”.

TL;DR

A production LLM inference cluster in 2026 can stop being monolithically NVIDIA if it accepts heterogeneity as an architectural decision. The motivation is not theory but three measurable operational advantages. (1) Cost: an Intel Xeon 6 with AMX (Advanced Matrix Extensions) delivers 7B INT4 at ~80 tok/s while serving embeddings and reranking at a fraction of the cost of dedicating an H100 to that task; the capacity planning adds up better with an Intel CPU handling the cheap work and an NVIDIA H100 the big LLM. (2) Sovereignty and supply chain diversification: NVIDIA holds roughly 94 % of the AI accelerator market (November 2025), single-vendor dependency with all its risks; Intel manufactures in Europe (Leixlip operational, Magdeburg planned) against a design-only NVIDIA with TSMC as foundry, which for a Spanish or European organisation subject to ENS / NIS2 / EU AI Act is a real hedging argument. (3) Edge: an Intel NUC with a Lunar Lake CPU (48 TOPS NPU) or Panther Lake (50 TOPS NPU + 120 TOPS Xe3 = 180 TOPS platform) runs 7B INT4 models at usable speed, which opens up the “branch office with local inference + central DC for complex cases” pattern. Relevant Intel hardware in June 2026: Intel Gaudi 3 (128 GB HBM2e, 1,835 TFLOPS BF16/FP8, 3.67 TB/s; a direct competitor to the H100, Intel claims +20 % on Llama 2 70B but Signal65 published H200 9× over Gaudi 3 on Llama 3.1 405B, you have to cite both; Falcon Shores cancelled in January 2025, Jaguar Shores 2026 as the restart bet, Gaudi 4 confirmed as never happening); Intel Xeon 6 with AMX (up to 288 E-cores on Sierra Forest or 86 P-cores on Granite Rapids, 1,024 FLOPS BF16/cycle/core with AMX, Intel claims 2.7× tok/s vs EPYC 9965 on the vLLM CPU backend); Intel Arc Pro B60 (Battlemage, 24 GB GDDR6, 456 GB/s, 197 TOPS INT8, launched September 2025, with a dual-GPU 48 GB variant and a “Battlematrix” rack with 8× = 192 GB VRAM); Intel NUC with NPU (Lunar Lake 48 TOPS, Arrow Lake similar, Panther Lake 50 TOPS at CES 2026; realistic for 7-13B INT4, not for the 30-70B Intel claims in its marketing). Software: OpenVINO 2025.3 with the GenAI API and vLLM-OpenVINO; IPEX-LLM with integrations into llama.cpp, vLLM, HF, LangChain; vLLM CPU backend with AMX; llama.cpp SYCL (better than Vulkan on Arc). Four canonical patterns: embeddings + reranker on Intel alongside the LLM on NVIDIA; guardrails + PII redaction on a near-edge NUC; speculative drafter on a NUC near the user and target on the H100; NUC dev workstations. Unified observability via DCGM + habana-metric-exporter + intel-gpu-exporter + Intel PCM federated in Prometheus. Pitfalls: tokenizer mismatch between engines, edge↔central round-trip latency, FP8 Hopper ≠ INT8 AMX in quality, version synchronisation. Applied to a generic cluster: central DC 4×H100 SXM + Xeon 6 AMX sidecar + 6-12 Intel NUCs in branch offices. Critical disclaimer: as of June 2026 there are no publicly verifiable cases of a mixed NVIDIA + Intel deployment in European banking or government; the pattern is emergent architecture and worth recommending, not established practice with an industrial track record.

You are here: DEPLOY (with heterogeneity as a decision)

The analogy: the factory with several different machines

A serious factory has several machines with different purposes, not one universal machine. A 200-tonne hydraulic press for heavy stamping; a bench lathe for turned parts; a 3D printer for quick prototypes; a pick-and-place robot for SMD. Each machine does what it does better than the others in its niche, and the plant manager sizes the mix according to the real product portfolio, not according to fashion. Buying three hydraulic presses because “they are the most impressive” when 60 % of the work is turned parts wastes capital: the lathe is cheaper, faster in its niche and frees the press for what genuinely needs it.

An LLM inference cluster with an NVIDIA H100 dedicated to computing embeddings for a RAG corpus is using a hydraulic press to drill bolts. The H100 is magnificent for a 70B LLM in BF16 with concurrency of 40+; for embeddings of an 800-token document with bge-m3, what you need is an Intel Xeon 6 with AMX at a fraction of the cost and power draw. A cluster that wants to serve light guardrails (Llama Guard 4 8B) on every request, with a 50 ms budget, does not need that guardrail on an H100 either: an Intel NUC with a 48 TOPS NPU covers the case with room to spare.

The heterogeneous factory is not theoretical elegance: it is maximising the useful utilisation of fixed capital. Nor is the heterogeneous LLM inference cluster.

Three operational reasons for heterogeneity

Reason 1 — cost

An H100 SXM 80 GB running 24/7 draws ~700 W (real measurement at the wall ~697 W with vLLM Llama 3.1 405B batch=4) and represents between 25,000 € and 35,000 € of amortised hardware. An Intel Xeon 6 with AMX (Granite Rapids 86 cores or Sierra Forest 288 E-cores) draws 350-500 W for the socket and costs a fraction. The operational split: the H100 is reserved for the big LLM (Llama 70B BF16 or FP8, where its HBM3 and FP8 tensor cores are worth their weight); the Xeon AMX absorbs embeddings (bge-m3, e5-large), reranking (bge-reranker-v2-m3), small models (Llama 3.2 1B / 3B INT4) and offline batch processing. It is the same logic as the capacity planning taken one step further: instead of budgeting KV cache VRAM only on the H100, budget each workload on the silicon where its arithmetic intensity fits best.

Reason 2 — sovereignty and supply chain diversification

As of November 2025, NVIDIA holds approximately 94 % of the AI accelerator market. That concentration is risk. For an organisation subject to ENS / NIS2 / EU AI Act, depending on a single supplier whose foundry is concentrated in Taiwan (TSMC) introduces supply chain vulnerabilities that recent regulations (NIS2, supply chain provisions) are starting to require you to document and mitigate. Intel diversifies: it has its own fabs in Europe (Leixlip operational in Ireland; Magdeburg planned in Germany, with EU Chips Act funding), which for a Spanish or European public sector client is a real contractual argument, not marketing.

Mandatory disclaimer: the Intel roadmap after Falcon Shores is unstable. Intel cancelled Falcon Shores in January 2025 and relegated Gaudi 4 to “will never exist”; the restart bet is Jaguar Shores in 2026 as a rack-scale platform, still with no confirmed public specs. Diversification is strategically correct, but assuming continuity of the Intel roadmap at NVIDIA’s level in 2026 would be naive. The operational strategy: Intel for workloads where lock-in is lower (CPU for embeddings, NUC for light edge, replaceable by AMD/Apple/SiFive if Intel pivots again), NVIDIA for the big LLM where the maturity of the software stack still has no rival.

Reason 3 — edge

The “everything travels to the central DC” pattern breaks in three cases: latency (branch office 100+ ms away from the DC, unacceptable for chat), data sovereignty (prompts with personal or classified data that must not leave the local perimeter), and offline operation (branch office with intermittent connectivity). The Intel NUC with a modern CPU (Lunar Lake / Arrow Lake / Panther Lake) brings 48-50 TOPS NPU + 100-180 TOPS Xe2/Xe3 iGPU in a box of 0.5-1.5 L volume drawing 30-65 W. 7B INT4 models run at usable speed; with more aggressive quantisation (Q3_K) a Llama 13B fits. For branch offices with RAG over a local corpus + a 7B LLM + guardrails, the NUC is perfect.

Relevant Intel hardware (June 2026)

Part	Memory	Key performance	Launch	Status
Intel Gaudi 3	128 GB HBM2e, 3.67 TB/s	1,835 TFLOPS BF16/FP8; 1,200 GB/s networking	Apr 2024	Active; successor Jaguar Shores 2026 (not Gaudi 4)
Intel Xeon 6 (Granite Rapids)	DDR5 + MRDIMM	86 P-cores, AMX 1,024 FLOPS BF16/cycle/core	2024-2025	Active
Intel Xeon 6 (Sierra Forest)	DDR5	288 E-cores	2024	Active
Intel Arc Pro B60 (Battlemage)	24 GB GDDR6, 456 GB/s	197 TOPS INT8; 12.28 TFLOPS FP32	Sep 2025	Active; dual 48 GB variant, rack 8× = 192 GB
Intel Data Center GPU Max	128 GB HBM	successor to Ponte Vecchio	discontinued	Discontinued Jan 2026
Intel NUC (Lunar Lake)	DDR5x	NPU 48 TOPS + Xe2 67 TOPS = 120 TOPS platform	2024	Active
Intel NUC (Arrow Lake)	DDR5	NPU 13 TOPS + Xe iGPU	2024	Active (less NPU than Lunar)
Intel NUC (Panther Lake)	DDR5x	NPU 50 TOPS + Xe3 120 TOPS = 180 TOPS platform	CES Jan 2026	Rolling out

Intel Gaudi 3 — the critical note on the marketing

Intel publishes that Gaudi 3 delivers +20 % throughput vs H100 on Llama 2 70B inference and 2× price/performance. The figure appears in the official whitepaper and in launch presentations. However, Signal65 (an independent firm) published in 2025 that the H200 beats Gaudi 3 by a factor of 9× on Llama 3.1 405B. The discrepancy matters: both numbers can be true for their specific benchmarks (Llama 2 70B FP16 vs Llama 3.1 405B FP8) but the operational conclusion changes radically depending on which one you keep.

This post’s recommendation: treat Gaudi 3 as a valid option for Llama-class 70B in BF16/FP8 where Intel claims parity or advantage, not for frontier 200B+ models where NVIDIA keeps a clear margin. And consider the roadmap risk: Gaudi 4 will never exist; the line’s successor is Jaguar Shores 2026 with a completely new rack-scale architecture, a discontinuity rather than an evolution.

Intel Xeon 6 with AMX — the CPU workhorse

The Advanced Matrix Extensions (AMX) are the non-obvious piece. Each Granite Rapids P-core executes up to 1,024 BF16 FLOPS per cycle via AMX, which turns a Xeon 6 with 64-86 cores into a respectable matrix accelerator for small and mid-sized models. Real reported figures: Llama 3.2 INT4 at ~57 tok/s with AMX vs 28 tok/s without AMX (a clean 2× factor). For serving 7B INT4 with the vLLM CPU backend + AMX, Intel claims 2.7× tok/s vs EPYC 9965, a figure with Intel bias but qualitatively corroborated by LMSYS in their DeepSeek R1 671B deployment on Xeon 6 + SGLang.

Operational use case: embeddings and reranking on a Xeon 6 sidecar next to the H100 cluster. Models such as bge-m3 (multilingual embedding) or bge-reranker-v2-m3 run at acceptable throughput on CPU AMX; they do not deserve a dedicated H100. Freeing the H100 for the 70B LLM raises the cluster’s effective RPS without buying more GPUs.

Intel Arc Pro B60 and Battlematrix

Launched in September 2025, the Arc Pro B60 (Battlemage) brings 24 GB GDDR6 with 456 GB/s of bandwidth and 197 TOPS INT8 at 200 W. Maxsun has a dual-GPU 48 GB variant. The “Battlematrix” rack configuration combines 8 units = 192 GB of aggregate VRAM. The interesting point: at a cost far below an H100 SXM 80 GB, which makes it a candidate for 30-70B INT4-INT8 LLMs served via OpenVINO or llama.cpp SYCL.

Phoronix verified that on SYCL the Arc Pro B70 reaches parity with the Radeon PRO W7900 (previous AMD generation) on DeepSeek R1 Llama 8B pp512. The Vulkan backend loses heavily (~1/4 of SYCL performance); for Arc Pro, always SYCL.

Intel NUC with NPU — the edge node

Intel NUCs with a Lunar Lake CPU (Core Ultra Series 2) bring NPU 4 with 48 TOPS and a platform total of 120 TOPS adding the Xe2 iGPU and CPU AVX. Panther Lake (CES January 2026) goes up to NPU 5 = 50 TOPS + Xe3 120 TOPS = 180 TOPS platform.

Intel states that Panther Lake “runs 30-70B models locally”. Reality check: it is marketing. A 30-70B INT4 fits in RAM (DDR5x 32-64 GB) but sustained speed with Q4_K_M quantisation on a NUC hovers around 2-8 tok/s; comfortable for occasional use, not for serving traffic. The NUC’s real sweet spot is 7B INT4 at 20-40 tok/s on the iGPU/NPU, perfect for a client branch office with casual queries.

Intel software — the relevant stack

OpenVINO 2025.3 (June 2026) is the central piece. It supports single-command deployment via the OVMS CLI with automatic download from the HF Hub; it integrates OpenVINO GenAI with a C++/Python API for generative pipelines; it exposes a vLLM v1-compatible API (vLLM-OpenVINO). GGUF model support: DeepSeek Distill, Qwen 2/2.5, Llama 3. Optimisations: Sage Attention (first token with long prompts), per-channel KV-cache compression.

Intel Extension for PyTorch (IPEX), XPU version 2.8.10+xpu, adds Intel backends to PyTorch. IPEX-LLM is the subproject that integrates with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM and DeepSpeed. May 2025: it ran DeepSeek V3/R1 671B and Qwen3MoE 235B on 1-2 Arc A770/B580 cards with FlashMoE.

vLLM CPU backend, the CPU branch of vLLM with AMX optimisations. For 7B INT4 on a 4th-gen Xeon with AMX: 12-50 tok/s; with a Xeon Gold 6530 + INT4: ~80 tok/s. Academic figures (arXiv 2410.04466).

llama.cpp SYCL, the recommended backend for Arc; Vulkan works but hovers around 1/4 of SYCL performance on an Arc B580. SYCL reaches parity with the previous AMD generation.

Habana SynapseAI, the Gaudi 3 stack. The PyTorch bridge habana_frameworks.torch registers the hpu device; it integrates with torch.compile. It is not a complete port to oneAPI but a partial integration via oneMKL. That implies the Gaudi ecosystem keeps a certain separation from Intel’s general oneAPI, relevant in view of the hypothetical Jaguar Shores and future unification.

The four canonical patterns

Pattern 1 — embeddings + reranker on Intel

The most mature and the easiest to adopt. In a typical RAG system, each user request invokes:

Query embedding (50 ms on an H100, 80 ms on a Xeon AMX, 30 ms on an Arc Pro B60).
Vector search (Qdrant / Milvus / Chroma; latency ~10-30 ms).
Reranker over the top-k candidates (60 ms on an H100, 100-150 ms on a Xeon AMX).
LLM over the augmented prompt (200-500 ms TTFT, 30-50 ms/token).

Steps 1 and 3 are memory-bound and relatively small (100M-1B models): a Xeon 6 with AMX (the Arc Pro B60 is faster but is already a dedicated GPU) does the job at a hardware cost that is a fraction of a dedicated H100. Step 4 stays on NVIDIA because that is where its tensor architecture + HBM3 + FP8 is worth what it costs.

Operational implication: a Xeon 6 sidecar (~40 cores, ~10-15 k€) serving embeddings + reranking frees the equivalent of 1-2 H100s of “cheap” load, recovering that capacity for the big LLM. Clear ROI in sizing.

Pattern 2 — guardrails + PII redaction on a near-edge NUC

A bank branch, a medical practice or a law office generates prompts with personal or classified data. Sending those prompts to the central DC (even if it is corporate on-premise) can clash with local retention policies or with specific compliance (GDPR, professional privilege).

The pattern: the NUC in the branch office runs two critical steps before the round-trip:

PII redaction with Presidio (CPU-only, fast) or Llama Guard 4 8B on the NUC’s NPU + iGPU. It replaces names, tax IDs and account numbers with placeholders.
Light guardrails (PromptGuard 2 86M, Llama Guard 4 8B) on the NPU + iGPU. It filters prompt injection, jailbreaks and prohibited content.

Only afterwards does the redacted prompt travel to the central DC so the big LLM can answer. The response comes back to the NUC, which rehydrates the placeholders with the real values before showing it to the user. The sensitive data never leaves the branch office.

Costs: a Panther Lake NUC is ~1,500-2,500 € per unit, scalable to dozens of branch offices with no additional central GPU cost. Latency: 50-150 ms for the edge step before the DC’s 200-500 ms round-trip.

Pattern 3 — speculative decoding drafter on a NUC

Speculative decoding uses a small drafter that proposes γ tokens and a large target that accepts or rejects them in a single forward pass. If the drafter is geographically close to the user (a NUC in the branch office) and the target is in the central DC, the client’s perceived latency falls further still.

Setup: drafter Llama 3.2 1B INT4 on a NUC + target Llama 3.1 70B FP8 on the central H100. The NUC generates γ=4 tokens in ~50 ms locally; the target verifies them in one pass (40-80 ms including round-trip); if the acceptance rate is > 60 %, effective TTFT falls ~50 % compared with Llama 70B without speculative decoding.

An important constraint: drafter and target must share a tokenizer. Llama 3.2 1B and Llama 3.1 70B have compatible tokenizers. Mixing a Llama drafter with a Qwen target breaks the pattern.

Pattern 4 — NUC dev workstations

Dev / CI iterating on prompts, evals and retrieval logic does not need a production GPU to validate correctness. A NUC with Llama 3.2 3B INT4 runs the end-to-end functional tests (including embeddings + retrieval + LLM + guardrails) at a tenth of the cost of iterating on an H100. Only the last pre-prod smoke test uses the production cluster.

A mature pattern in organisations with many developers and scarce production GPU. Iteration that is 10× faster and cheaper translates into feature delivery speed.

Unified observability in a heterogeneous cluster

The GPU observability post covered DCGM Exporter for NVIDIA. In a mixed cluster you need more:

Hardware part	Exporter	Key metrics
NVIDIA H100/A100	`nvidia/dcgm-exporter`	DCGM_FI_DEV_* + DCGM_FI_PROF_*
Intel Gaudi 3	`HabanaAI/habana-metric-exporter`	habana_hpu_utilization, habana_hbm_used
Intel Arc Pro	`intel/intel-gpu-exporter` (unofficial; alternatives exist)	xe_engine_utilization, xe_memory_used
Intel Xeon CPU + AMX	`prometheus/node-exporter` + Intel PCM	cpu_amx_utilization (via PCM)
Intel NUC (NPU+iGPU)	`intel/intel-gpu-exporter` + custom NPU exporter	npu_utilization, xe_iGPU

All federated into a single Prometheus + Grafana. The dashboards are organised by hardware family (NVIDIA, Intel server, Intel edge) plus an aggregate “heterogeneous cluster” view with per-tenant SLOs combining all four.

Cardinality: ~1.5-2× that of a pure NVIDIA cluster. Manageable with Thanos / Mimir for long retention.

Capability routing — from the L7 router to the heterogeneous one

The LLM inference router stops being a version selector for the same model and becomes a capability dispatcher:

models:
 - name: "llama-70b-chat"
 endpoint: "vllm-llama70b.inference.svc:8000"
 backend: nvidia-h100
 capabilities: [chat, tool_use, json_mode]
 - name: "embedding-multilingual"
 endpoint: "ipex-bge-m3.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [embeddings]
 - name: "reranker-multilingual"
 endpoint: "ipex-bge-reranker.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [reranking]
 - name: "guardrail-prompt-injection"
 endpoint: "openvino-llama-guard.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [guardrails, redact-pii]
 region: sucursal-01
 - name: "llama-3b-draft"
 endpoint: "openvino-llama-3b.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [speculative-drafter]
 region: sucursal-01
 target_model: "llama-70b-chat"

The router resolves model=embedding-multilingual → Intel Xeon; model=llama-70b-chat → H100; model=guardrail-prompt-injection with region=sucursal-01 → local NUC. If the branch office NUC goes down, failover to an equivalent replica in the central DC, accepting the latency cost.

LiteLLM Proxy, NVIDIA Dynamo and Envoy AI Gateway support this capability routing. The non-obvious piece: the router has to know the compatible tokenizer between drafter and target for pattern 3, which is modelled in extra catalogue metadata.

Specific pitfalls

Tokenizer mismatch between engines. OpenVINO with a Llama 3.2 GGUF and vLLM with the same nominal Llama 3.2 can use slightly different tokenizers (chat template, special tokens). Validate token identity with tokenizer.encode("hola") on both sides before assuming interchangeability. For speculative decoding, a single different token breaks the pattern.

Edge ↔ central round-trip latency. Patterns 2 and 3 assume the NUC and the DC are on the same corporate WAN with controlled latency. If the branch office is on 4G/5G with 100-200 ms of jitter, the speculative drafter compensates for nothing; on the contrary, it adds latency. Measure before promising.

FP8 Hopper ≠ INT8 AMX in output quality. The operator assumes a request that runs FP8 on an H100 and INT8 on a Xeon AMX will produce the same output. That is not true: the two quantisations have different degradation profiles. If the system expects idempotence (for example, evals with golden output), validate offline that the Intel version reproduces the expected behaviour within tolerance.

Model version synchronisation between sites. The model in the central DC gets updated, but the branch office NUCs keep the old version of the drafter or the guardrail for weeks. Result: divergent behaviour between branches with no easy diagnosis. Policy: the central model and the edge model advance together or within a documented window; the canary extends to the NUC fleet.

Unstable Intel roadmap. Falcon Shores cancelled, Gaudi 4 will never exist, Jaguar Shores 2026 still with no confirmed public specs. Buying Gaudi 3 today is reasonable if the use case justifies the 18-24 months of amortisation; committing an architecture for 5+ years on an Intel accelerator is a riskier bet than the NVIDIA equivalent, at least until Jaguar Shores materialises with a mature software stack.

Absence of public production deployments. As of June 2026, the confirmed Gaudi 3 deployments are IBM Cloud, Dell AI Factory and a handful of early adopters (Bharti Airtel, Bosch, Naver). There is no publicly verifiable case of a mixed NVIDIA + Intel cluster in European banking or government. This pattern is recommended emergent architecture, not practice with an industrial track record. The first adopter takes on a validation cost that a second adopter avoids.

Applied to a generic on-premise cluster

For an organisation with a generic heterogeneous LLM inference cluster:

Central DC: 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink = 16 H100s. Serves the big LLMs (Llama 70B, Mixtral 8×22B, Qwen 72B) in BF16 or FP8.
Xeon 6 sidecar: 2-4 Xeon 6 servers (Granite Rapids 64-86 cores) with AMX, 512 GB DDR5, in the same rack as the H100 cluster. Serves embeddings (bge-m3), reranking (bge-reranker-v2-m3) and small models (Llama 3.2 1B/3B) on the vLLM CPU backend with AMX.
Arc Pro sidecar (optional): 1-2 servers with 4-8× Arc Pro B60 24 GB each (Battlematrix), for 13-30B INT8 models via OpenVINO. Useful if the cost per mid-sized LLM has to come down from the H100.
Edge NUCs in branch offices: 1-2 Panther Lake NUCs per branch, with a 50 TOPS NPU + Xe3 120 TOPS, serving Llama Guard 4 + Presidio + a Llama 3.2 1B INT4 drafter via OpenVINO. Corporate WAN connectivity with latency < 80 ms to the DC.

Estimated envelope: central cluster ~120 kW of GPU peak + ~10-15 kW of Intel sidecars. Edge: ~50 W per NUC, negligible compared with office costs.

Observability: federated Prometheus in the DC + pull scrape from the NUCs (via corporate VPN). Dashboards “NVIDIA GPU fleet”, “Intel server fleet”, “Intel edge fleet” plus a “consolidated SLO” view.

Router: LiteLLM Proxy or NVIDIA Dynamo in the DC, with an extended model catalogue to include Intel backends and regions (sucursal-01, sucursal-02, …). Documented edge→central failover.

What we have not covered (upcoming posts)

Reproducible benchmarks of Llama 70B on Gaudi 3 vs H100 SXM on equivalent hardware, the missing material for making decisions with your own data rather than Intel’s or Signal65’s.
AMD ROCm in the mix: how MI300X / MI355X fit into this heterogeneous pattern and what changes in the router catalogue.
Apple Silicon as edge: M3/M4 Max with a Neural Engine of ~38 TOPS + a 40-core GPU, hardware equivalent to the Panther Lake NUC but with a different software stack (MLX).
Energy cost optimisation: how nvidia-smi -pl 500W + an Intel TDP cap on Xeon 6 cuts the bill by 25-30 % with a 15-20 % throughput loss.
Model CI/CD for an edge fleet: how a rolling update of a Llama Guard reaches 50 branch office NUCs without any of them losing service.

References

Intel Gaudi 3

Intel — Gaudi 3 AI Accelerator White Paper. https://cdrdv2-public.intel.com/817486/gaudi-3-ai-accelerator-white-paper.pdf
Intel — Hot Chips 2024 Gaudi 3 deep dive. https://hc2024.hotchips.org/assets/program/conference/day1/60_HC2024.Intel.RomanKaplan.Gaudi3-0826.pdf
Signal65 / DataCenterDynamics — NVIDIA H200 outperforms Intel Gaudi 3 by factor of 9× across first Llama 3.1 405B benchmark test. https://www.datacenterdynamics.com/en/news/nvidia-h200-outperforms-intel-gaudi-3-by-factor-of-nine-across-first-llama-31-405b-benchmark-test-exclusive/
IEEE Spectrum — Intel Gaudi 3 review. https://spectrum.ieee.org/intel-gaudi-3
Tom’s Hardware — Intel cancels Falcon Shores GPU; Jaguar Shores to be successor. https://www.tomshardware.com/tech-industry/artificial-intelligence/intel-cancels-falcon-shores-gpu-for-ai-workloads-jaguar-shores-to-be-successor

Intel Xeon 6 + AMX

Intel — Xeon 6 (Granite Rapids) Product Brief. https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-02/xeon-6-granite-rapids-product-brief.pdf
OpenMetal — Intel AMX AI Inference Performance. https://openmetal.io/resources/blog/intel-amx-ai-inference-performance/
LMSYS — Intel Xeon 6 + SGLang for DeepSeek R1 671B. https://www.lmsys.org/blog/2025-07-14-intel-xeon-optimization/
arXiv 2410.04466 — CPU-LLM benchmarks with AMX.
Intel community blog — Accelerating vLLM Inference on Intel Xeon 6 Processor.

Intel Arc Pro Battlemage

Intel — Arc Pro B60 Graphics Specifications. https://www.intel.com/content/www/us/en/products/sku/243916/intel-arc-pro-b60-graphics/specifications.html
StorageReview — Intel Arc Pro B60 Battlematrix Preview: 192GB VRAM for On-Premise AI. https://www.storagereview.com/review/intel-arc-pro-b60-battlematrix-preview-192gb-of-vram-for-on-premise-ai
Phoronix — Intel Arc Pro B-series review. https://www.phoronix.com/review/intel-arc-pro-b-series

Intel NUC / NPU

HotHardware — Intel CES 2026 Panther Lake is a Go. https://hothardware.com/news/intel-ces-2026-panther-lake-is-a-go
TechPowerUp — Intel Panther Lake Technical Deep Dive.
arXiv 2412.11053 — NITRO: LLM inference on laptop NPU.
Intel — AI PC brings larger LLM development to your desk.

Software

OpenVINO — Release Notes 2025.3. https://www.intel.com/content/www/us/en/developer/articles/release-notes/openvino/2025-3.html
HuggingFace — Deploy with OpenVINO. https://huggingface.co/blog/deploy-with-openvino
Intel — Intel Extension for PyTorch XPU 2.8.10. https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/releases.html
IPEX-LLM — github.com/intel/ipex-llm.
Habana — SynapseAI PyTorch Theory of Operations. https://docs.habana.ai/en/latest/PyTorch/PyTorch_Gaudi_Theory_of_Operations.html

Market context

MLCommons — MLPerf Inference v6.0 benchmark results. https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
Intel newsroom — Gaudi 3 Expanded Availability. https://newsroom.intel.com/artificial-intelligence/intel-gaudi-3-expands-availability-drive-ai-innovation-scale

Sources: the full URLs are linked inline on each reference.

The LLM inference router: the L7 switchboard we called a LoadBalancer in the canary post

Tue, 02 Jun 2026 03:00:00 +0200

This post is the natural continuation of Canary, blue-green and shadow for LLM models. There the promotion mechanics dumped all the traffic-splitting complexity into a box we called “LoadBalancer”. The description was operational, good enough to understand the choreography, but structurally vague: what actually does that splitting is an L7 inference router with LLM awareness, a piece of the stack in its own right (layer 1 of the seven layers) that deserves its own post.

TL;DR

In the previous post on canary we called LoadBalancer the piece that splits traffic between the stable v1 and candidate v2 pools. The description was enough to understand the flow, but technically it was blurry: neither an L4 LoadBalancer (kube-proxy, MetalLB, IPVS) nor a generic L7 HTTP LoadBalancer (NGINX or HAProxy with no extension) knows what a model is, what a version is, how many tokens a request costs, what prefix the prompt has or which replica has which KV cache hot. The right piece is an LLM inference router: an L7 proxy with explicit knowledge of the domain. It combines four functions: model catalogue (resolving model=llama-70b@v2 → service.namespace:port), traffic splitting (applying the canary weight with a deterministic hash or deliberate stickiness for A/B), cross-cutting policy (OIDC auth, per-tenant rate limit and quota, pre-prompt PII redaction, lightweight inline guardrails, gen_ai.* tracing propagation) and failover/degradation (if v2 falls over, redirect to v1; if the whole cluster is saturated, return 503 with Retry-After instead of queueing forever). The non-obvious piece that justifies its technical existence beyond the operational one is prefix-aware routing: the router decides which fleet replica each request goes to based on the prompt prefix, so that a RAG system with the same system prompt plus the same block of retrieved documents systematically hits the prefix cache (RadixAttention in SGLang, PrefixCaching in vLLM, KV reuse in TensorRT-LLM) of the same replica, multiplying the hit rate from 5–15 % (blind round-robin) to 60–85 % (prefix affinity). The concrete pieces in May 2026 are LiteLLM Proxy (the simplest option, OpenAI-compatible, declarative YAML catalogue), vLLM Production Stack router (specific to vLLM fleets, aware of the KV cache and the prefix), Envoy AI Gateway (LLM-aware Envoy filters, integrable with Istio), Kong AI Gateway (enterprise alternative with a plugin ecosystem), KGateway (CNCF, still incubating) and the NVIDIA Dynamo router (production-grade, aware of disaggregated prefill/decode serving). In the seven-layer stack it lives in layer 1 (gateway); in the five maturity levels it appears from level 3 onwards; in the seven-phase deployment cycle it is the last piece that F6 closes. This post includes a minimal manifest applicable to a generic 4×H100 SXM cluster.

You are here: DEPLOY (layer 1 of the stack)

The background: what the canary post called a “LoadBalancer”

In Canary, blue-green and shadow for LLM models we described the flow like this: “the LoadBalancer progressively splits traffic following a schedule: 1 % → 5 % → 25 % → 100 %”. It was a correct operational description, and the reader understood the choreography without needing more. But technically it left unnamed a piece that deserves explicit treatment, because neither of the two usual senses of “LoadBalancer” does what that paragraph assumed:

An L4 LoadBalancer (kube-proxy with iptables/IPVS, MetalLB, F5 BIG-IP in TCP mode) splits IP packets without looking inside the payload. It does not know which model is being requested, nor which version, nor how many tokens it carries, nor whether the client has quota. It cannot apply the canary weight “for model X version 2”: to it, every packet heading for the vllm-llama70b VIP is indistinguishable.
A generic L7 HTTP LoadBalancer (NGINX or HAProxy in HTTP mode with no extension, a ClusterIP Service with multiple backends) does split by URL and can route by header, but it does not understand the OpenAI-compatible body of the request. It does not know that {"model": "llama-70b", "messages": [...]} carries the routing key in the model field; it does not count tokens; it does not apply policies over LLM structures; it does not do prefix-aware routing because that requires parsing messages and hashing the common prefix.

The piece the canary post assumed was doing this work is an L7 inference router with LLM awareness. A layer in its own right, with its own configuration, its own CI/CD, its own metrics and its own pitfalls. This post names it and takes it apart.

The analogy: the switchboard and triage of a hospital with multiple specialities

A large hospital receives patients who arrive at A&E through different doors and who need different specialities: trauma, cardiology, paediatrics, oncology. There are three possible models of “front door”.

A single door with no triage. All patients wait in the same room and are sent in arrival order to the first free doctor, whatever their speciality. It works in a village surgery with a single general practitioner. When there are 200 patients a day and 12 specialities, it falls into dysfunction fast: the cardiologist treats sprains, the paediatrician treats heart attacks, specialised resources are wasted. This is the equivalent of the L4 LoadBalancer, which distributes bodies without understanding what they bring.

A door with a receptionist who asks about the symptom. Someone at the front desk asks “what’s wrong?” and directs the patient to the right corridor. The cardiologist sees only cardiology, the paediatrician only children. Better, but the receptionist is slow, does not gauge urgency and does not know the state of the rooms: they may send you to the cardiologist in corridor A when the one in B is free. This is the equivalent of a generic L7 HTTP proxy with path-based routing, which distributes by category but with no information about internal state.

Professional triage with full awareness. A trained triage nurse who knows the catalogue of specialities, knows which bay is busy and which is free, remembers the recurring patient whose file is already open in the system (and sends them to the same doctor for continuity), applies cross-cutting policy (checks insurance cover, records allergens, redirects to paediatric A&E if the patient is a minor) and, if the cardiology room goes down because the electrocardiograph has broken, redirects to the hospital on the other side of the city. This is the piece a large hospital needs. In LLM terms it is called an inference router.

The analogy holds down to the last detail, including the “file already open” one: the patient who returns to the same doctor is exactly the client whose prompt shares a prefix with the one from five minutes ago. If the router sends it to the same replica, that replica still has the KV cache hot and the request hits the prefix cache. If it sends it to a different replica because that one was “next in round-robin”, the KV cache has to be rebuilt from scratch and TTFT doubles. The triage nurse knows this. The blind LoadBalancer does not.

The four functions of the inference router

Function 1 — Model catalogue

The router keeps a declarative catalogue mapping model identity to a concrete deployment:

models:
 - name: "llama-70b" # stable alias
 version: "v2" # canary version
 weight: 5 # 5% of the traffic
 endpoint: "vllm-llama70b-v2.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: canary
 - name: "llama-70b"
 version: "v1"
 weight: 95
 endpoint: "vllm-llama70b-v1.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: stable
 - name: "embedding-multilingual"
 version: "v1"
 weight: 100
 endpoint: "tei-bge-m3.inference.svc.cluster.local:8080"
 capabilities: [embeddings]
 lifecycle: stable

The client sends {"model": "llama-70b", "messages": [...]} without knowing there are two replica pools behind it. The router resolves. If tomorrow you migrate from vLLM to SGLang for a specific version, the client never notices; you change the endpoint in the catalogue and that is it.

What this decoupling buys you is the freedom to move topology without breaking clients. What it costs is keeping the naming convention disciplined (llama-70b is always the stable alias; llama-70b@v2 is the specific version for canary). Without that discipline, the aliases get dirty with llama-70b-prod-fixed-real-final-v3 and the catalogue stops being navigable within a few weeks.

Function 2 — Traffic splitting

The splits from the canary post (1 % → 5 % → 25 % → 100 %) materialise here, not in the inference engine. The router computes a deterministic hash of the request_id (or of the user_id, if you want stickiness) and maps it to the weight range in the catalogue. For a weight of [v1: 95, v2: 5], 5 % of the hash space falls in v2 and 95 % in v1.

Three design decisions that matter:

Hash by random request_id = independent sampling. Each request is an independent observation of the v1 vs v2 distribution. It is the right setting for statistically comparable canaries.
Hash by user_id = per-user stickiness. The same client always sees the same pool. Useful for A/B testing with persisted conversational memory, but it breaks the statistical comparability of the canary because user populations are not symmetric, a pitfall explained in the previous post.
Hash by tenant_id = hard partitioning. Tenant A goes to v1, tenant B to v2. This is the pattern for clients with different SLAs, or for validating v2 in an internal tenant before exposing it to external clients.

Function 3 — Cross-cutting policy

Once, across all models, the router applies:

Auth: OIDC with JWT tokens validated against Keycloak / Authentik. Authorization: Bearer ... headers translated into tenant_id and roles.
Rate limit: token bucket per tenant (X req/min) or per model (Y req/min for llama-70b because it is expensive).
Quota: monthly quota of tokens consumed per tenant. The router counts gen_ai.usage.input_tokens + gen_ai.usage.output_tokens and rejects with 429 Quota exceeded when it runs out.
Pre-prompt PII redaction: Presidio or Llama Guard inline before the prompt touches the model. What the model does not see is not trained on, not logged, not leaked.
Lightweight inline guardrails: PromptGuard 2, Llama Guard 4, Granite Guardian, the ones covered in Guardrails and safety in LLMs, run in the router because their latency (30–150 ms) fits within the TTFT budget.
gen_ai.* tracing propagation: the router starts the parent span, propagates traceparent to the engine and emits the gen_ai.system, gen_ai.request.model and gen_ai.request.version attributes that OTel GenAI tracing consumes.
Semantic cache: for prompts that repeat exactly or with high semantic similarity (embedding cosine > 0.97 against the previous cache), it returns the cached answer without touching the engine. Typical saving in RAG with frequently asked questions: 20–40 % of requests.

Function 4 — Failover and degradation

The router knows the health state of each endpoint (active health probes every 5–15 s, recent TTFT latency) and decides:

If v2 returns persistent 5xx or does not respond, the circuit breaker opens: the router redirects the traffic that was going to v2 towards v1 until the probes go green again. This is automatic canary rollback in its simplest form.
If the whole cluster is saturated (all replicas reporting num_requests_waiting > N for T seconds), the router returns 503 Service Unavailable with Retry-After: 30 instead of queueing forever. It is better to tell the client “come back in 30 seconds” than to keep them waiting four minutes and then time out.
If there is multi-region or multi-cluster, cross-cluster failover via DNS or L7: the primary region goes down and the secondary router takes over.

The non-obvious piece: prefix-aware routing

This is the function a conventional LoadBalancer cannot do and the one that justifies an LLM-specific router beyond the four generic ones.

The KV cache in vLLM, SGLang and TensorRT-LLM can reuse common prefixes across requests, see KV cache. Specifically:

vLLM with --enable-prefix-caching: it detects that the current request shares a prefix (of a length that is a multiple of the block size, 16 tokens by default) with a previous request whose pages are still in HBM, and reuses those pages instead of reprocessing them.
SGLang with RadixAttention: it structures the cache as a radix tree indexed by tokens; each request hits the common path of the tree and only computes the tail.
TensorRT-LLM: a similar feature, called KV cache reuse.

The prefix cache hit rate is the key metric: every token that hits is a token that is not processed in prefill, reducing TTFT in direct proportion. For a typical RAG system, with a 400-token system prompt plus 2,000 tokens of retrieved documents plus a 50-token user question, the common prefix (system_prompt + docs) is 2,400 of the 2,450 total tokens. If the cache hits, prefill only processes 50 tokens instead of 2,450: TTFT falls to roughly a twentieth.

But the cache lives per replica, not globally. If two requests with the same 2,400-token prefix land on different replicas, both do the full prefill: the first one’s cache is no use to the second. The second pays the full cost.

With blind round-robin (any conventional LB), requests are spread uniformly across N replicas. For a cluster of 4 replicas and 1,000 requests with the same system_prompt + docs, each replica receives ~250 requests, but all 4 do their own “first prefill” and the following 249 benefit within their replica. The global hit rate is decent but not optimal. For traffic with many different system prompts and little intra-prefix repetition, the hit rate hovers around 5–15 %.

With prefix-aware routing, the router computes a hash of the prompt prefix (the first N tokens, or the system_prompt declared in messages[0]) and keeps an affinity table of hash → replica. All requests with the same prefix land on the same replica. The first pays the full prefill; the following 999 hit the cache. Global hit rate: 60–85 %.

The cost of implementing it: the router has to parse the request body (not just the HTTP header), apply a lightweight tokeniser or a byte-based hash, and maintain an LRU/consistent-hash affinity table that rebalances when a replica comes in or out. That is server work, not generic proxy work. vLLM Production Stack router implements it natively. NVIDIA Dynamo does too. LiteLLM has a beta in its enterprise version. Envoy AI Gateway is adding it as an experimental filter.

Correction (September 2026). The previous sentence about LiteLLM is false: prefix-aware routing does not exist in any version, open or paid, verified against the code of 1.102.0. The detail, what LiteLLM does have, and the real state of the field are in Prefix routing: what LiteLLM does not do.

The operational difference for a production RAG system: with prefix-aware routing, the same cluster serves 2–4× more requests without adding GPUs, simply because prefill disappears in most cases.

Token-aware load balancing

The second non-obvious piece. Classic round-robin splits by number of requests; but a 50-token prompt and an 8,000-token one cost radically different amounts (a factor of ~160× in prefill). Splitting equally by count severely unbalances the real load.

Token-aware load balancing adds up the expected prefill tokens (prompt length) and expected decode tokens (the client’s max_tokens) per active replica, and sends the new request to the replica with the lowest accumulated load. It is what both vLLM Production Stack and NVIDIA Dynamo implement as the default strategy when it is enabled.

The metric feeding the calculation is, once again, vllm:num_requests_running and vllm:gpu_cache_usage_perc, see GPU observability for LLM inference, ideally complemented with an estimator of the incoming prompt’s tokens. Mature routers use tiktoken or the model’s real tokeniser to count prompt tokens before choosing a replica.

Comparison of concrete pieces (May 2026)

Piece	LLM awareness	Prefix-aware	Token-aware LB	Multi-model	Semantic cache	Plug & play
LiteLLM Proxy	High	Beta (enterprise)	Yes	Excellent	Yes (Redis)	Very high
vLLM Production Stack router	vLLM-specific	Yes, native	Yes	vLLM only	No (external)	Medium
NVIDIA Dynamo router	High + disagg-aware	Yes	Yes	vLLM/SGLang/TRT-LLM	No (external)	Low
Envoy AI Gateway	Medium (filters)	Experimental	Yes	Yes	Via filter	Medium
Kong AI Gateway	Medium (plugins)	No	Yes	Yes	Yes (plugin)	Medium
KGateway	Medium	Roadmap	Yes	Yes	Roadmap	Low (CNCF incubating)
NGINX + custom Lua	Manual	No	Manual	Manual	No	Low (build it yourself)

LiteLLM Proxy is the default option to start with. OpenAI-compatible, simple YAML, supports the commercial providers plus any self-hosted OpenAI-compatible backend. The OSS version covers the four basic functions and semantic cache; prefix-aware routing and the enterprise version add advanced multi-tenancy.

vLLM Production Stack router is the right option if the fleet is 100 % vLLM. Aware of the KV cache, the prefix and the LoRA loaded per replica. It integrates better with native vLLM metrics.

NVIDIA Dynamo router is the most complete production-grade option, especially if you operate disaggregated serving (separate prefill workers and decode workers). It requires an NVIDIA-aligned stack.

Envoy AI Gateway and Kong AI Gateway are the options if the organisation already has Envoy or Kong as its corporate gateway and wants to extend it with LLM awareness without introducing another new piece.

Minimal manifest: LiteLLM Proxy on a generic cluster

apiVersion: v1
kind: ConfigMap
metadata: { name: litellm-config, namespace: inference }
data:
 config.yaml: |
 model_list:
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v1.inference.svc:8000/v1
 weight: 95
 model_info:
 version: v1
 lifecycle: stable
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v2.inference.svc:8000/v1
 weight: 5
 model_info:
 version: v2
 lifecycle: canary
 - model_name: embedding-multilingual
 litellm_params:
 model: openai/bge-m3
 api_base: http://tei-bge-m3.inference.svc:8080
 router_settings:
 routing_strategy: least-busy # basic token-aware
 num_retries: 1
 timeout: 60
 general_settings:
 master_key: "os.environ/LITELLM_MASTER_KEY"
 database_url: "os.environ/DATABASE_URL"
 litellm_settings:
 cache: true
 cache_params:
 type: redis
 host: redis.inference.svc
 port: 6379
 similarity_threshold: 0.97
 success_callback: ["langfuse"]
 failure_callback: ["langfuse"]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: litellm-router, namespace: inference }
spec:
 replicas: 3
 selector: { matchLabels: { app: litellm } }
 template:
 metadata: { labels: { app: litellm } }
 spec:
 containers:
 - name: litellm
 image: ghcr.io/berriai/litellm:v1.55.0
 args: ["--config=/config/config.yaml", "--port=4000", "--num_workers=4"]
 ports: [{ containerPort: 4000, name: http }, { containerPort: 4000, name: metrics }]
 env:
 - { name: LITELLM_MASTER_KEY, valueFrom: { secretKeyRef: { name: litellm-secret, key: master_key } } }
 - { name: DATABASE_URL, valueFrom: { secretKeyRef: { name: litellm-secret, key: db_url } } }
 - { name: LANGFUSE_PUBLIC_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: public } } }
 - { name: LANGFUSE_SECRET_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: secret } } }
 volumeMounts: [{ name: config, mountPath: /config }]
 readinessProbe: { httpGet: { path: /health, port: 4000 } }
 volumes: [{ name: config, configMap: { name: litellm-config } }]
---
apiVersion: v1
kind: Service
metadata: { name: litellm-router, namespace: inference }
spec:
 selector: { app: litellm }
 ports: [{ name: http, port: 80, targetPort: 4000 }]
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: litellm-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: litellm } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s

The end client points at litellm-router.inference.svc:80/v1/chat/completions, sets model=llama-70b, and the router decides on each request whether it goes to v1 (95 %) or v2 (5 %), applies the rate limit, looks in the semantic cache, propagates tracing to Langfuse, and translates from OpenAI-compatible to the OpenAI-compatible API of the destination vLLM. Three router replicas for HA and so the gateway itself can scale horizontally with KEDA if needed, see LLM autoscaling on Kubernetes.

Four operational pitfalls

Pitfall 1 — the router becomes a SPoF if it is not replicated. Three or more replicas of the router itself, behind a LoadBalancer Service (this one genuinely L4) with healthchecks. A single router replica means every configuration deploy closes the whole service for a few seconds.

Pitfall 2 — the router’s latency adds to the model’s. Each function adds milliseconds: body parsing (5–10 ms), JWT auth (2–5 ms), rate limit (1–2 ms), PII redaction with Presidio (20–80 ms), inline guardrails with Llama Guard (50–150 ms), prefix hash (5–10 ms), token counting with a tokeniser (10–30 ms). In total, 100–300 ms of overhead before touching the engine. If the model’s TTFT is 400 ms and the router’s is 200 ms, the client sees 600 ms, so it is worth measuring what each function costs and disabling the non-critical ones in the low-latency path.

Pitfall 3 — the catalogue drifts from the real state of the cluster. The router believes vllm-llama70b-v2 exists because it is in its YAML, but the deployment was withdrawn three days ago and nobody updated the config. The router returns 502 on 5 % of the traffic. Fix: validate the catalogue against kubectl get svc in CI; no endpoint in the catalogue may point at a non-existent Service. Or better: the router discovers the available endpoints dynamically via a label selector (app=vllm,model=llama-70b) and applies the catalogue weights over the ones that are alive.

Pitfall 4 — semantic cache with an outdated embedding. The semantic cache compares the new prompt’s embedding against the embeddings of cached prompts. If you update the embedding model (see RAG corpus curation), the distances are computed in a different space and the cache stops working correctly (false hits or false misses). Policy: the cache is invalidated when the embedding model changes; generations are never mixed.

Fit within the stack and maturity

In the seven-layer stack, the router is layer 1: the front door that precedes the inference engine (layer 2), the KV cache + PagedAttention (layer 3) and the rest. It is the only piece that sees all the traffic from the outside; any policy not applied here gets duplicated N times in the engines.

In the five maturity levels, the router appears from level 3 (MANAGED) onwards: without OIDC + RBAC + cert-manager + default-deny NetworkPolicy, the router has nobody to authenticate and nobody to apply quotas to; before level 3, what you should build is a minimal proxy with no pretension of a catalogue. Platforms that try to have a polished router at level 1 end up with a large YAML nobody maintains.

In the seven deployment phases, the router is what closes F6: the last atomic step that puts the cluster into production. Without a router, F6 does not finish, because the catalogue, the quotas, the canaries and the failovers are a necessary condition for opening production traffic.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB, the inference router consumes modest resources: 3 replicas of the router pod (2 CPU cores, 4 GiB of memory each) are enough to sustain thousands of RPS because its work is light (parsing, hashing, routing, no inference). The router lives on non-GPU nodes of the cluster (control plane nodes or general workload nodes), and never consumes nvidia.com/gpu.

Traffic volume that a LiteLLM with 3 replicas and 4 workers each will sustain: 2,000–5,000 RPS routing to a vLLM backend, with 80–150 ms of overhead on the full path (auth + rate limit + cache check + propagation). If more is needed, scaling the router with KEDA on litellm:requests_per_second is trivial.

For larger clusters (16+ GPU nodes), consider the vLLM Production Stack router or the NVIDIA Dynamo router, which are more complex but squeeze out the prefix-aware routing and token-aware LB that LiteLLM OSS does not cover. For multi-region clusters, Envoy AI Gateway with Istio Service Mesh is the standard choice.

What we have not covered (upcoming articles)

Deep comparison of LiteLLM vs vLLM PStack vs Dynamo, with prefix-aware benchmarks on a real on-premise cluster.
Semantic cache with Redis Stack + RedisVL: hit rate, false positives, TTL policy.
Multi-region routing: how the router decides between the DC1 and DC2 clusters based on latency, health and load.
AI Gateway specific features: token-bucket cost-based rate limiting (penalising long prompts), guardrails policy engine in the router.
Migration path: how to introduce a router into a cluster that already has clients pointing straight at the vLLM service, with no downtime.

References

LiteLLM project — litellm.ai (documentación de Proxy, routing strategies, semantic cache).
vLLM Production Stack — github.com/vllm-project/production-stack (router con prefix-aware nativo).
NVIDIA Dynamo — developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/ (router production-grade con disaggregated-aware).
Envoy AI Gateway — gateway.envoyproxy.io/docs/tasks/ai-gateway/ (proyecto en gestación dentro de Envoy).
Kong AI Gateway — konghq.com/products/kong-ai-gateway (proxy enterprise con plugin LLM).
KGateway — kgateway.dev (alternativa CNCF en gestación).
Zheng et al. — SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) — RadixAttention y prefix caching.
vLLM project — Automatic Prefix Caching (docs.vllm.ai/en/latest/features/automatic_prefix_caching.html).
Patel et al. — SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024) — base teórica del routing prefill/decode aware.

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

Mon, 01 Jun 2026 15:30:00 +0200

This post complements those on LLM tracing with OpenTelemetry GenAI (the tracing layer above the metrics), Capacity planning (what was sized and what must be watched) and Continuous batching (the mechanism that explains several of the engine metrics).

TL;DR

Observability for an LLM inference cluster is built on two complementary sources: the GPU hardware metrics exposed by the DCGM (Data Center GPU Manager) Exporter, part of the NVIDIA GPU Operator, and the metrics from the inference engine (vLLM, SGLang, TensorRT-LLM) exposed on a Prometheus-compatible /metrics. Neither is enough on its own. The classic nvidia-smi metric called GPU utilization is misleading for LLMs: it reads high whenever any kernel is running, without distinguishing tensor cores burning from SMs waiting on HBM. The full cockpit has twelve DCGM metrics in four families (compute: DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE; memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL; thermal-power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS; health: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_RETIRED_DBE) and five vLLM engine metrics (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds). Each one has a defensible green/amber/red threshold, a PromQL query for alerting, and at least one common false reading that confuses the junior operator. The six critical alerts that any production cluster must fire are: HBM > 92 %, thermal or power throttle, XID error, ECC double-bit, KV cache pool > 95 %, and TTFT P95 outside SLO for 5 minutes. The point of having this panel: that the operator on shift diagnoses the origin of a degradation in under five minutes, without opening an SSH console to the GPUs. When that holds, the cluster has moved to professional operation; until then, it is operated on intuition.

You are here: OBSERVE (the other half of tracing)

Tracing, already covered in LLM tracing with OpenTelemetry GenAI, answers what happened in this specific request. Metrics answer what is happening across the cluster in aggregate. They are complementary: an alert on the metrics side tells you “the cluster is degrading”, tracing tells you “and here is the specific trace that proves it”. A cluster without tracing but with metrics operates; a cluster without metrics but with tracing does not operate, it debugs.

The analogy: the cockpit of a modern aircraft

In a modern commercial aircraft, the pilot’s instrument panel has more than 70 active indicators. If there were only one, the altimeter, say, the aircraft would fly into the ground at the first moment of low visibility. You need the altimeter and the attitude indicator, and the airspeed indicator, and the turn indicator, and the fuel gauge, and the oil pressure gauges for each engine, and the turbine exit temperatures. Each answers a different question. And together they cover the operational question: is the aircraft healthy, is it where it should be, and is it going where we want?

Observability for an LLM inference cluster works the same way. A single metric, “GPU utilization 99 %”, answers nothing useful. It is like looking only at the car’s odometer to diagnose why the engine is making a noise. The full cockpit is twelve instruments on the hardware side plus five on the inference engine side, organised into families that answer different questions:

Compute and efficiency: are the tensor cores doing the work we expect, or are they waiting?
Memory: is there VRAM left for new requests, or are we on the edge of OOM?
Thermal and power: is the hardware healthy, or is it silently limiting throughput?
Health and errors: is there hardware degradation under way (ECC, XID, NVLink)?
Inference engine: is the queue growing, is the KV pool saturated, is the SLO being met?

The first four answer “is the GPU fine?”. The fifth answers “is it delivering the service we promised?”. The two questions are different and both must have an answer at a glance.

Why `nvidia-smi` `GPU-Util` misleads with LLMs

The classic metric that appears in nvidia-smi as GPU-Util corresponds to DCGM_FI_DEV_GPU_UTIL. Its official definition: “percentage of time during which one or more kernels were executing on the GPU”. The problem with LLMs: the decode phase is memory-bound, not compute-bound. When the inference engine decodes token by token, the GPU spends 90 % of the time waiting for the HBM to finish delivering the model weights and the KV cache. There is a kernel running (the HBM read), so GPU-Util reports values close to 100 %. But the tensor cores are idle: the bottleneck is memory, not compute.

Practical result: the operator sees “GPU-Util 99 %” in Grafana and assumes “GPU saturated, no more load can go in”. But the reality may be “compute at 25 %, HBM saturated at 95 %”, which changes the operational decisions (quantisation, batch size, parallelism). The classic metric lies by oversimplifying.

The right move is to look at the three DCGM profiling metrics of the _FI_PROF_* subsystem:

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps over the maximum per SM. Is there parallel work?
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores effectively active. Is compute working?
DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring. Is memory saturated?

A typical decode-bound GPU running Llama 70B on an H100 shows: SM occupancy 35–55 %, tensor active 15–30 %, DRAM active 80–95 %. That is the real “saturated GPU” for LLMs. The three together tell the regimes apart; none of them alone says anything actionable.

How they are wired up in production

The platform side is covered in Five maturity levels (level 4, the GPU plane) and Seven deployment phases (phase F5). For the observer, the key pieces are:

NVIDIA GPU Operator. Helm manifests that deploy on each GPU node: drivers, container toolkit, MIG manager and DCGM Exporter. The last one exposes /metrics in Prometheus format with all the DCGM_FI_* listed above. It is scraped from the cluster’s internal Prometheus.

Inference engine. vLLM exposes /metrics on port 8000 (default) with vllm:* metrics. SGLang exposes it too, with the sglang: prefix. TensorRT-LLM exposes it through Triton Inference Server with the nv_inference: prefix. The basic naming convention is similar across the three engines; the thresholds and queries in this post assume vLLM, but they translate.

ServiceMonitor / PodMonitor. The Prometheus operator resource that says what to scrape. Minimal example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: vllm-inference
spec:
 selector:
 matchLabels: { app: vllm }
 podMetricsEndpoints:
 - port: metrics
 interval: 15s

Dashboards. The NVIDIA operator publishes reference Grafana dashboards for DCGM in nvidia/dcgm-exporter (the official repo). vLLM publishes one in vllm-project/vllm (the examples/ folder). Both work as a base; each team adds the panels specific to its own SLO.

The twelve DCGM metrics organised by family

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps per SM over the maximum possible. Value between 0 and 1.

Green: 0.30–0.70 (typical LLM decode regime).
Amber: < 0.20 sustained (batch too small, GPU under-used in parallelism).
Red: 0.95 sustained with low DRAM_ACTIVE (pathological kernel saturating the SMs).

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores executing. The key metric for “is compute producing?”.

Green in prefill: 50–80 %.
Green in decode: 15–30 % (decode is memory-bound, this is not a symptom of a problem).
Red: < 5 % sustained in prefill, or the engine is not using the tensor cores at all (bad config, incompatible format).

DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring data. The key metric for detecting memory saturation.

Green in decode: 60–85 %.
Amber: > 90 % sustained (HBM is a firm bottleneck, which explains high TPOT).
Red: > 95 % sustained with KV cache pool < 70 % (something other than the engine is asking for HBM; investigate leaks).

Family 2 — Memory

DCGM_FI_DEV_FB_USED: Frame Buffer (HBM) used, in MiB.

Green: 70–85 % of total.
Amber: 86–92 %.
Red: > 92 % (risk of OOM on the next paged-attention allocation).

PromQL for the cluster-wide percentage: 100 * sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_TOTAL).

DCGM_FI_DEV_FB_FREE: free Frame Buffer. Complementary to the previous one; useful for absolute alerts (< 4096 MiB free).

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: aggregate NVLink bandwidth in MB/s. For TP (tensor parallel) topologies that cross GPUs over NVLink, this metric reveals whether the parallelism split is saturating the bus.

Green: varies with topology. On 4×H100 SXM with NVLink 4.0, theoretical capacity is 450 GB/s per GPU. A typical TP=4 regime: 50–150 GB/s sustained.
Red: > 90 % of capacity sustained (check whether the model would fit with lower TP or with pipeline parallel).

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP: die temperature in °C.

Green: < 75 °C.
Amber: 75–82 °C.
Red: > 83 °C (close to the H100’s automatic thermal throttle; check ventilation, air flow, rack inlet temperature).

DCGM_FI_DEV_POWER_USAGE: draw in watts. For the H100 SXM, nominal TDP is 700 W. Useful for three things: spotting an unusually low workload (suspect idle or a stall), billing real energy cost, and firing alerts if the draw approaches the PDU limit.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: encoded bitmap with the active throttle reasons. This is the metric that silently explains TPOT degradations.

Relevant bits:

0x0000000000000001: Idle (not a problem).
0x0000000000000002: App clocks setting.
0x0000000000000004: SW Power Cap (software limit, e.g. from nvidia-smi -pl).
0x0000000000000008: HW Slowdown.
0x0000000000000010: Sync Boost (NVIDIA Sync).
0x0000000000000020: SW Thermal Slowdown (software thermal limit).
0x0000000000000040: HW Thermal Slowdown (hardware thermal limit, an emergency).
0x0000000000000080: HW Power Brake Slowdown (PSU voltage drop).
0x0000000000000100: Display Clock Setting.

Any throttle other than Idle with a sustained value > 0 is an alert. TPOT degradation with DRAM_ACTIVE already high and thermal throttle active is the classic “the rack is badly ventilated, it is not the engine’s fault”.

Family 4 — Health

DCGM_FI_DEV_XID_ERRORS: cumulative counter of driver XID errors. XIDs are critical event codes that NVIDIA documents exhaustively (XID 13: graphics engine exception; XID 31: GPU memory page fault; XID 43: reset channel verif error; XID 79: GPU has fallen off the bus; XID 95: uncontained ECC error; and so on). Any increment is an immediate alert: many XIDs require a node reset or an RMA of the GPU.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC errors (uncorrectable). Unlike single-bit ones (which ECC corrects silently and which are counted in DCGM_FI_DEV_ECC_SBE_*), double-bit errors corrupt data. Any value > 0 is a critical alert: the GPU must be drained and inspected.

DCGM_FI_DEV_RETIRED_DBE: physical HBM pages retired because of accumulated double-bit errors. NVIDIA retires faulty pages automatically to prevent future corruption. More than 4–8 retired pages on one GPU suggests silicon degradation: document it and plan a replacement in the next maintenance window.

The five inference engine metrics (vLLM)

The DCGM metrics answer “is the GPU healthy?”. The engine metrics answer “is the service meeting the SLO?”. Without them, you know the hardware works but you do not know whether the clients are happy.

vllm:num_requests_running: requests currently in the batch. If it reaches the configured --max-num-seqs and does not come down, the engine is saturated on concurrency (check VRAM and rebalance through the autoscaler, see Autoscaling LLM on Kubernetes).

vllm:num_requests_waiting: requests queued, not yet in the batch. Any value > 0 sustained over minutes means the cluster is not scaling with the load. This is the primary metric for HPA.

vllm:gpu_cache_usage_perc: % of the KV cache pool in use.

Green: 50–80 %.
Amber: 80–92 %.
Red: > 92 % (risk of preempt-on-OOM: vLLM will drop requests to free memory, which raises TTFT visibly).

vllm:time_to_first_token_seconds: histogram of TTFT per request. Consumed as histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket[5m]))). Compared against the TTFT P95 SLO, it fires the primary service alert.

vllm:time_per_output_token_seconds: histogram of TPOT. Equivalent to the previous one but for streaming smoothness. Compared against the TPOT P95 SLO, it fires the secondary alert.

The six alerts that must page in production

Any serious production cluster fires these six alerts to a channel with an on-call rotation. Without them, the SLO is met by luck, not by process.

groups:
 - name: gpu-llm-critical
 rules:
 - alert: GpuHbmNearOom
 expr: 100 * (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 92
 for: 2m
 labels: { severity: critical }
 annotations:
 summary: "HBM on {{ $labels.gpu }} at {{ $value }}% — OOM risk"

 - alert: GpuThermalOrPowerThrottle
 expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0) and ignoring(reason) (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 1)
 for: 1m
 labels: { severity: warning }
 annotations:
 summary: "GPU {{ $labels.gpu }} throttling (reasons={{ $value }})"

 - alert: GpuXidErrorDetected
 expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
 labels: { severity: critical }
 annotations:
 summary: "XID error on GPU {{ $labels.gpu }} — investigate immediately"

 - alert: GpuEccDoubleBit
 expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
 labels: { severity: critical }
 annotations:
 summary: "ECC double-bit on GPU {{ $labels.gpu }} — drain node"

 - alert: VllmKvCachePoolNearFull
 expr: vllm:gpu_cache_usage_perc > 0.95
 for: 3m
 labels: { severity: warning }
 annotations:
 summary: "KV cache pool > 95% on {{ $labels.instance }}"

 - alert: VllmTtftP95OutOfSlo
 expr: histogram_quantile(0.95, sum by(le, instance)(rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.5
 for: 5m
 labels: { severity: warning }
 annotations:
 summary: "TTFT P95 over SLO ({{ $value }}s > 1.5s)"

These six cover 80 % of the incidents that affect the SLO. The remaining 20 % demand investigation with tracing (see LLM tracing with OpenTelemetry GenAI).

Master table: thresholds and queries

Metric	Green	Amber	Red	Base query (PromQL)
SM occupancy	0.30–0.70	0.15–0.30	< 0.10 sustained	`DCGM_FI_PROF_SM_OCCUPANCY`
Tensor active (decode)	15–30 %	< 10 %	< 3 %	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
DRAM active	60–85 %	85–95 %	> 95 % with low KV	`DCGM_FI_PROF_DRAM_ACTIVE`
FB used	70–85 %	86–92 %	> 92 %	`100 * DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL`
NVLink BW	< 70 % cap	70–90 % cap	> 90 % cap	`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`
GPU temp	< 75 °C	75–82 °C	> 83 °C	`DCGM_FI_DEV_GPU_TEMP`
Power usage	< 90 % TDP	90–98 % TDP	> 98 % TDP	`DCGM_FI_DEV_POWER_USAGE`
Throttle reasons	0 or Idle	App/SW	HW Therm/Power	`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`
XID errors	no change	—	any delta	`increase(DCGM_FI_DEV_XID_ERRORS[5m])`
ECC DBE	0	—	> 0	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
Retired pages	< 4	4–8	> 8	`DCGM_FI_DEV_RETIRED_DBE`
KV cache used	50–80 %	80–92 %	> 92 %	`vllm:gpu_cache_usage_perc`
Requests waiting	0	1–5 sustained	> 10 sustained	`vllm:num_requests_waiting`
TTFT P95	< SLO	80–100 % SLO	> SLO	see alert query above
TPOT P95	< SLO	80–100 % SLO	> SLO	`histogram_quantile(0.95, sum by(le)(rate(vllm:time_per_output_token_seconds_bucket[5m])))`

Three pitfalls that confuse the junior operator

Pitfall 1 — “GPU-Util at 99 % = saturated”. As explained at the start, DCGM_FI_DEV_GPU_UTIL lights up with any kernel. The right move is to look at the three _PROF_* metrics (SM occupancy, tensor active, DRAM active) together. GPU util 99 % + tensor active 8 % + DRAM active 92 % means “saturated by memory, not compute”; GPU util 99 % + tensor active 75 % + DRAM active 50 % means “saturated by compute, prefill heavy”. The two situations call for different levers.

Pitfall 2 — confusing single-bit ECC (SBE) with double-bit (DBE). Single-bit errors are corrected silently and are unavoidable in any HBM under load (cosmic radiation, voltage fluctuations). An SBE counter growing slowly is not an alert, it is physics. DBE is: it corrupts data. Telling the two metrics apart avoids false alarms and false negatives in equal measure.

Pitfall 3 — alerting on num_requests_waiting > 0 without context. An instantaneous value of 1 or 2 during a spike is normal. What matters is a sustained queue: use for: 5m with a threshold of 3–5. Without that window, the system floods the alert channel with noise.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter deployed through the NVIDIA GPU Operator, one DaemonSet per GPU node.
Internal Prometheus with 30-day retention for high-frequency metrics, 1 year for downsampled ones (Thanos/Mimir if the volume justifies it).
Grafana with three standard dashboards: GPU hardware (DCGM), engine (vLLM), SLO (TTFT/TPOT/RPS against written targets).
Alertmanager with an on-call rotation and rate limiting through silences grouped by node.
Cardinality kept under control: gpu (local id), node, pod, model. Do not add request_id or other high-cardinality labels to metrics; that is tracing’s job.

Estimated volume for a 16-GPU cluster scraped every 15 s: around 2 million samples/min, around 25 GB/day of raw Prometheus. Manageable with one Prometheus per cluster plus retention; if the team scales beyond 64 GPUs, consider a Thanos sidecar or VictoriaMetrics. See OSS LLMOps tooling catalogue for equivalent alternatives.

What we have not covered (upcoming articles)

Tracing of LLM workloads: already covered in LLM tracing with OpenTelemetry GenAI.
Autoscaling based on these metrics: see Autoscaling LLM on Kubernetes.
Incident response runbooks: how each of these alerts translates into concrete action (drain, restart, RMA, scaling, rollback).
Cost accounting: using DCGM_FI_DEV_POWER_USAGE and vllm:request_success_total for per-tenant cost showback.
Multi-tenant fairness monitoring: when several tenants share a cluster, which metrics detect that one of them is hogging the KV cache.

References

NVIDIA — DCGM Exporter (repo nvidia/dcgm-exporter, documented metrics and units).
NVIDIA — DCGM Field Identifiers reference (complete list of DCGM_FI_*).
NVIDIA — XID Errors documentation (catalogue of XID codes and remediation procedures).
NVIDIA — NVIDIA GPU Operator (official Helm chart).
vLLM project — examples/production_monitoring/ (reference PromQL and Grafana dashboards).
Prometheus — Histogram and summary best practices (for building defensible percentile queries).
NVIDIA — H100 Tensor Core GPU datasheet (TDP, HBM bandwidth, NVLink capacities).

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

Seven greenfield deployment phases for an on-premise LLM platform: from hardware in the room to the first production token

Sun, 31 May 2026 08:00:00 +0200

TL;DR

The two previous posts in this architectural trilogy settled the pieces: the seven layers of the LLM inference stack describe the components above the cluster, and the five maturity levels of the platform describe the strata below. This post settles the when: in what order each thing gets deployed when you start from zero, hardware bought, racks installed, physical cabling done, and you want to reach a cluster serving the first production token to a customer. Seven nominal phases F0 to F6 with no calendar commitments, organised by technical dependencies (you do not enter F3 without the F2 gate) and with an identifiable critical path. F0 hardware inventory and electrical/network connectivity. F1 bare metal OS + drivers + container runtime. F2 Kubernetes cluster with CNI and Ceph storage operational. F3 GitOps and infrastructure observability. F4 identity, TLS, secrets and policies. F5 GPU platform with LLM-aware observability. F6 the LLM stack operational and open to production traffic. For each phase: what gets built, what has to be ready first (dependencies between phases), the gate that validates its closure, and the typical trap that delays the critical path. The thesis: an on-premise LLM platform sinks far more often from sequencing badly than from choosing badly. The tools have all been invented; the order is the only thing every team rediscovers.

You are here: the seven phases and their dependencies

The phases do not run in pure series. F2 and F3 can be started at the same time to speed things up (installing Kubernetes and preparing the GitOps repo in parallel). F4 can overlap with the tail end of F3. F5 waits for F4 to close because GPU pods demand NetworkPolicy and RBAC from day one. F6 is a single atomic step: the cluster goes into production or it does not.

The red arrows are the critical path: the sequential bottleneck that no parallelism can shorten. The grey arrows are dependencies that admit partial overlap. Recognising where to overlap and where not is the difference between a three-month deployment and a six-month one for the same perimeter.

The analogy: the expedition to an eight-thousand-metre summit

An expedition to a high alpine summit is not a long trek. It is a series of camps that are set up in order, each with its altitude, its function and its validation gate: if you do not acclimatise properly at base camp, you cannot go up to C1 without risk; if C2 does not have its kitchen and its radio running, you cannot send people higher; if the summit push is attempted without porters in the high camps, there is no safe descent.

Greenfield deployment of an LLM platform works identically. F0 is the arrival of the material at base camp: boxes, sponsors, permits, first review. F1 is setting up an operational base camp: kitchen, tents, generator. F2 is the climb to C1: there is real altitude now (a k8s cluster running) and you breathe differently. F3 is C2: it adds communications, planning and operational acclimatisation. F4 is C3, the last night before the push: team roped up, oxygen ready, every protocol verified. F5 is the day of the summit push, intense effort, thin margins. F6 is the summit and the start of a safe descent: from here on the expedition is in day-to-day operation, no longer in construction.

The analogy holds two useful lessons: you do not skip a camp (going straight from base camp to the summit kills the team), and the gates are technical, not emotional (if the barometer says storm, you do not set out, however much enthusiasm there is). The platform team that follows those two rules reaches the summit. The one that negotiates them does not.

F0 — Hardware in the room: base camp

What gets built in this phase. An inventory of the hardware received (servers, switches, PDUs, BMC), racks assembled, electrical and data cabling terminated, physical labelling of each machine (rack/U/function), connectivity to the corporate network, management IPs assigned, BMC reachable over VPN with MFA, first ping to each node from the bastion.

Dependencies. None technical, this is the phase before the software. There are procurement dependencies (servers bought, switches bought), civil works dependencies (a room with enough cooling, a raised floor) and administrative ones (datacenter access for the technicians).

Validation gate that closes F0.

Every node appears in the inventory with (hostname, MAC, management IP, data IP, rack, U, function, owner).
Each node’s BMC responds to ipmitool power status and to the HTTPS UI from the management VPN.
The top-of-rack switch has its configuration versioned in git (even if there is no cluster GitOps yet, the switch configs are).
A command for h in $(cat hosts); do ping -c1 -W1 $h.mgmt; done returns 100 % success.

Typical trap. Cabling “however it comes out” with no physical labelling and no diagram. When F4 arrives and a NetworkPolicy has to be troubleshot, not knowing which physical interface carries which VLAN doubles the diagnosis time of every incident for ever.

Why F0 does not overlap with F1. Until every server has a management IP and a live BMC, OS bootstrap cannot be automated. Every hour invested in F0 saves hours in every later phase. It is the phase with the best ROI in the project and the only one that admits no shortcuts.

F1 — Bare metal: the operational base camp

What gets built. The operating system image (stable Debian or Ubuntu LTS) provisioned via PXE or cloud-init with the cloud-config versioned in git. Each node has: a coherent hostname, LVM partitions, kernel ≥ 6.6, the containerd container runtime, NVIDIA drivers on the GPU nodes, chrony synchronising against your own servers, the operator’s SSH key as the only way in, and nvidia-smi passing a smoke test on the GPU nodes.

Dependencies. F0 closed. It needs the management network working so PXE responds and so the bastion can reach every node.

Validation gate that closes F1.

ansible -i inventory all -m ping returns 100 % success (or the equivalent with Salt / Pulumi / etc).
Every GPU node passes nvidia-smi showing the expected GPUs with a consistent driver across nodes.
Each node’s clock drifts by less than 50 ms from the reference NTP.
A physical reboot of a node leaves it in exactly the same state after boot (idempotency).

Typical trap. NVIDIA drivers installed by hand with apt install or with NVIDIA’s .run script. It works on day one and breaks the day of the first kernel update. The operational rule already established in the post on the five levels: the drivers end up managed by the GPU Operator in F5; whatever is done now is only so that nvidia-smi passes the smoke test, not for production.

Possible overlap. F1 can start for some nodes while F0 is still being finished on others (real greenfield rarely delivers every server on the same day). The F1 gate is per cluster, not per individual node.

F2 — Kubernetes cluster operational

What gets built. RKE2 installed with three HA control plane nodes, all workers joined (CPU and GPU), Cilium as CNI with kubeProxyReplacement enabled and a BGP control plane pointing at the ToR switches from F0, Rook-Ceph deployed on the storage nodes to cover block (RBD), filesystem (CephFS) and object (S3-compatible RGW), kubectl get nodes returning every node Ready, a first test pod with a PVC mounting and data persisting after the pod restarts.

Dependencies. F1 closed (drivers + container runtime). Switches with BGP configured (closed in F0). NVMe disks partitioned or available raw for Ceph OSDs.

Validation gate that closes F2.

kubectl get nodes -o wide shows every node Ready with the expected Kubernetes version.
A Deployment with replicas=3 and per-node antiAffinity starts and the pods land on different nodes.
An RWO PVC (RBD) creates a volume, the pod writes data, the pod is deleted, another pod mounts it and reads the data.
An RWX PVC (CephFS) does the same with two pods writing simultaneously.
An RGW bucket via s3cmd or mc accepts put and get over TLS.
Hubble (the read side of the CNI) shows flow logs between two pods in different namespaces.
Chaos test: draining a non-GPU worker node; the workloads are rescheduled automatically.

Typical trap. Starting to kubectl apply real workloads in F2 without GitOps. The backlog of things-applied-by-hand grows faster than the capacity to migrate it to git afterwards. The rule: in F2 only the cluster prerequisites are applied (CNI, CSI, default storage class). Any application workload waits for F3.

Possible overlap. F2 ↔ F3. While the cluster is being built, the GitOps repo is prepared in parallel (directory structure, first Helm releases). When F2 closes, Flux is plugged into the repo and everything that was going to be kubectl apply is already a reconciled manifest.

F3 — GitOps and infrastructure observability

What gets built. Forgejo deployed first (it is a prerequisite for everything that follows). A gitops-infra repo with the initial structure (apps/, infrastructure/, tenants/, clusters/). Flux installed and reconciling that repo. The prerequisite workloads applied by hand in F2 are moved to the repo and reconciled (no operational kubectl apply remains). VictoriaMetrics + vmagent scraping metrics. Grafana with initial dashboards (USE/RED + cluster + Ceph + Cilium). Loki receiving logs via vector/fluent-bit. Alertmanager + Keep routing alerts to a chat channel. Barman Cloud backups for Postgres (future CNPG) and scheduled Ceph snapshots.

Dependencies. F2 closed. An RGW bucket to store backups (covered by the Ceph from F2).

Validation gate that closes F3.

A change applied to the repo is reflected in the cluster in under 5 minutes with no manual intervention.
A change applied with kubectl edit directly to the cluster is detected by Flux and reverted (binding drift detection, not merely observational).
Grafana shows dashboards for the cluster, Ceph, Cilium and GPU nodes (DCGM does not arrive until F5, but basic node metrics do).
A test alert sent to Alertmanager reaches the chat channel in under 1 minute.
Restoring a Postgres backup on a temporary cluster returns coherent data (the test defines the real RPO).

Typical trap. Having Helm charts in git but still applying them with helm install from the terminal. That is level 1 dressed up as level 2. F3 only closes when Flux is the only authority that applies changes and humans edit the repo, not the cluster.

Possible overlap. F3 ↔ F4. While F3 is being closed, the Defguard and cert-manager manifests can be prepared in the repo. When they reconcile they have somewhere to land.

F4 — Identity, certificates, secrets, policies

What gets built. Defguard deployed with its dedicated Postgres (CNPG). An initial realm with the platform operators enrolled with MFA and WireGuard. OIDC integrated into kube-apiserver (--oidc-issuer-url, --oidc-client-id, --oidc-username-claim), into Forgejo, into Grafana, into Alertmanager, one single SSO. cert-manager installed with an internal CA issuing internal certs for mTLS and with Let’s Encrypt ACME for edge certs. SOPS configured with a KMS (it can be a physical HSM, an age key in a vault, or an external Vault) and External Secrets Operator syncing secrets into the cluster. Kyverno deployed with initial policies in audit mode for a week, then promoted to enforce. A default-deny NetworkPolicy applied to every existing namespace. Tetragon enabled for runtime security. The kube-apiserver audit log sent to Loki with long retention.

Dependencies. F3 closed (Flux applies the manifests, VM/Loki ingest metrics and logs).

Validation gate that closes F4.

kubectl with a shared admin kubeconfig stops working; every operator uses their own OIDC token with MFA.
A secret in plain data: in a commit is rejected by the pre-commit hook (or by Kyverno admission).
A pod without securityContext.runAsNonRoot=true is rejected by Kyverno at admission.
A deliberately wrong NetworkPolicy (allow-all) in a tenant namespace is rejected.
An audit of the last day returns the complete list of actors and changes (minimum regulatory footprint).
Basic internal pen-test: an attacker with a forged kubeconfig fails MFA and is recorded.

Typical trap. Kyverno in permanent audit mode because “we do not want to break production workloads”. F4 closes when the policies are in enforce. Until then, you are still in F3 wearing an F4 face.

Why F4 does not overlap with F5. F5 introduces GPU pods that move a lot of VRAM and a lot of compute. Without default-deny NetworkPolicy, without OIDC RBAC, without Kyverno blocking insecure configurations, the GPU pods are the juiciest attack surface in the cluster. Any compromise in F5 with F4 not closed is near-total access to the expensive hardware.

F5 — GPU platform with LLM-aware observability

What gets built. NVIDIA GPU Operator via Flux with the driver version decided in F1 (no longer touched by hand). DCGM Exporter exposing GPU metrics to VictoriaMetrics. MIG manager configured for the nodes where it makes sense (for example, on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs and embeddings). Topology Manager with the single-numa-node policy. KEDA with the Prometheus scaler installed and an example ScaledObject pointing at a vLLM metric (vllm:num_requests_running). OpenTelemetry Collector with OTLP receivers, attributes processors (enriching spans with tenant_id, priority_tier), exporters to Langfuse and to Tempo. LeaderWorkerSet API enabled for tensor parallel topologies. OME (Operator Model Engine) or vLLM Production Stack deployed as a controller, still with no models loaded.

Dependencies. F4 closed (the GPU pods inherit default-deny NetworkPolicy, OIDC RBAC and Kyverno policies).

Validation gate that closes F5.

A test pod requesting nvidia.com/gpu: 1 is scheduled on the right node and nvidia-smi from inside the container sees the right GPU (whole or a MIG slice).
DCGM Exporter exposes metrics in Grafana (utilisation, VRAM, temperature, NVLink bandwidth) for every GPU.
A test vLLM Deployment starts with a small model (for example, a 7B FP16) loaded from Ceph RGW.
An OpenTelemetry span generated by that vLLM reaches Langfuse with correct gen_ai.* attributes.
KEDA scales the test Deployment from 1 to N replicas under synthetic load and back to 1 when it stops.
A GPU Operator upgrade to a new version drains and reschedules the GPU pods with no loss of service.

Typical trap. Loading the large model “just to try” before DCGM and OTel are green. When something fails, there will be no metrics to distinguish OOM, thermal throttling, a driver mismatch or a network problem, and you diagnose blind. The rule: small model first, golden path green, then the large model.

Possible overlap. None with F6. F6 is atomic.

F6 — LLM stack in production

What gets built. The seven layers of the inference stack described in the corresponding post, deployed in this order:

Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets). Some components already existed from F3 as data; here they specialise for RAG with their initial collections and schemas.
Embeddings + reranker (Infinity with multilingual-e5-large, TEI with bge-reranker-v2-m3). This is the layer that must be green before any large model, because RAG depends on it.
LLM inference (vLLM Production Stack with the general LLM and the code LLM). It loads models from Ceph RGW. The initial multi-LoRA pool is empty.
Gateway (Envoy AI Gateway) with Defguard OAuth, routing by body.model, rate-limit per tenant. This is the point that opens traffic to the outside.
LLM-aware observability (Langfuse plugged into the OTel from F5).
GitOps control plane and dependency tracking were already active since F3 and F4 respectively; here you simply add the catalogue of the new LLM services to them.

Dependencies. All the previous phases closed.

Validation gate that closes F6.

A curl to the public endpoint with a Defguard bearer token receives a correct chat completion response in technical Spanish, with trace_id propagated.
The trace appears in Langfuse with complete gen_ai.* attributes, a latency breakdown and its own tenant_id.
A 5 % traffic canary to the new model over 24 h degrades neither quality nor latency metrics.
A controlled traffic burst triggers KEDA, the replicas scale, and P95 latency stays within budget.
A deliberate failure of one vLLM pod does not affect endpoint availability (replicas + reschedule).
The internal operator demonstrates the complete path for revoking a tenant’s access in under 5 minutes (Defguard → Kyverno → closing the NetworkPolicy).

Typical trap. Opening real customer traffic before having the incident runbook signed off, the SLO negotiated and the continuity plan tested. F6 is technically closed; operationally, the platform is still an experiment until the first real postmortem proves the team knows how to respond.

The maths that matter: relative weight of effort per phase

Without committing to calendar weeks, we can quantify the relative weight of the engineering effort per phase in a typical greenfield. The curve is not uniform:

$$ \text{effort}_{F_i} \approx \text{base}_i \cdot (1 + \epsilon_i) $$

where $\text{base}_i$ is the nominal effort and $\epsilon_i$ is the surprises factor (wrong cabling, incompatible drivers, badly issued certificates, version conflicts). The table below gives the nominal relative weight and the typical surprise factor observed:

Phase	Nominal weight	Typical surprise factor ε	Mean effective weight
F0 — Hardware	8 %	0.5 (1× to 2×)	12 %
F1 — Bare metal	6 %	0.3	8 %
F2 — k8s cluster	12 %	0.4	17 %
F3 — GitOps + obs	14 %	0.5	21 %
F4 — Identity + policies	18 %	0.7	31 %
F5 — GPU plane	10 %	0.4	14 %
F6 — LLM stack live	8 %	0.3	10 %
Buffer / integration	24 %	—	—

Two operational observations. F4 concentrates more surprises than any other phase (OIDC federation across four or five apps with different configurations, Kyverno policies that knock over legitimate workloads, secrets broken by poorly tested encryption). F0 has a high surprise coefficient relative to its size because any cabling or labelling error is discovered late and paid for dearly. The two practical consequences: plan F4 with generous margin and do not skimp on time in F0, because every hour saved there costs five later.

Critical path and slack. The critical path is linear F0 → F1 → F2 → F3 → F4 → F5 → F6. The only real slack is the overlaps already identified:

F2 ↔ F3 (slack ~30 %): prepare the repo and initial dashboards while the cluster is being built.
F3 ↔ F4 (slack ~20 %): identity manifests ready as F3 closes, applied immediately.
Within F4: policies in audit mode running in parallel with the Defguard setup.

Nothing shortens the critical path by more than about 15 % of the total. Anyone promising a productive greenfield in half a reasonable timescale is selling something else: probably skipping F4, or loading F6 with an F5 that is green-but-not-validated.

Final diagram: the complete deployment schedule

The schedule is not decorative: each row defines what gets built in its phase and the gate that closes it. A phase is not considered finished until its gate is green. A phase with a yellow gate drags every later one; trying to jump to the next with a partially met gate is what produces, several weeks later, the incident that forces you to “go back to F4 with production running”, the most expensive situation in the whole cost matrix of the post on the five levels.

Typical planning mistakes

Patterns that delay or sink the greenfield deployment, regardless of the tools chosen:

1. Buying the LLM before the cluster. Starting the project with “which model are we going to serve” instead of “what platform can sustain any reasonable model”. The model is an interchangeable parameter; the platform is not.

2. Underestimating F0. “The network team does that.” Yes, but the output of F0 is consumed by every later phase. If the network team delivers late, the whole project is late, and nobody had marked it as the critical path.

3. Overlapping F4 with F5 “to save time”. It is the one dependency with no slack. If you try to overlap, F5 ends up operating with policies in permanent audit (you are not in F4) or without OIDC integrated (operators with a shared kubeconfig touching GPUs). Both antipatterns stay in production.

4. Skipping the small model smoke test in F5. “Let’s go straight for the 70B.” When something fails (and something will), there will be no baseline to diagnose against.

5. Treating F6 as “turning on vLLM”. F6 includes gateway, LLM-aware observability, runbook, SLO, continuity plan. Turning on vLLM takes five minutes; closing F6 is weeks of validation and sign-off.

6. Not defining gates in writing. If the gates are not written down, they are negotiable after the fact. “This already counts as F4” is the sentence that precedes the next six months of retrofit.

7. Assigning a phase to a single owner. Every phase needs at least two people who understand it. Staff turnover on long projects destroys knowledge; written gates + cross review preserve it.

8. Forgetting the descent route. The post focuses on going up. Day-to-day operation (the descent, in the analogy) is another story that also deserves planning: runbooks, on-call, upgrade capacity, end-of-life plan. Teams that only plan the ascent reach the summit and stay there with no oxygen.

Applied to typical on-premise hardware: 4×H100 SXM

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM per GPU node, 3 control plane nodes, 3-5 CPU worker nodes, 2 GPU worker nodes), the temporal split of the work is distributed like this:

F0 (hardware)
└─ 8 physical servers racked + switches + BMC + management IPs
 ├─ 3 nodes cp-01..03 — control plane (no GPU)
 ├─ 3 nodes worker-cpu-01..03 — CPU plane (Forgejo, Ceph, observability)
 └─ 2 nodes worker-gpu-01..02 — GPU plane (4×H100 SXM each)

F1 (bare metal)
└─ OS + drivers + containerd on all 8 nodes alike
 (NVIDIA drivers only on the 2 GPU nodes, `nvidia-smi` smoke)

F2 (k8s cluster)
└─ RKE2 control plane on cp-01..03 (HA with embedded etcd)
 workers joining: 3 CPU + 2 GPU
 Ceph OSDs on the 3 CPU worker nodes
 default pools: RBD-replicated-3, CephFS-replicated-3, RGW

F3 (GitOps + obs)
└─ Forgejo + Flux + VM/Grafana/Loki + Keep on the CPU plane
 first `gitops-infra` repo reconciles what came from F2

F4 (identity)
└─ Defguard on the CPU plane (StatefulSet with Postgres CNPG)
 OIDC in kube-apiserver, Forgejo, Grafana, Alertmanager
 Kyverno as a Deployment on the control plane

F5 (GPU plane)
└─ NVIDIA GPU Operator targets the GPU workers
 MIG manager: 1st GPU MIG 7g.80gb (= passthrough), 2nd 2×3g.40gb
 OTel Collector as a DaemonSet on the GPU plane + CPU plane
 first vLLM with a 7B FP16 model green

F6 (LLM stack)
└─ The 7 layers reconcile via Flux from a second repo `gitops-llm`
 first public endpoint with Defguard OAuth
 first production customer enrolled under an SLA

The physical distribution of the cluster exploits the isolation between planes defined in F0: the control plane never touches a GPU, the CPU plane concentrates relevant state (Forgejo, Ceph, Postgres CNPG, Langfuse, Defguard) and the GPU plane specialises as much as possible. That separation, decided in F0 before the first server is installed, conditions the success of the remaining phases. It is another reminder of why F0 matters more than it looks.

What we have not covered (upcoming posts)

This post walks the ascent route to the summit. Some pieces still deserve an article of their own:

The safe descent: day-to-day operation, runbooks per component, on-call, continuous capacity planning, the cluster upgrade cycle with no downtime.
Multi-site (the second summit): how two clusters are federated with Cilium Cluster Mesh and what extra phases that introduces. F3.5 (Cluster Mesh) and F4.5 (cross-site replication) are the missing phases.
The brownfield route: what changes when there is already a cluster with workloads. The phases stay the same, but the gates are applied retroactively and every step requires migration planning.
The real calendar cost: typical ranges in weeks for a platform team of 2-3 people, split by phase, with uncertainty bands.
The handoff to operations: how the platform is handed from the deployment team to the operations team, which documents are signed, what is inherited and what is renegotiated.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook documentation — rook.io/docs
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai

Five maturity levels of the platform under the LLM: from a Linux server to a cluster ready for vLLM

Sun, 31 May 2026 07:00:00 +0200

TL;DR

The post on the seven layers of the LLM inference stack took many pieces for granted: an operational Kubernetes cluster, GitOps reconciling, identities resolved, GPUs visible to the scheduler, observability capable of carrying gen_ai.*. Before vLLM makes any sense, you have to reach that starting point, and you reach it by levels. This post defines five maturity levels of the platform that lives under the LLM, from a bare metal server with Linux installed (level 0) to a cluster ready to run the inference layer (level 4) and the handoff to the previous post (level 5). Each level unlocks a concrete capability, running containers reproducibly, rebuilding the cluster from git, authenticating humans via OIDC, scheduling GPUs with MIG and DCGM metrics, demonstrating compliance with no manual intervention, and each has a validation test that decides whether you are really there or just telling yourself you are. For each level: which OSS pieces cover it in 2026 (Cilium, RKE2, Flux, cert-manager, Defguard, NVIDIA GPU Operator, KEDA, Trivy, Kyverno…), the deployment order within the level, the decisions that are expensive to skip, and the antipatterns that drop you a level when you thought you were higher. The thesis: moving up a level costs little effort if you do it in time, and a lot of refactoring if you try to skip it. LLM inference demands at least level 4; anyone trying to serve LLMs from level 1 or 2 ends up paying in night-time incidents what they saved on the platform.

You are here: the five levels at a glance

Before the detail, the ladder. Each rung adds a capability absent from the one below. The level’s test is the question whose honest answer says whether you are already there.

The levels are not interchangeable. A cluster at level 2 cannot run LLMs in production with any guarantee: technically the vLLM pod loads, but at the first night-time incident you discover there is no TLS, no identities, no alerting, no GPU metrics, and no way to know who changed what. Moving up a level after LLMs are already in production costs orders of magnitude more than moving up while the cluster is still empty.

The analogy: from the street stall to the starred restaurant

Picture the scale of a hospitality business. Level 0 is the street stall: a griddle, a gas bottle, a cook improvising. It can sell food, it works, but anything that deviates from a normal day (a health inspection, an allergic customer, an order for 200 portions) sinks the business. Level 1 is the tapas bar: a properly sized kitchen, a short repeatable menu, several shifts. The cook no longer improvises daily; he works from a written menu, even if the recipes live in the head of the boss. Level 2 is the restaurant with a set lunch menu: written procedures, fixed suppliers, stock control, an incident book. If the head cook falls ill, the second can run the service without carnage. Level 3 is the restaurant with an à la carte menu and formal service: traceability of every ingredient, allergens on the menu, health certification, contracts with suppliers, mandatory staff training. Level 4 is the kitchen specialised in a complex product (sushi, haute cuisine, artisan bakery): specific tools the normal restaurant does not need (wood-fired oven, special knives, proving chamber), calibrated processes, quality metrics. Level 5 is the restaurant with a Michelin star: the whole system works, the dish is the result of the organisation, not of one person’s talent.

The analogy holds to the end, including the most interesting detail: you can operate at any level, but the promises you can keep are different. The street stall cannot promise a consistent experience to 80 diners with a booking. The cluster at level 1 cannot promise productive multi-tenant LLM service with an SLA. In both cases the problem is not the technical capability of the last component (the griddle cooks; the pod starts); it is the organisational capability of the whole system.

Let us go level by level.

Level 0 — Chaotic: the server with Linux and nothing else

The capability it gives. Running containers with docker/podman, running binaries, connecting the server to the network. The operator can SSH in, do things, and see results.

The test for the level. “If I reinstall the server from scratch, can I leave it identical to how it was in one afternoon, using only saved notes?”. If the answer is no (because the steps are in the head of whoever built it, in .bash_history, in an out-of-date wiki), you are at level 0.

Minimum pieces to settle before moving up to level 1.

Piece	Suggested decision in 2026	Why it matters when moving up
Linux distribution	Stable Debian or Ubuntu LTS	Long, predictable support, recent kernel available
Kernel	Recent LTS (≥ 6.6) with BPF and modern schedulers	Cilium/eBPF and recent NVIDIA drivers demand it
NVIDIA drivers	The version that matches the CUDA of the LLM engine you will serve	A driver/CUDA mismatch blocks vLLM before you start
Container runtime	`containerd`	CNCF standard, integrated with RKE2/kubeadm
Root filesystem	XFS or ext4 + LVM thin pools	Snapshots, online expansion
Time synchronisation	`chrony` with your own servers	TLS, correlated logs and short certificates demand it
Management network	Dedicated VLAN, ACLs on the switch	Isolate the control plane from workload traffic
Cluster network	LACP + jumbo frames + BGP (if you are going with Cilium)	Intra-node NVLink does not rescue the service network
BMC / IPMI	Out-of-band access with TLS and MFA	Recovery when the operating system does not boot

Antipatterns that pin you at level 0.

Pet servers (with their own names, configured by hand, not replaceable).
Changes applied with vi directly over /etc/... with no commit to a repo.
Deployment with docker-compose with no healthchecks and no automatic restart.
An inventory that lives in an Excel sheet nobody updates.

Deployment order within the level. System image from PXE/cloud-init with the initial configuration (LVM, hostname, network, SSH key, chrony) → bastion/jump host bootstrap → inventory in Ansible (or an equivalent declarative tool, even if it is later replaced) → NVIDIA drivers + container runtime → smoke test (a CUDA container passes nvidia-smi). At this point, the server is ready for Kubernetes to come in.

Level 1 — Repeatable: an operational Kubernetes cluster

The capability it gives. Scheduling containers with a scheduler, network abstraction between pods, persistent volumes, workload lifecycle, manual horizontal scaling.

The test for the level. “Can I lose a node and have the workloads rescheduled without human intervention?”. If yes, you are at level 1. If not, because the pods are pinned to nodes, because there are no replicas, because the PVCs do not reattach, you are still at 0 with Kubernetes on top.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Main alternative
k8s distribution	RKE2 (CIS-hardened by default, no commercial surcharge)	k3s for very small edge, plain kubeadm for custom cases
CNI	Cilium with kube-proxy replacement, BGP, Gateway API	Calico (without BGP it does not compete with Cilium in 2026)
CSI block + filesystem + object	Rook-Ceph (RBD + CephFS + S3-compatible RGW)	OpenEBS Mayastor + Garage for small deployments
Ingress	Cilium Gateway API (better to unify with the CNI)	NGINX Ingress, Traefik
Basic certs	Self-signed bootstrap	(cert-manager comes in at level 3)
Workload handling	`kubectl apply` + Helm from the terminal	No GitOps yet
Container registry	Any internal registry (or a trusted external one) with TLS	(a managed internal registry comes in at level 2)

Antipatterns that drop you to level 0.

Services deployed with kubectl apply from one person’s terminal and with the YAML saved nowhere.
Persistent volumes with no backup policy.
A “single-node cluster” as permanent production, an architectural single point of failure.
A CNI with no NetworkPolicy available or no BGP when the network requires it.

Deployment order within the level. RKE2 installed on at least three nodes for an HA control plane → Cilium installed in kube-proxy replacement mode + BGP control plane → Rook-Ceph on at least three nodes covering block (RBD) + filesystem (CephFS) + object (S3-compatible RGW) with 3× replication or Erasure Coding depending on the pool → smoke test (a Deployment with a PVC starts, the pods reschedule when a node is cordoned, the data persists).

Level 2 — Defined: the cluster rebuilds from git

The capability it gives. The cluster state lives in a repository. Every change goes through a commit. Anyone can rebuild the cluster (or an equivalent one) from the repo and the backups. Basic observability warns when something breaks.

The test for the level. “If I lose the whole cluster, can I recreate it in X hours from the repo + the backups, with no manual intervention beyond the bootstrap?”. The hours are negotiable; what defines the level is that the repo + the backups are enough, not that the person-who-knows is available.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
Forge	Forgejo (or Gitea, GitLab CE)	Self-hosted OSS, community fork of Gitea, open governance
GitOps reconciler	Flux	CNCF graduated, native multi-tenancy, lightweight
Image registry	Forgejo Container Registry	Next to the code, no extra piece
Metrics TSDB	VictoriaMetrics + vmagent	Higher throughput than plain Prometheus, long retention, PromQL compatible
Visualisation	Grafana	The de facto standard
Logs	Loki or Vector	OSS, integrated with Grafana
Alerting	Alertmanager + Keep (OSS orchestrator)	Keep adds multi-channel routing with no lock-in
DB backups	Barman Cloud (Postgres)	The standard for CNPG
Object / dataset backups	Ceph RGW multisite + CephFS snapshots	Cross-pool and cross-site

Antipatterns that drop you to level 1.

kubectl apply run in production outside the repo (undetected drift).
main branches with write permissions for humans without review.
A monolithic repo with no tenant/infra/apps separation (cross changes that cannot be audited).
Metrics kept no longer than 7 days (no SLO observable over a month).
Alerting that fires for everything (fatigue) or for nothing (silence).

Deployment order within the level. Forgejo deployed first (it is a prerequisite for everything else) → Flux installed and pointing at the manifests repo → an initial repository with the Cilium and Rook-Ceph Helm releases reconciled by Flux (replacing the level 1 kubectl apply) → VictoriaMetrics + Grafana + Loki via Helm/Flux → Postgres backups and Ceph snapshots scheduled → smoke test (tear the cluster down, restore from repo + backup, the services come back).

Level 3 — Managed: identities, certificates, secrets and policies

The capability it gives. Every human operating the cluster does so with their own identity (no shared kubeconfig), with MFA and with limited permissions. Automatic internal TLS. Encrypted versioned secrets. Policies that reject insecure configurations before they reach the cluster. A complete audit of who did what.

The test for the level. “If an attacker gets hold of an administrator’s laptop, what can they do in production?”. At level 3 the answer is “little”: MFA blocks the second factor, Kyverno policies block destructive changes without approval, the NetworkPolicies prevent lateral movement, the secrets are encrypted with an external KMS, and the audit log remains. At level 2, “everything”.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
IdP / OIDC	Defguard	Spanish OSS, WireGuard + OIDC + 2FA, multi-org
Federation with the cluster	OIDC in kube-apiserver, OIDC in Forgejo, OIDC in Grafana	Consistent SSO
Internal PKI	cert-manager + Trust Manager	De facto standard, ACME and internal CA
External ACME	Let’s Encrypt for edge certs	No cost, automated
Secrets in git	SOPS + age or an external KMS	Versionable, encrypted in the repo
Secret syncing	External Secrets Operator	Pull from KMS / Vault into the cluster
Policy as code	Kyverno (or OPA Gatekeeper)	Kyverno has a shallower learning curve
NetworkPolicy	Cilium NetworkPolicy + L7	Default deny per namespace
Runtime security	Tetragon (Cilium)	eBPF, complements NetworkPolicy with detection
Vulnerability scanning	Trivy in the CI pipeline + admission	SBOM per image, blocking of critical CVEs
Audit log	kube-apiserver with `--audit-policy-file` sent to Loki	Regulatory traceability

Minimum Kyverno policies to keep alive.

Deny :latest images or images without a sha digest.
Deny pods without securityContext.runAsNonRoot=true.
Deny pods without resources.limits (CPU + memory).
Deny Services without an owner=<team> label.
Deny changes in critical namespaces (kube-system, flux-system) without an approval label.

Antipatterns that drop you to level 2.

A kubeconfig shared between administrators.
Secrets in plain data: in a manifest committed to the repo.
NetworkPolicy absent from new namespaces by default (implicit allow-all).
kubectl edit or kubectl patch in production without going through the repo.

Deployment order within the level. Defguard deployed and enrolled with WireGuard / OIDC → OIDC integration with kube-apiserver, Forgejo and Grafana → cert-manager installed and issuing internal certificates (your own CA for mTLS, Let’s Encrypt for the edge) → SOPS configured and External Secrets Operator installed → migration of secrets from plain to encrypted → Kyverno with initial policies in audit mode, then enforce → default-deny NetworkPolicy per namespace → Tetragon enabled → smoke test (try to bypass each policy and check that admission rejects it).

Level 4 — Optimised for GPU: the cluster now knows what an H100 is

The capability it gives. The Kubernetes scheduler sees the GPUs, distinguishes them, can partition them (MIG) or multiplex them (time-slicing), expose DCGM metrics, autoscale with KEDA using metrics from the LLM workload itself (vllm:num_requests_running, vllm:gpu_cache_usage_perc), and carry traces with the GenAI semantic conventions. Everything needed for the LLM inference stack to rest on a platform that understands its nature.

The test for the level. “If I deploy a pod asking for nvidia.com/gpu: 1, does it get scheduled on the right GPU, with the right slice, with DCGM metrics exposed, with GenAI observability ready to receive spans?”. If yes, you are at level 4. If the answer requires “it depends which node and who deploys it”, not yet.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
GPU device plugin	NVIDIA GPU Operator	Deploys drivers, container toolkit, DCGM and MIG manager with one operator
HW partitioning	MIG (Multi-Instance GPU) on the H100 where it applies	Real hardware isolation, not time-slicing
GPU metrics	DCGM Exporter	SM utilisation, VRAM, temperature, throttling, NVLink bandwidth
LLM metrics	vLLM Prometheus endpoint + scrape	TTFT, TPOT, KV cache, prefix hit rate
Autoscaling	KEDA with a Prometheus ScaledObject	Scales on LLM metrics, not on CPU
LLM operators	vLLM Production Stack / OME (Operator Model Engine)	Declarative handling of models / adapters
Traces	OpenTelemetry Collector with OTLP receivers + processors + exporters	`gen_ai.*` semantic conventions (post)
LeaderWorkerSet	LeaderWorkerSet API (k8s 1.30+)	Tensor parallel topology coherent with NVLink
Topology Manager	enabled with `single-numa-node`	Pinning GPU pods to the right NUMA node

The key decision: MIG, time-slicing or passthrough.

MIG divides an H100 into 1g.10gb, 2g.20gb, 3g.40gb, 7g.80gb (slices with real HW isolation). Useful for serving several small models or reserving capacity per tenant with a guarantee. Limitation: up to 7 instances per GPU, predefined profiles.
Time-slicing shares a GPU between several pods with no HW isolation. Useful for dev/test, not for multi-tenant production with an SLA.
Passthrough assigns the whole GPU to one pod. Useful for tensor parallel across several GPUs on the same node (a large LLM with TP=4).

For a productive LLM platform, the practical rule: passthrough for the large models with TP, MIG for embeddings and small models that cohabit, never time-slicing in production.

Antipatterns that drop you to level 3.

Installing NVIDIA drivers by hand outside the GPU Operator (silent breakage when Kubernetes is upgraded).
Serving an LLM with requests.gpu: 1 without having decided on MIG / passthrough (you end up with idle GPUs from fragmentation or pods stepping on each other).
KEDA autoscaling on CPU (the classic HorizontalPodAutoscaler) on pods that are almost always at 10 % CPU but 95 % KV cache.
OpenTelemetry deployed but without the gen_ai.* semantic conventions (the traces are not LLM-aware).

Deployment order within the level. NVIDIA GPU Operator installed via Helm/Flux with the driver version that matches the chosen LLM engine → DCGM Exporter enabled and metrics visible in Grafana (NVIDIA dashboards imported) → MIG manager configured for the nodes where it makes sense (a typical mix on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs + embeddings) → OpenTelemetry Collector with attributes processors to enrich spans with your own labels (tenant_id, priority_tier) + exporters to Langfuse and to Tempo → KEDA installed with an example ScaledObject pointing at vllm:num_requests_running → vLLM Production Stack or OME to declare models as a CRD → smoke test (a vLLM Deployment declared via CRD starts, serves a token, exposes metrics, the trace reaches Langfuse, KEDA scales under synthetic load).

Level 5 — Handoff: the cluster is an LLM platform, the seven layers go on top

Once at level 4, the cluster meets the contract that the post on the seven layers assumed as its starting point. Level 5 adds no infrastructure: it adds the LLM stack proper. For completeness, the seven components of level 5 are:

Gateway (Envoy AI Gateway) — it comes first, directing traffic to LLM inference and embeddings.
LLM inference (vLLM Production Stack or OME with vLLM) — on the GPUs already discovered by the level 4 GPU Operator.
Embeddings + reranker (Infinity, TEI) — a pod separate from the LLM, already covered in the previous post.
Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets) — most of it already existed at level 2 as data; now it specialises for RAG.
LLM-aware observability (Langfuse) — it plugs into the level 4 OTel chain.
GitOps control plane — the one from level 2 remains the only legitimate authority.
Dependency tracking (Hubble flows + Otterize) — on the Cilium that already existed at level 1.

The criterion for promoting from level 4 to level 5 is not technical: it is contractual. The cluster already supports LLMs; the decision is when to open real customer traffic. Promotion demands: a green golden eval of the model, a signed incident runbook, negotiated SLOs, a continuity plan, and mapping to ENS / NIS2 / 42001 where it applies.

The maths that matter: what skipping a level costs

To quantify the thesis of the post, an order-of-magnitude estimate of the cost of moving up each level in time versus moving up after production is running. The figures are engineering time with a small platform team (2-3 people), assuming templates and prior experience.

Level	Time to build on an empty cluster	Time to retrofit with production running
0 → 1	1-2 weeks	1-2 weeks (little downstream refactoring)
1 → 2	2-3 weeks	4-8 weeks (migrating everything to git)
2 → 3	2-4 weeks	8-16 weeks (image rebuilds, secret migration, retroactive RBAC)
3 → 4	1-2 weeks	4-8 weeks (reconfiguring GPUs, moving models to MIG, instrumenting `gen_ai.*`)
4 → 5	1-2 weeks	2-4 weeks
Total 0 → 5	~10-15 weeks	~20-40 weeks if done in the wrong order

The typical multiplier observable in practice: 2× to 3× the cost if it is done in the wrong order. And that assumes it gets done at all, since many projects never get past level 2 because “the identity thing” can always wait for another sprint. By the time the incident arrives, it is too late to start.

Beyond time, the operational cost (night-time incidents, security escapes, invisible debt) grows exponentially with the gap between the actual level and the required level. A cluster at level 2 serving productive LLMs to regulated customers is a time bomb: technically it works, organisationally it does not.

Final diagram: the complete ladder with pieces

The ladder is not decorative: each level enables the next. You cannot have LLM-aware observability (level 4) without OTel deployed via Flux (level 2). You cannot have automatic internal TLS (level 3) without a root PKI that lives somewhere (registry and certificates managed from level 2). You cannot have KEDA scaling on vLLM metrics (level 4) without Prometheus / VictoriaMetrics scraping (level 2). The levels are not a conceptual hierarchy: they are a hierarchy of installation dependencies.

Typical design decisions that break progress

Mistakes seen repeatedly that drag the cluster back a level:

1. Jumping from level 1 straight to level 4. “We are in a hurry to serve the LLM, we will do the identity and GitOps stuff later.” Later is always two orders of magnitude more expensive and always arrives after the first incident.

2. Confusing Helm with GitOps. Having Helm charts is not level 2. It is level 1 with templates. Level 2 demands that a reconciler (Flux/ArgoCD) applies the charts from a repo, detects drift and warns.

3. cert-manager with no usage policy. Having auto-renewed certificates but using TLS only at the ingress, with no internal mTLS between services, leaves the TLS promise lame and turns level 3 into a cosplay of level 3.

4. NVIDIA drivers by hand. It works on day one and breaks the day of the first kernel upgrade. The rule: drivers always via the GPU Operator, never operating system packages.

5. Prometheus metrics but 7 days of retention. Without long retention (≥ 90 days) there is no honest SLO. VictoriaMetrics with a year of retention costs little more than Prometheus with 7 days, and it unlocks compliance and serious postmortems.

6. OIDC only for kube-apiserver. If Forgejo, Grafana, Defguard and vLLM each have their own auth system, you do not have SSO, you have islands. An honest level 3 demands federation.

7. Kyverno in permanent audit mode. Policies that do not reject are not policies, they are alerts. At some point you have to move to enforce. Until then, you are still at level 2 wearing a level 3 face.

8. MIG with no conscious decision on the profile. Configuring MIG with the default profile without having measured the size of the models that will cohabit leaves fragmented GPUs with slices nobody uses. The rule: MIG only if you have measured and decided the profiles in advance.

They all share one root: declaring the level without passing the level’s test. Saying “we already did GitOps” when things are still applied with kubectl edit in prod. Saying “we already did identity” when there is a shared admin kubeconfig. Saying “we are ready for LLMs” when there is no DCGM Exporter and no Langfuse plugged in.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), a reasonable setup after passing the five levels distributes the components like this:

control plane (3 nodes without GPU, hostnames cp-01..03)
├── kube-apiserver, etcd, controller-manager, scheduler
├── Flux, Forgejo, cert-manager, External Secrets, Kyverno
└── Tetragon (DaemonSet here too)

worker plane (≥ 3 nodes without GPU, hostnames worker-cpu-01..03)
├── Cilium agent (DaemonSet)
├── Rook-Ceph OSDs + MONs + MDS (CephFS) + RGW (S3)
├── VictoriaMetrics + Grafana + Loki
├── Defguard (StatefulSet)
└── Langfuse + OTel Collector

GPU worker plane (≥ 2 nodes with 4×H100 SXM, hostnames worker-gpu-01..02)
├── NVIDIA GPU Operator (driver + container toolkit)
├── DCGM Exporter (DaemonSet)
├── MIG manager (configuring the chosen profile)
├── vLLM (Deployment) — general LLM TP=4 occupies 4 GPUs (passthrough)
├── vLLM (Deployment) — code LLM TP=2 occupies 2 GPUs
├── Infinity (embeddings) — 2 replicas cohabit on 2 MIG slices
└── KEDA scaler listening to vLLM metrics

The operational rule: the control plane and the CPU plane are separated from the GPU plane. An incident on the GPU plane must not take the control plane with it (the control plane is what recovers the cluster). And the CPU plane concentrates everything that moves relevant state (Forgejo, Rook-Ceph, Postgres CNPG, Langfuse): it is the heart to protect.

The GPU hardware specialises as much as possible: GPU pods run only on GPU nodes, and GPU nodes run nothing CPU-bound beyond the operational overhead (Cilium, GPU Operator, DCGM). This is enforced with nodeSelector + taints/tolerations + a Kyverno policy that rejects pods with no GPU requests being scheduled onto GPU nodes.

What we have not covered (upcoming posts)

This post walks the vertical path upwards. Some horizontal and cross-cutting pieces still deserve an article of their own:

Multi-site active/standby: how two clusters are federated with Cilium Cluster Mesh and what changes at each level when there are two sites instead of one.
Migration between levels with real traffic: how a cluster already in production is retrofitted to the next level with no downtime.
Day-to-day operation: runbooks per level, which dashboards to look at each morning, which SLOs to define per component.
The cost plane: what each level costs in hardware, energy, engineering hours, optional OSS licences (commercial support from Rancher, Cilium Enterprise, etc.) and when each expense is justified.
Operationalised compliance: how levels 3 and 4 map to ENS High, NIS2 and ISO/IEC 42001 controls without turning the cluster into a paperwork exercise.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook-Ceph — rook.io
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
vLLM Production Stack — docs.vllm.ai/en/latest/serving/production_stack.html
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
CIS Kubernetes Benchmark
NIST SP 800-207 — Zero Trust Architecture

Anatomy of an on-premise LLM inference stack: the seven layers that have to hold each other up

Sat, 30 May 2026 16:00:00 +0200

TL;DR

The on-premise LLM inference stack is not a model server: it is a building of seven layers that hold each other up. The inference layer (vLLM / SGLang) serves tokens, but without a dedicated embeddings layer RAG does not work; without the gateway layer the client couples its SDK to the specific engine; without an LLM-aware observability layer (Langfuse + OpenTelemetry GenAI) any quality degradation goes unnoticed; without a GitOps control plane layer (Flux + Forgejo) any manual change leaves invisible debt; and without a dependency tracking layer (Hubble + intent-based policies) decommissioning a Service silently breaks applications nobody remembers were using it. This post is born from a concrete incident: a pipeline that reported status: completed and matched_jobs: 0 for days because it kept calling an Ollama that had already been scaled to zero, while a badly written except labelled the generic ConnectError as “ChromaDB indexing error” and the vector store, innocent, took the blame. Each of the three symptoms was the cry of a layer that was missing or badly designed. The body of the post walks the seven layers with their canonical OSS piece, the design decisions that break them, the sizing maths on a generic 4×H100 SXM cluster (320 GB of VRAM, NVLink) and a final diagram of the connected stack. The thesis: a stack that passes the incident test is not measured by its peak throughput, it is measured by how long it takes to shout when something drifts from the design.

You are here: the seven layers seen from above

Before going into detail, the map. The seven layers are not seven servers: they are seven responsibilities the stack has to cover. A layer can collapse into one pod (gateway) or spread across several components (observability = traces + metrics + flow logs). What it cannot do is be missing.

Layers 1–4 sit on the request path: if they fall, the client notices in seconds. Layers 5–7 sit on the path of design and operations: if they fall, there is no immediately visible error, and that is why they are the ones that produce silent incidents, which is how most serious incidents end up. This post argues that the quality of the stack is measured in layers 5, 6 and 7, because 1–4 are commodities where everyone picks roughly the same pieces.

The analogy: the office building with shared services

Picture a twelve-storey office building with several tenants. It has a concierge desk (gateway: filters who comes in), it has lifts (LLM inference: they move the heavy load), it has stairs and a goods lift (embeddings: light but constant traffic, hardly anyone notices them until they break), it has plumbing and water tanks (vector store: what actually holds the state), it has a distribution board and sensors (observability: what warns you when something is drawing more than planned), it has a building manager (GitOps control plane: the only legitimate authority to move anything), and it has a tenant register (dependency tracking: who is connected to which shared service).

When a new client moves into the third floor and asks to install a server that needs more amperage than planned, the problem is not electrical: it is administrative. If the client can go straight to the board and plug in whatever they like, the building survives for a while and then a breaker trips at three in the morning. If the client has to go through the manager, the manager consults the tenant register (is anyone else hanging off that same circuit?), reviews the electrical planning (are we at the limit?) and either authorises or redirects. The building stays standing not because of its electrical installation but because of the discipline of going through the manager.

The LLM inference stack works identically. The physical layers (1–4) are the visible ones, the ones marketing people put on the slide. The governance layers (5–7) are the ones that distinguish a platform from a pile of pods with good luck.

Now to the incident that motivates the whole post.

The hook: the log that lied for six days

The application is called jobhunter internally. It is a cron pipeline that every six hours sweeps public job-ad sources, filters by EU geography, embeds the new ads, indexes them in a vector store and matches them against search profiles. The last step fires notifications.

For six days the pipeline reported the same thing on every run:

status: completed
total_found: 756
new: 23
matched_jobs: 0 ← zero, run after run

And in the logs:

[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/delete → 200 OK
[INFO] matcher: Purged 12 expired jobs from ChromaDB
[ERROR] pipeline: ChromaDB indexing error: All connection attempts failed
[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/get → 200 OK
[ERROR] pipeline: Matching error: All connection attempts failed

This is a log that invites you to blame ChromaDB. And in fact the first post-mortem written internally pointed at a version incompatibility between the Python client and the v2 server. A reasonable hypothesis, technically plausible, completely false.

The real cause: weeks earlier the general LLM on the platform had been migrated from Ollama to vLLM. The migration was clean for the two applications that depended directly on the large model, their manifests were pointed at the new endpoint. What nobody did was look at who else was calling the Service ollama.ollama.svc:11434. jobhunter called it to generate the embeddings for the ads. When the Ollama deployment was scaled to zero, the Service was left empty, and any outbound connection received a generic ConnectError("All connection attempts failed") from httpcore. The try/except wrapping the whole matching stage caught the exception and labelled it “ChromaDB indexing error”, because that was the lexical wrapper of the block, not because ChromaDB had anything to do with it. ChromaDB was answering 200 to delete and get in those same logs.

Three factors kept the incident alive for six days:

The except named the wrapper rather than the stage that failed. The log said “ChromaDB indexing error” when the error was an embeddings call against a nonexistent Service.
The pipeline returned status: completed even when there were errors. No alerting based on status fired. The metric that would have fired (matched_jobs stuck at zero) was not instrumented.
The pipeline image ran as :latest with no versioning, no SBOM, no reproducibility. When it started failing, it was impossible to know which Ollama client it had been built with.

Each of the three symptoms is the cry of a layer that was missing. The first asks for LLM-aware observability that distinguishes stages (layer 5). The second asks for the pipeline logic and the Prometheus metrics to behave like contracts (layer 5, SLI/SLO dimension). The third asks for GitOps with pinned images and an auditable SBOM (layer 6). And the whole incident, decommissioning a Service without knowing who consumes it, shouts dependency tracking (layer 7).

The rest of the post walks the seven layers through that lens: what each one solves, which OSS piece implements it in 2026, and which typical design decision breaks it.

Layer 1 — Gateway: the client SDK must not couple to the engine

What it has to solve. That the client sends POST /v1/chat/completions with the standard OpenAI SDK and never learns which engine (vLLM, SGLang, TensorRT-LLM), which specific model, which LoRA adapter or even which GPU pool is serving the request. Authentication, rate limit, routing by body.model and by tenant, header injection for tracing.

Canonical OSS piece in 2026. Envoy AI Gateway (Envoy with GenAI extensions: routing by model, token-based rate limit, fallback chains) or Cilium Gateway API with your own filters. For cases where the client wants multi-provider without distinguishing on-prem from SaaS, LiteLLM Proxy is the lightweight equivalent.

What breaks the layer. Exposing the inference engine endpoint directly. If clients call http://vllm-prod.svc:8000, any change of engine, model or pool forces you to touch the code of every app. The rule: the engine changes, the contract does not. The OpenAI SDK is standard; the routing behind the gateway is where design freedom lives.

Where jobhunter failed. There was no gateway. The app called ollama.ollama.svc:11434 directly. When Ollama died, there was no intermediate layer that could answer with a fallback, a retry against another pool, or at least a 503 error with a descriptive body.

Layer 2 — LLM inference: vLLM as the default choice, SGLang when prefix caching rules

What it has to solve. Serving tokens with throughput and latency under control: continuous batching, PagedAttention so the KV cache does not fragment the VRAM, FP8 so a 32B model fits with room to spare, multi-LoRA for per-tenant customisation without replicating the base, structured output for function calling with schema guarantees.

Canonical OSS piece in 2026. vLLM is the default choice: it covers the state of the art (PagedAttention, continuous batching, FlashAttention, FP8 quantisation, multi-LoRA, structured output, speculative decoding). SGLang comes in when the workload has high prefix caching (long chat with large system prompts, agents with repeated instructions), because its RadixAttention composes better than standard prefix caching. For very specialised inference with proprietary NVIDIA kernels, TensorRT-LLM, accepting the hardware lock-in.

What breaks the layer. Assuming a single model “does everything”. A 32B chat LLM does not serve /v1/embeddings; if you try, vLLM answers BadRequestError: "The model does not support Embeddings API". Assuming it did was one of the concrete wounds of the incident: the app expected an endpoint the model did not implement.

The design decision that costs more later. Serving the LLM with aggressive quantisation (INT4) without a quality eval calibrated for your corpus. INT4 with AWQ or GPTQ saves VRAM, but it degrades answers in technical or legal Spanish in a measurable way. The rule: any quantisation change goes through the same golden eval as a model change.

Layer 3 — Embeddings: separate from the LLM, fixed dimension, a life of its own

What it has to solve. Generating dense vectors for the RAG corpus and for retrieval queries. The embeddings model is a different thing from the chat LLM: different architecture (encoder, not decoder), different size (hundreds of millions of parameters, not tens of thousands of millions), different API (an /embeddings endpoint that takes text and returns a fixed-dimension vector).

Canonical OSS piece in 2026. Infinity or Hugging Face Text Embeddings Inference (TEI) to serve models of the bge-*, multilingual-e5-*, nomic-embed-* families with high throughput and multi-model support. OpenVINO Model Server when Intel hardware is available. sentence-transformers as a fallback embedded in the application itself when the corpus is small and the deployment is constrained.

What breaks the layer. Treating it as “the LLM does that too”. It does not if it is chat-only; and even when it does (some models have a dual endpoint), mixing chat and embeddings serving in the same process punishes the throughput of both. The physical separation is the design.

The technical fact people forget. The vector store and the embeddings model form an indivisible unit. Changing the model from multilingual-e5-large (1024 dim) to multilingual-e5-small (384 dim) is not a substitution: it is creating a new collection (my_corpus_v2) and re-embedding the whole corpus. If you upsert into the old collection, you hit a dim mismatch at runtime that takes the pod down. This seems obvious and is violated constantly, because the embeddings model is chosen once and then forgotten.

Layer 4 — Vector store + relational data + storage: what actually holds the state

What it has to solve. Persisting the vectors with efficient filters (tenant_id, created_at, source), persisting the relational metadata (users, configs, versioned prompts, traces), and persisting the model weights, the LoRA adapters, the datasets and the original corpora.

Canonical OSS pieces in 2026.

Qdrant for large collections (>200 k vectors), payload-aware filters, multi-tenancy via collection or via field.
pgvector for small collections with mandatory relational joins (wanting to do WHERE doc.author = ... AND vector <=> $1 in the same SQL).
PostgreSQL operated by CloudNativePG (CNPG) for the relational side: Barman Cloud backups, replication, connection via pooler.
MinIO for S3-compatible objects: bucket per tenant, cross-site replication, weights and adapters versioned by sha256.
Redis for queues, rate-limit counters and a cache of frequent retrievals.

What breaks the layer. Assuming the vector store is stateless ephemeral. It is exactly the opposite: it is the component where losing state costs the most. Without verified backups of the vector store, an index corruption forces you to re-embed the whole corpus, and on a corpus of millions of documents that is hours or days of GPU.

The design decision that pays later. Forgetting to version the collection by dimension and embeddings model. Suggested convention: my_corpus__embed-multie5l__1024d__v3. The name carries metadata; any change in any of the three attributes forces a new collection. It is ugly but it protects against an accidental upsert with the wrong dim.

Layer 5 — Observability: LLM-aware traces + infrastructure metrics + flow logs

What it has to solve. That any inference can be recovered from its trace_id, with all the gen_ai.* attributes (semantic conventions) plus your own attributes (tenant_id, adapter_id, priority_tier), a breakdown of latency (queue → prefill → decode → network), tokens consumed, tools invoked, exact model and adapter. And in parallel: Prometheus metrics from vLLM (vllm:num_requests_running, vllm:gpu_cache_usage_perc, vllm:prefix_cache_hit_rate), from the GPU (DCGM Exporter), from the network (Hubble flow logs with drops and NetworkPolicy enforcement).

Canonical OSS pieces in 2026.

OpenTelemetry Collector as the single transport for traces, metrics and logs, with OTLP receivers and separate exporters per destination.
Langfuse self-hosted for the LLM-aware side: tracing, prompt versioning, evals with LLM-as-judge (post and post).
VictoriaMetrics + Grafana for high-throughput TSDB metrics with long retention.
Hubble (Cilium) for L3/L4/L7 flow logs and NetworkPolicy visualisation.
DCGM Exporter for GPU metrics.

What breaks the layer. Wrapping the whole pipeline stage in a single try/except that labels the exception with the wrapper’s name. The operational rule: one try/except per stage, with the stage label in the message and a Prometheus metric with labels={"stage": "<name>"}. That way “ChromaDB indexing error” would never have been the log for an embeddings failure; it would have been “Embeddings call failed: ConnectError(ollama:11434)”.

Complementary rule. The pipeline returns status: completed if and only if there were no errors. With errors it returns completed_with_errors or failed, and the metric pipeline_errors_total{stage} is incremented. An alert based on increase(pipeline_errors_total[1h]) > 0 fires before the second failed run. Without this discipline, observability exists but does not warn.

Layer 6 — GitOps control plane: the only legitimate authority

What it has to solve. That the cluster state is the state declared in git. That any divergence between git and the cluster is visible and, in critical components, auto-reconciled or auto-alerted. That every deployed image has an immutable tag (sha digest or semver pin), an SBOM (Trivy) and traceability back to the commit that introduced it.

Canonical OSS pieces in 2026.

Flux (or ArgoCD) as the reconciler.
Forgejo (or Gitea, GitLab CE) as the self-hosted OSS forge.
cert-manager + Trust Manager for internal PKI.
External Secrets Operator + SOPS for encrypted versioned secrets.
Kyverno (or OPA Gatekeeper) for binding policies: deny :latest images, deny pods without a NetworkPolicy, deny Services without an owner label.
Trivy for SBOM and vulnerability scanning of images in the CI pipeline.

What breaks the layer. Images with a mutable tag (:latest, :main). Any kubectl edit in production that is not reflected in git. main branches with write permissions for humans without review. The rule: if a human can mutate the cluster without going through a signed commit, you do not have GitOps, you have somebody’s whiteboard.

How it applies to the incident. If the pipeline image had been pinned to a sha digest (registry.interno.local/jobhunter@sha256:9af2...), the team could have audited immediately which Ollama client it carried. With :latest, not even that.

Layer 7 — Dependency tracking: the layer the incident exposed

What it has to solve. Knowing who calls which Service, both declaratively (what the gitops repo says) and observed (what has been seen crossing the network over the last N days). And, on mature platforms, propagating that information as policy: if nobody declares and nobody observes traffic to the Service ollama.ollama.svc, decommissioning it is safe; if somebody declares it or uses it, decommissioning it opens a ticket.

Canonical OSS pieces in 2026.

Hubble (Cilium) for observed flow logs: hubble observe --to-namespace ollama --since 14d gives the list of source namespaces that have talked to Ollama in the last two weeks.
Otterize for intent-based policy: each Deployment declares “I need to talk to ollama-svc”, and the operator generates the corresponding NetworkPolicy and maintains a browsable catalogue of who is trying to talk to what.
manual kubectl-grep as a fallback: kubectl get deployments,cronjobs,statefulsets -A -o yaml | grep -E 'ollama[.-]' produces the declarative list.
NetworkPolicy as code reviewed in CI: every PR that touches a Service requires the associated policy to be kept or explicitly updated.

A pre-decom checklist that the incident suggests codifying as a CI hook:

SVC="ollama.ollama.svc.cluster.local"

# (a) declarative grep in the gitops repo
git -C $GITOPS_REPO grep -l "$SVC" || echo "OK declarative"

# (b) observed grep in Hubble (last 14 days)
hubble observe --to-fqdn "$SVC" --since 336h --output json \
 | jq -r '.source.namespace' | sort -u || echo "OK observed"

# (c) live grep in the cluster
kubectl get all -A -o yaml | grep -E "$SVC" || echo "OK live"

If all three return empty, the decom is safe. If any of them has content, there is downstream debt still open. The jobhunter incident is exactly what happens when this check does not exist: the team that decommissioned Ollama looked at the list of applications it knew depended on it directly; nobody looked at the list of those that depended on it silently.

The maths that matter: sizing the stack on 4×H100 SXM (320 GB)

The generic reference cluster for everything that follows: 4×H100 SXM 80 GB, NVLink between the four, 640 GB of system RAM, 2×NVMe NVMe-oF for local storage of weights and caches, redundant 25/100 GbE to the switch. Aggregate VRAM: 320 GB.

The VRAM budget is not free. A first reasonable split for a stack that serves one large general LLM, a medium code-specialised model, embeddings and a reranker, with room for multi-LoRA and for the KV cache:

Component	Reference model	Quant	Model weights	Reserved KV cache	Total VRAM
General LLM (TP=4)	70B-instruct	FP8 W8A8	70 GB	60 GB	130 GB
Code LLM (TP=2)	32B-coder	FP8 W8A8	32 GB	28 GB	60 GB
Embeddings	multilingual-e5-large	FP16	1.3 GB	n/a	8 GB (×2 replicas)
Reranker	bge-reranker-v2-m3	FP16	0.6 GB	n/a	4 GB
Multi-LoRA pool (over the general LLM)	up to 16 adapters	bf16	16 × 0.4 GB ≈ 6 GB	reuses the LLM’s KV	6 GB
Reserved for fragmentation + overhead					~30 GB
Total committed					~238 GB / 320 GB
Free headroom					~82 GB

The 26 % of free headroom is not waste: it is what lets the vLLM scheduler avoid preempting requests under moderate pressure, lets continuous batching group large batches without aborting, and lets a failover from the other site promote a standby without OOM.

Expected throughput, with the 70B general LLM in FP8 and tensor parallel 4, on an H100 SXM with continuous batching active and prefix caching at 35–55 % (typical in multi-turn chat with shared system prompts):

$$ \text{aggregate tokens/second} \approx 1500 \text{ to } 2500 $$

for concurrency between 32 and 64 requests, with sub-second TTFT P95 on short prompts (<2k tokens) and TPOT P95 around 40–60 ms/token as perceived by the client. These numbers are reasonable orders of magnitude, not guarantees: real throughput depends on the prompt mix, on whether speculative decoding (EAGLE-3) is active and burnt-in, and on the network cost between gateway and inference pods.

Embeddings throughput on two replicas of multilingual-e5-large with dynamic batching:

$$ \text{embeddings/second} \approx 3000 \text{ to } 6000 \quad (\text{optimal batch} \sim 64) $$

Enough to reindex a corpus of 1 million documents in roughly an hour, assuming chunks of 512 tokens and two chunks per document on average. For corpora of tens of millions of documents, re-embedding is done by delta via CDC over the source (covered in RAG corpus curation), not by full sweep.

Retrieval latency on Qdrant with HNSW (M=16, ef_construct=200) over a collection of 5 million 1024-dim vectors filtered by tenant_id:

$$ \text{P95 latency retrieve top-50} \approx 8 \text{ to } 25 \text{ ms} $$

Below the cost of cross-encoder reranking (bge-reranker-v2-m3 over top-50 = roughly 30–60 ms more), and below any call to the LLM. The bottleneck in a well sized RAG pipeline is never the vector store: it is the LLM’s decoding.

Final diagram: the complete connected stack

The solid lines are the request path: client → gateway → inference engine → vector store/embeddings → response. The dashed blue lines are telemetry: each component emits OTel to the collector, which routes traces to Langfuse, metrics to VictoriaMetrics and logs to Loki. The dashed red lines are reconciliation: the GitOps control plane keeps any layer in its declared state and warns of divergence.

The diagram is not decorative: every arrow is a stable contract between two layers. If one layer changes (vLLM → SGLang, multilingual-e5 → bge-m3, Qdrant → pgvector), the arrows hold. That stability of contracts is the architectural property that lets a team migrate components without breaking downstream apps.

Typical design decisions that break the stack

A short list of mistakes seen repeatedly in stacks that looked well designed on paper:

1. Coupling the client SDK to the inference engine. Dropping the gateway because “vLLM already speaks OpenAI-compatible” works on day one and hurts the day you have to add a fallback, a canary or a second model.

2. Sharing the LLM and embeddings endpoint. A qwen2.5-32b-Instruct is chat-only; BadRequestError: "The model does not support Embeddings API" is the cry of a design that confused the two layers.

3. Reusing the vector store collection when changing the embeddings model. Different dimensions do not allow upsert. Versioning the collection by (model, dim, version) is ugly but it saves the day of the change.

4. A try/except wrapping an entire pipeline with the wrapper’s label. The log lies because the label is lexical, not causal. Every stage in its own try/except with its own label and its own metric.

5. status: completed with errors. The pipeline has to distinguish completed, completed_with_errors and failed, and the alerting has to fire on the last two. Without this, observability exists in theory and does not warn in practice.

6. Images with a mutable tag. :latest and :main are not tags, they are aliases. Without a sha digest there is no reproducibility and no auditable SBOM.

7. Decommissioning a Service without a pre-decom check. The three-grep check (declarative + observed + live) takes two minutes and costs six days of incident when it is skipped.

8. Default limits.memory on pods that load models. A sidecar loading sentence-transformers + torch + tokenizer needs 2–4 GB; with limits.memory: 1Gi you hit OOM on the first pod restart, and sometimes with no alert because the liveness probe answers through another route.

All of them are variants of the same principle: the stack does not fail in its most expensive layer (inference, where nobody underestimates the cost), it fails in the cheap, boring layers (gateway, observability, GitOps, dependency tracking) where it is tempting to save.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), the suggested split into pods:

gpu-node-01 (4×H100 SXM, intra-node NVLink)
├── vllm-llm-general (TP=4) ~130 GB VRAM (4 GPUs)
└── (shares GPUs with the multi-LoRA pool on the same deployment)

gpu-node-02 (4×H100 SXM, second node)
├── vllm-llm-code (TP=2) ~60 GB VRAM (2 GPUs)
├── infinity-embeddings (×2) ~16 GB VRAM (shared on 1 GPU with optional MIG)
├── tei-reranker ~4 GB VRAM (cohabitant)
└── failover reserve ~120 GB VRAM free for canary / standby

Not everything fits comfortably on a single node; two nodes with two H100 SXM each would be enough for the conservative setup, and the rest of the cluster (CPU-bound: gateway, vector store, observability, control plane) runs on GPU-free nodes.

The operational rule: inference concentrates, the rest of the stack distributes. Concentrating inference maximises the use of NVLink (cross-GPU tensor parallel without going through PCIe); distributing the rest prevents an event on the GPU node from taking the control plane down with it.

An even more conservative configuration, for SMEs with a single 4×H100 SXM node as a starting point, serves the general LLM (TP=4) and embeddings/reranker cohabiting with MIG (Multi-Instance GPU to partition an H100 into hardware-isolated slices). The code LLM is deferred to a second phase. It is viable and cost-aware; what is not viable is doing without layers 5, 6 and 7.

What we have not covered (upcoming posts)

This post focuses on the static design of the stack. Some pieces still deserve an article of their own:

The multi-site active/standby plane: Cilium Cluster Mesh, cross-cluster Qdrant replication, realistic RTO/RPO, when active-active pays off and when it does not.
The continuous fine-tuning plane: how the LoRA pipeline closes the loop from production feedback to a promoted adapter, in the spirit of the retrain post.
The safety/guardrails plane: where Llama Guard fits, Presidio for PII and XGrammar for guaranteed structured output, connected to layer 1, the gateway.
The cost plane: gen_ai.usage.* instrumentation at tenant and model level, tokens/euro dashboards, GPU elasticity decisions via KEDA.
The compliance plane: how the stack maps to ENS High, NIS2 and ISO/IEC 42001 without turning the deployment into a compliance exercise that paralyses delivery.
Below the engine: the mini-series that opens the stack’s basement — the interconnect (NVLink/NCCL), the host (NUMA, hugepages, CPU isolation) and the RKE2 resource managers that pin each pod to the right NUMA node.

Each one falls into a different series on the blog and will be covered with the same discipline: a concrete piece, a justified decision, the typical mistake seen in practice.

References

The six-stage LLMOps pipeline — the general framework this stack operates in.
The OSS catalogue for LLMOps in six stages — a card per tool in the stack.
Anatomy of an LLM request in production — the same architecture seen from the perspective of an individual request.
OSS vs hyperscalers in LLMOps — a comparison with the AWS/Azure/GCP stacks.
LLM tracing with OpenTelemetry GenAI — the backbone of layer 5.
Multi-LoRA serving — the piece that gives per-tenant customisation without replicating the base.
RAG corpus curation and Reranker and hybrid retrieval — everything that goes into layers 3-4 so that RAG does not degrade.
Quantisation for LLM inference — the reason for the FP8 W8A8 in the sizing.

Relevant official documentation

vLLM Production Stack — docs.vllm.ai
SGLang RadixAttention — github.com/sgl-project/sglang
Envoy AI Gateway — aigateway.envoyproxy.io
Langfuse self-hosted — langfuse.com/docs/self-hosting
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
Hubble flow observability — docs.cilium.io/en/stable/observability/hubble
Otterize intent-based access — docs.otterize.com
Flux GitOps toolkit — fluxcd.io

On-Premise on lo0 — Tech Blog

Real ClickHouse capacity in Langfuse v4: 2,782 bytes per observation, and why almost half of that is not data

TL;DR

You are here: the sizing you cannot do by reading

The analogy: the archive and the index card

The benchmark

What one observation takes up

Where the bytes go

The split by observation type

The indexes, which are almost half

The materialized view as an amplifier

What happens on deletion

Projection

What this benchmark does not measure

Checklist

Traps and things that are not what they look like

Closing

The series: the ten articles

See also

Sources

deepagents on a cluster of your own: the SDK is free, the server is not, and what you have to build in between

TL;DR

You are here: the harness arrives after the model

The analogy: the scaffolding and the crane

What deepagents actually is

What is no longer there

Where state lives

The backends

What grows in the database

The store, which does have cleanup

The isolation that does not exist

The model, and the trap you need to know about

The tools, and the MCP that is not where it seems

The subagents

Code execution

Observability, and the package it goes through

The server: here is where the free part ends

What you have to build

How it fits with activity logging

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

Making the model prefer our tools: the priority that exists in no layer, the two primitives that do, and the `_meta` that crosses the gateway

TL;DR

You are here: the question that arrives after the third server

The analogy: the counter with two trays

What the specification says: nothing

The priority that exists and is no use

The only blessed filtering is by credential

Discovery does not discover tools

What there is in proposals

The layer that decides: the client

Claude Code

The other clients

The API layer

The test: the _meta crosses the gateway

The setup

The result

The three routes where it is lost

The library trap

What we have not tested

What LiteLLM can and cannot do

The cheap lever: the description, not the name

What does not work

The ladder, updated

The risk that has to be declared

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

TL;DR

You are here: the resource side

The analogy: the visa and the border checkpoint

Part 1. The split of obligations

What is new in the revision and affects the setup

Part 2. What the SDKs give, and what they do not

The `priority` that exists and is no use

The test: the `_meta` crosses the gateway