Operations on lo0 — Tech Blog

Langfuse v4, day 2 (5 of 10): the worker's thirty-nine queues, and the three that decide whether you lose data

Mon, 14 Sep 2026 08:30:00 +0200

Fifth article in a series of ten about operating Langfuse v4 in production. The previous one covered migrating from version 3. This one goes inside the worker that is already running. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

Thirty-nine queues and thirty-two switches. The catalogue is in packages/shared/src/server/queues.ts:406. The QUEUE_CONSUMER_*_IS_ENABLED variables are read once at boot, so turning a queue off is a restart, not a hot change.

Sharding spreads nothing without Redis Cluster. The producer only applies the hash if REDIS_CLUSTER_ENABLED === "true", and that variable ships "false". Raising LANGFUSE_INGESTION_QUEUE_SHARD_COUNT on an installation with single-node Redis starts N workers and leaves N−1 with no work.

The dead-letter retrier ships disabled. QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED is the only one of the thirty-two whose default is "false".

And even switched on, it does not cover ingestion. Its list holds five queues and none is ingestion-queue. An event that exhausts its six attempts sits in the failed set.

The ClickHouse writer drops rows. Three attempts with a flat hundred-millisecond delay and then out. The code comment is explicit: // TODO - Add to a dead letter queue in Redis rather than dropping. The metric to watch is called langfuse.queue.clickhouse_writer.rows_dropped.

Turn a queue’s consumer off and you stop seeing its depth. The metrics emitter only polls queues with a worker registered in that container. The queue keeps growing and the dashboard goes flat.

Project deletion runs one job every ten minutes. By limiter, deliberately, because each one scours ClickHouse.

There are two optional health probes and one of them can cause the very restart loop it is meant to fix if configured with less than sixty seconds of initial delay.

You are here: OBSERVE, day 2

The previous article left the installation migrated. This is the map of the process that does the work, and it comes before the capacity article because the queues are where saturation shows up before ClickHouse notices.

The analogy: the sorting hub

A postal sorting hub has separate belts for parcels, registered mail, returns and internal pouches. Each belt has its speed and its staff, and some are throttled on purpose because the destination cannot take more.

The job of whoever runs it is not to watch the fastest belt, it is to know which one jams first, which can stop for an afternoon without anyone noticing and which cannot stop for ten minutes. And above all, where the bin for items nobody could sort is, because every hub has one and in many of them nobody looks at it.

The Langfuse worker has thirty-nine belts. This article is the plan, with the speeds and the location of the bin.

The map of the thirty-nine

Grouped by what they do, with what stops happening if the group stops:

Ingestion (4). ingestion-queue, secondary-ingestion-queue, otel-ingestion-queue, secondary-otel-ingestion-queue. If they stop, the web keeps accepting events and uploading them to object storage, and keeps enqueueing. Nothing reaches ClickHouse. The SDK sees no error at all. This is the group that cannot stop.

Propagation (1). event-propagation-queue, the dual-write one from the previous article, with a global concurrency of one across the cluster.

Evaluation (8). trace-upsert, dataset-run-item-upsert-queue, create-eval-queue, evaluation-execution-queue and its secondary, llm-as-a-judge-execution-queue, code-eval-execution-queue, experiment-create-queue. If they stop, evaluations are neither created nor executed. Traces keep arriving. This is the group that can wait.

Deletion and retention (7). trace-delete, score-delete, dataset-delete-queue, project-delete, data-retention-queue and its processing queue, batch-action-queue. If they stop, neither UI-requested deletions nor per-project retention run. That is not an operational annoyance, it is a breach if you have a retention commitment in writing.

Exports (3). batch-export-queue, core-data-s3-export-queue, metering-data-postgres-export-queue. Downloads a user asked for stay pending.

Outbound integrations (8). The PostHog, Mixpanel and blob storage pairs, plus webhook-queue, entity-change-queue and notification-queue. In each pair, the bare queue is the scheduler that fans out per project and the processing one does the work.

Cloud billing (4). They only register with a Stripe key. On a self-hosted installation they never start.

Maintenance (4). dead-letter-retry-queue, monitor-queue, in-app-agent-run-queue and v4-legacy-api-usage-queue, which ingests nothing: it is a cron that scans ClickHouse’s system.query_log to find which projects still use the old API.

The switches number thirty-two, all QUEUE_CONSUMER_*_IS_ENABLED and all "true" except the dead-letter retrier. Four queues have no switch of their own: the model-as-judge one rides inside the evaluation switch, the two Cloud data export queues have their own variable, and the processing queues share a switch with their scheduler.

The mechanism is a top-level if around each worker registration. A false switch means the worker is never instantiated, not that the queue stops receiving work: producers keep enqueueing regardless. Turning a queue off is not consuming it, not not filling it.

The sharding that does not shard

Nine queues support sharding: the four ingestion ones, three evaluation ones, model-as-judge and trace-upsert. Each has its own shard-count variable, and they all default to 1.

Raising them looks like the obvious first move to scale ingestion. It is not, because of this (packages/shared/src/server/redis/ingestionQueue.ts:48):

 const shardIndex =
 IngestionQueue.getShardIndexFromShardName(shardName) ??
 (env.REDIS_CLUSTER_ENABLED === "true" && shardingKey
 ? getShardIndex(shardingKey, env.LANGFUSE_INGESTION_QUEUE_SHARD_COUNT)
 : 0);

The : 0 at the end is the whole story. The producer only computes a shard if Redis is in cluster mode, and REDIS_CLUSTER_ENABLED ships "false". On an installation with single-node Redis, which is most self-hosted installations, raising the shard count to four starts four workers, three of which stare at empty queues, while all traffic keeps coming through ingestion-queue.

The sharding key, when it does apply, is projectId-eventBodyId run through SHA-256, taking the first eight hex characters and a modulo. Shard names are ingestion-queue for zero and ingestion-queue-N for the rest.

Without Redis Cluster, the lever that does work is concurrency, which is a different thing.

Concurrencies, and the ones set to one on purpose

The defaults, worth knowing because several sit at one for good reasons:

Queue	Variable	Default
`ingestion-queue` (per shard)	`LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	20
`secondary-ingestion-queue`	`LANGFUSE_INGESTION_SECONDARY_QUEUE_PROCESSING_CONCURRENCY`	5
`otel-ingestion-queue`	`LANGFUSE_OTEL_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	5
`trace-upsert`	`LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY`	25
`monitor-queue`	`LANGFUSE_MONITOR_QUEUE_PROCESSING_CONCURRENCY`	10
evaluation and model-as-judge	several	5
`create-eval-queue`	`LANGFUSE_EVAL_CREATOR_WORKER_CONCURRENCY`	2
`trace-delete`, `score-delete`, `dataset-delete-queue`, `project-delete`	several	1
`event-propagation-queue`	fixed in code	1 cluster-wide

Deletions sit at one and carry a rate limiter on top. The project one is the most aggressive (worker/src/app.ts:234):

 limiter: {
 // Process at most `max` delete jobs per LANGFUSE_CLICKHOUSE_PROJECT_DELETION_CONCURRENCY_DURATION_MS (default 10 min)
 max: env.LANGFUSE_PROJECT_DELETE_CONCURRENCY,

One project deletion every ten minutes. If a customer asks you to delete forty projects, that is six hours and forty minutes on the clock, and rightly so, because each of those deletions is a broom sweeping through ClickHouse.

The ingestion queues, by contrast, have no limiter at all. There the brake is concurrency and the ClickHouse writer.

One BullMQ detail that recurs across queues and explains a good number of other people’s stalls (worker/src/app.ts:261):

 // The default lockDuration is 30s and the lockRenewTime 1/2 of that.
 // We set it to 60s to reduce the number of lock renewals and also be less sensitive to high CPU wait times.
 lockDuration: 60000, // 60 seconds
 stalledInterval: 120000, // 120 seconds
 maxStalledCount: 3,

An event’s path, from API to ClickHouse

It is worth following it end to end once, because there are three places it can be lost.

The web groups the batch’s events by body id, uploads one JSON file per group to object storage and aborts if that upload fails: no blob, no enqueue. Then it enqueues one job per body id on ingestion-queue.

The worker picks up the job, optionally checks a recently-processed cache in Redis (off by default, five-minute lifetime), decides whether to redirect the project to the secondary queue, and downloads the file from storage. There is a comment here worth a capacity chart on its own:

Process files in batches. If a user has 5k events, this will likely take 100 seconds.

Then it merges and writes. And writing is not writing: the service does not talk to ClickHouse, it puts the row in a buffer whose defaults are 1,000 rows, 1,000 milliseconds and 3 attempts.

That buffer is the third place things get lost, and the most serious one in the system (worker/src/services/ClickhouseWriter/index.ts:535):

 // Re-add the records to the queue with incremented attempts
 let droppedCount = 0;
 queueItems.forEach((item) => {
 if (item.attempts < this.maxAttempts) {
 entityQueue.push({ ...item, attempts: item.attempts + 1 });
 } else {
 // TODO - Add to a dead letter queue in Redis rather than dropping
 recordIncrement("langfuse.queue.clickhouse_writer.error");
 droppedCount++;
 }
 });

Once the three attempts are spent, the row is dropped. And no queue retry saves it, because the BullMQ job already completed successfully: as far as the queue is concerned, that event was processed. The internal retries are not properly exponential either: the delay is flat at a hundred milliseconds, so all three attempts happen within three hundred milliseconds and a ClickHouse that is unwell for half a second takes the batch with it.

The runbook consequence is direct. langfuse.queue.clickhouse_writer.rows_dropped has to be on a dashboard and it has to have an alert at zero. Any non-zero value is silent data loss.

Retries, and the bin nobody looks at

Each queue defines its own policy. The ones that matter:

Queue	Attempts	Backoff
`ingestion-queue`	6	exponential, 5 s
`secondary-ingestion-queue`	5	exponential, 5 s
`evaluation-execution-queue`	10	exponential, 1 s
`code-eval-execution-queue`	3	exponential, 30 s
`trace-upsert`	2	exponential, 5 s, with a 30 s initial delay
`trace-delete`, `score-delete`	2	exponential, 30 s
`monitor-queue`, `in-app-agent-run-queue`	1	no backoff, deliberately

The two single-attempt ones are a design decision, and the comment explains it well: “Postgres owns correctness (claim CAS + reconcile-on-read); BullMQ is delivery-only, so never redeliver on its own.”

And now the bin. There is no dedicated dead letter queue. What exists is BullMQ’s native failed set per queue, with its own retention, plus a queue that retries from there. That queue has two problems for whoever operates it:

First, it ships disabled. It is the only one of the thirty-two defaulting to "false".

Second, even switched on, its scope is hand-written and has five entries (worker/src/services/dlq/dlqRetryService.ts:9):

 private static retryQueues = [
 QueueName.ProjectDelete,
 QueueName.TraceDelete,
 QueueName.ScoreDelete,
 QueueName.BatchActionQueue,
 QueueName.DataRetentionProcessingQueue,
 ] as const;

ingestion-queue is not there. Ingestion events that exhaust their six attempts sit in the failed set, up to a hundred thousand of them, and nobody ever retries them. To get them out there is an admin route, POST /api/admin/bullmq with action retry, which the file itself describes as the one the managed service uses for exactly this.

The metrics, with their exact names

Each queue’s metric name is built at runtime from the queue name, turning dashes into underscores and stripping the _queue suffix. So ingestion-queue produces langfuse.queue.ingestion, and secondary-ingestion-queue produces langfuse.queue.secondary_ingestion. Sharded queues always add a shard tag.

What comes out of that:

langfuse.queue.<queue>.rate with type in request, completed, failed, error and stalled.
langfuse.queue.<queue>.time_distribution with type in wait and processing.
langfuse.queue.<queue>.depth with type in waiting, failed and active, plus an aggregate series with shard: "all".
langfuse.queue.<queue>.dlq_oldest_age, the age of the oldest failed job, which emits zero when the set empties so the dashboard resets.
From the writer: langfuse.queue.clickhouse_writer.rows_dropped, .error, .processing_time and ingestion_clickhouse_insert_queue_length.

Two things to know about those metrics. The waiting depth includes paused jobs, to match BullMQ’s counter. And the big trap (worker/src/features/queue-metrics-runner/index.ts:70):

 // Only poll queues that have registered workers. This avoids calling
 // getInstance() on queues this worker doesn't consume, which would
 // create unnecessary Redis connections and can trigger side effects

Only queues that container consumes emit metrics. If you split roles and turn a queue’s consumer off across every container, that queue stops appearing on dashboards while it keeps growing in Redis. It is the perfect way not to find out.

The metric that saturates first is still the same one: langfuse.queue.ingestion.depth with type: "waiting". The emission interval defaults to one second.

The two probes, and the one that can restart you in a loop

The worker exposes /api/health and /api/ready, identical but for one thing: the second returns 500 after SIGTERM, which is how draining works. Both always check Postgres with a SELECT 1 and Redis with a ping capped at two seconds, hardcoded.

Then there are two optional checks enabled by query parameter:

?failIfEventPropagationStuck=true looks at the propagator’s heartbeat and returns 503 if it has not been refreshed for more than 35 minutes. A missing heartbeat does not count as stuck, deliberately, so a freshly booted container is not restarted in a loop. And the condition that must be respected:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

?failIfQueueConsumptionStuck=true is an in-memory signal of that container, with no Redis involved: it is marked on the active and completed events of any worker. It returns 503 if that container has gone 60 minutes without picking up or finishing a single job. It exists for a very specific failure mode:

After Redis lock loss BullMQ workers can wedge permanently: the process stays alive and connectivity checks pass, but no queue picks up jobs ever again.

The one-hour threshold leans on the default crons keeping a healthy worker busy at least once an hour. In multi-replica deployments each scheduler tick lands on a single replica, so the comment itself warns about raising the threshold if a replica can legitimately sit idle.

Draining, by the way, has no cap. The order is close the HTTP server, stop the periodic runners, close the BullMQ workers and then flush the ClickHouse writer, which on shutdown empties the whole queue rather than one batch. The only thing bounding that drain is the Kubernetes grace period, so it had better be generous.

The crons

Queue	Pattern	What it does
`event-propagation-queue`	`* * * * *`	propagates one staging partition
`dead-letter-retry-queue`	`0 /10 * * *`	retries failed jobs from five queues
`v4-legacy-api-usage-queue`	`/15 * * *`	scans `system.query_log` for old-API usage
`blobstorage-integration-queue`	`/20 * * *`	fans exports out per project
`cloud-usage-metering-queue`	`5 * * * *`	Cloud billing
`posthog-integration-queue`, `mixpanel-integration-queue`	`30 * * * *`	hourly fan-out
`metering-data-postgres-export-queue`	`30 2 * * *`	daily export
`core-data-s3-export-queue`, `data-retention-queue`	`15 3 * * *`	daily export and retention

And a warning the scheduler file itself writes in capitals: cron registration is fire-and-forget from the queue constructors, and a failure is only logged. A transient Redis failure at boot can leave a cron unscheduled until the next restart, without the container ever being marked unhealthy.

Splitting roles, which is what all this is for

With thirty-two switches, what you can build is one worker per family. The split that makes sense on an installation with agentic load:

# Pool 1: ingestion. This is the one that scales with traffic.
QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_OTEL_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED=true # note: global concurrency 1
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=20
# everything else false

# Pool 2: evaluations. Scales with the number of evaluators, not with traffic.
QUEUE_CONSUMER_EVAL_EXECUTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_CREATE_EVAL_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_TRACE_UPSERT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EXPERIMENT_CREATE_QUEUE_IS_ENABLED=true

# Pool 3: maintenance. One replica is enough.
QUEUE_CONSUMER_TRACE_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_PROJECT_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DATA_RETENTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED=true # switch it on

Two warnings about this split. Each worker registration opens its own Redis connection, so the connection count grows with queues times shards times replicas, and that is checked against Redis maxclients before multiplying replicas. And the moment you split roles you lose the metrics of the queues that container does not consume, so some replica needs to consume every queue or you go blind on that one.

Checklist

There is an alert on langfuse.queue.clickhouse_writer.rows_dropped, threshold zero.
There is an alert on the depth of langfuse.queue.ingestion with type: "waiting".
There is an alert on dlq_oldest_age for the queues that matter, and somebody knows how to use POST /api/admin/bullmq.
The dead-letter retrier is on in some replica, knowing it does not cover ingestion.
Nobody raised the shard count without Redis Cluster.
The optional health probes are configured with initialDelaySeconds of at least 60 seconds.
The Kubernetes termination grace period allows the ClickHouse writer to flush.
Every queue is consumed by at least one replica, so its metrics are not lost.
Redis maxclients copes with queues times shards times replicas.

Traps

Sharding without Redis Cluster does not spread load. It starts the workers and leaves them idle.

The dead-letter retrier ships disabled and does not cover ingestion. Two independent facts, both uncomfortable.

The ClickHouse writer drops rows after three attempts. With a TODO in the code admitting it and three hundred milliseconds of total headroom.

Turning a consumer off also turns its metrics off. The queue keeps growing, the dashboard stays flat.

A cron can end up unscheduled while the container stays healthy. A transient Redis failure at boot, and it is gone until the next restart.

The PostHog and blob storage integrations disable themselves when an error is classified as a customer-config fault, and if the notification also fails they stay off and silent.

A misconfigured propagation probe restarts in a loop. Less than sixty seconds of initial delay and you get the cycle.

Draining has no cap of its own. The Kubernetes grace period bounds it and nothing else.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues (this article): the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/server/queues.ts, packages/shared/src/server/redis/, worker/src/env.ts, worker/src/app.ts, worker/src/queues/, worker/src/services/ClickhouseWriter/, worker/src/services/dlq/, worker/src/features/health/, worker/src/features/queue-metrics-runner/, worker/src/utils/shutdown.ts.
Langfuse self-hosting documentation, checked on 14 September 2026.
BullMQ: going to production, checked on 14 September 2026.

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

Mon, 14 Sep 2026 08:00:00 +0200

Fourth article in a series of ten about operating Langfuse v4 in production. The previous three looked at the client side: what goes into a trace, what an agent turn costs and two well-known agents instrumented. This is the first one on the server side. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

The default write mode is the destination, not the origin. LANGFUSE_MIGRATION_V4_WRITE_MODE ships as events_only (packages/shared/src/env.ts:337). That is right for a new installation and wrong for a migration: in that mode the traces and observations tables stop being written.

The worker refuses to start on three combinations. legacy with direct OTel writes, events_only without the preview opt-in, and events_only with OTel behaviour set to dual. The code comment names them: combinations that would silently lose data (worker/src/env.ts:722).

In events_only twenty-four REST routes return 404. Traces, observations, sessions, v1 and v2 scores, metrics, spans, generations and the dataset-run ones. Not a bug: a deliberate short circuit so stale data from tables nobody writes is never served.

History is backfilled by a chain of five background migrations, not by the ClickHouse migrations. Not one of the ten new ClickHouse migrations copies a single old row.

A failed background migration does not block the next one. The manager has no dependency model and skips them. The chain protects itself with predecessor guards that the first two steps do not declare, so a failure of step 1 does not stop step 2 from starting.

The old API’s ten-minute delay has an exact explanation. Three-minute partitions in the staging table, a lock four minutes after creation, and a per-minute cron that only touches partitions older than ten minutes, with a global concurrency of one.

That staging table has a forty-eight-hour TTL. If the propagator stops over a long weekend, those partitions disappear and the hole in events_full is permanent. No code detects it and none repairs it.

The point of no return is in Postgres, not in ClickHouse. A Prisma migration runs DROP TABLE on traces, observations and scores, and Prisma Migrate has no down migrations.

You are here: OBSERVE, day 2

The first three articles measured what an agent produces. This one starts on operating the backend that receives it, and it comes before the capacity article because you cannot size what has not been migrated yet.

The analogy: changing the track gauge without closing the line

Changing the gauge of a line in service is not done in one go. The third rail is laid, rolling stock of both gauges runs for a while, and only when every train is of the new gauge is the old rail lifted.

The Langfuse migration has exactly that shape. dual mode is the third rail: the old tables and the new ones are written at the same time. And as with the track, the danger is not the day of the change, it is at both ends. Laying the new rail and lifting the old one on the same day strands trains. And once the old rail is up, going back is not a matter of will, it is a matter of the material no longer being there.

The three-rung ladder

Everything turns on one variable with three values, defined three times over in the three packages that have to agree, with the same enum and the same default (packages/shared/src/env.ts:337, worker/src/env.ts:586, web/src/env.mjs:610):

LANGFUSE_MIGRATION_V4_WRITE_MODE: z
 .enum(["legacy", "dual", "events_only"])
 .default("events_only"),

The two helpers that read it say what each rung does better than any explanation (worker/src/env.ts:705):

export const v4WritesToEventsTable = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "legacy";

export const v4WritesToLegacyTables = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "events_only";

What gets written in each mode:

Table	`legacy`	`dual`	`events_only`
`traces`, `observations`	yes	yes	no
`scores`	yes	yes	yes
`dataset_run_items_rmt`	yes	yes	no
`observations_batch_staging`	no	yes	no
`events_full`	no	yes	yes
`events_core`	no	yes (through the materialised view)	yes

The middle rung is the only one where both coexist, and therefore the only place you can step back from without losing anything. The code itself offers it as the remedy when something breaks in events_only (web/src/pages/api/public/ingestion.ts:296):

As a temporary migration bridge, set LANGFUSE_MIGRATION_V4_WRITE_MODE=dual on both the web and worker services and redeploy.

The both in that sentence matters. All three packages read the variable and the code comment insists three separate times on keeping them in sync. Deploying the worker with one value and the web with another is the first way to break this.

The three combinations that refuse to start

The worker validates its configuration at boot and throws in three cases, under a comment that leaves no doubt (worker/src/env.ts:718):

 // Hard errors: combinations that would silently lose data.
 if (mode === "legacy" && otel === "direct") {
 throw new Error(
 "Invalid V4 config: LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct " +
 "requires LANGFUSE_MIGRATION_V4_WRITE_MODE in {dual, events_only}. " +
 "Direct OTel writes target events_full, which is not read in legacy mode.",
 );
 }

The second one bites in a real migration: events_only also requires LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true, because otherwise the web reads would target the legacy tables that mode no longer writes. The third forbids events_only with OTel behaviour set to dual.

A failed boot is the good news. A failed boot shows up in thirty seconds. The combinations it does not validate take days to surface.

What breaks in each mode

This is the table to have in front of you before moving the variable, because the two ends break different things.

In legacy the event tables are not written, so whatever depends on them disappears. Monitor routes return 404 through a dedicated middleware (web/src/server/api/trpc.ts:413). Metrics v2 is unavailable. Dashboard widgets stay at version 1 (DashboardService.ts:39). The v4 preview is forced off and cannot be switched on.

In events_only the legacy tables are not written, so twenty-four REST routes short-circuit with a 404 (web/src/features/public-api/server/createAuthedProjectAPIRoute.ts:319):

 // Short-circuit routes that read from legacy traces/observations tables
 // when the deployment is in events_only mode — those tables are no longer
 // populated, so the response would be stale or empty.

The list covers traces, traces/{id}, observations, observations/{id}, sessions, sessions/{id}, scores and scores/{id} in v1 and v2, metrics, metrics/daily, spans, generations, events, dataset-run-items and the two datasets/{name}/runs routes. If you have your own integrations against those routes, they break on cutover day, and returning 404 rather than stale data is the right call even though it stings.

One detail that is usually reported wrongly: /api/public/ingestion is never shut down. The deprecation notice says so itself (deprecations.ts:104):

This route is never removed: scores (and sdk-log) stay accepted.

In events_only that route stays alive and keeps accepting two event types (web/src/pages/api/public/ingestion.ts:274):

const EVENTS_ONLY_ALLOWED_TYPES = new Set<string>([
 eventTypes.SCORE_CREATE,
 eventTypes.SDK_LOG,
]);

The rest are rejected per event, not per batch: the batch comes back as a 207 with 400s inside. An old client sending traces and scores mixed together will see the scores go through and the rest fail, which is exactly the kind of partial failure that takes a long time to notice.

The three steps, in order

Given all of the above, the order of a no-window migration is this:

Finish the v3 background migrations before upgrading. This comes before everything else and there are no shortcuts. v4 deletes the rows of seven old migrations, and the comment explains why (prisma/migrations/20260723124010_drop_dataset_run_items_table/migration.sql:7): their worker scripts are deleted alongside, so an unfinished row would leave the manager with no script to resolve. If one was halfway, that data is lost and the row disappears quietly.
Upgrade to v4 with LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy, apply the ClickHouse and Prisma migrations. Here the installation behaves like v3 and you verify everything still stands.
Move to dual. From then on both table families are written and the propagator starts. Let it run for as long as it takes and launch the historic backfill.
Once history is complete and verified, move to events_only with the preview opt-in enabled, and only then lift the old rail.

All the work is between 3 and 4, and it can take weeks. There is no rush: dual is a stable state.

The ClickHouse migrations backfill nothing

There are 96 files in packages/shared/clickhouse/migrations/canonical/, 48 up and 48 down, which have to be materialised before running (pnpm ch:migrations:materialize) and cleaned afterwards, including after a failed migration. The runner is golang-migrate.

The ten v4 ones are 0039 through 0048. The first three create events_full, events_core and the materialised view that feeds the second from the first. The three drop migrations remove event_log, project_environments and dataset_run_items. The ones in between add columns and indexes.

What matters for planning is what they do not do. None copies an old row. The materialised view only sees rows inserted into events_full after it exists. The indexes in migrations 0043 and 0047 are added without materialising historical parts, and 0047 says so in writing:

Skip indexes are added without materializing historical parts to keep this migration metadata-only.

Translated into operations: during and after the migration, searches over older data do full scans. Not a bug, the price of a migration that does not block.

And one cluster trap: migrations 0042, 0043, 0047 and 0048 carry alter_sync = 2, meaning every ALTER waits for all replicas. One slow or down replica hangs the whole migration. On a single-replica installation you never notice; on three, you do.

The five-step chain that does backfill history

The backfill is done by five background migrations running in the worker, registered in Postgres with names that carry the order inside them:

20260701_v4_step_1_create_root_spans_from_traces: creates virtual root spans in events_full from existing traces, with the identifier concat('t-', t.id). It skips traces referenced by a dataset run item, which belong to step 4. Chunked by monthly partition.
20260701_v4_step_2_rewrite_observations_to_pid_tid_sorting: copies observations into a scratch table with a different key order. Chunked by monthly partition.
20260701_v4_step_3_backfill_events_full_from_observations: the bulk of it. Chunked by ClickHouse part, not by partition, and the comment explains why that is the only safe granularity:

One todo per part keeps each INSERT bounded to a single ClickHouse part (single-digit GBs in most cases), which is the only granularity that’s safe to assume self-hoster hardware can chew through without OOM or memory-limit failures.
20260701_v4_step_4_backfill_events_full_from_dataset_run_items: walks the dataset run items by cursor and writes the root and its descendants. Batches of 200, a cap of 50,000 descendants per item, a 90-day window.
20260701_v4_step_5_drop_pid_tid_sorting_tables: drops step 2’s scratch tables.

There are only two switches for the five. The first four share LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL, which ships true, and the fifth has its own, LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES, which ships false on purpose so the scratch tables stay around until the operator confirms the backfill went well.

One migration at a time across the whole cluster. The manager takes a lock in Postgres, refreshes it with a heartbeat every 15 seconds and treats a lock older than 60 seconds as expired, all inside a serialisable transaction. Adding worker replicas does not speed the backfill up by one minute. That is the first thing that surprises anyone used to scaling out to go faster.

They are resumable, and rather better than this usually is. Progress lives in a JSONB field on the row, and on resume the worker re-attaches to the queries the previous worker left running, by query id, instead of relaunching them:

In-flight queries keep running server-side; the next run re-attaches to them via the persisted queryIds.

On how long they take, the code says nothing. Not the files, not the folder’s README, not the migrations. The only time threshold anywhere is a design one: “A good threshold is something that takes more than 5 minutes to run”. Any figure in hours or days you read elsewhere, including the one I repeated myself before checking, is not in the code.

The failure that does not block

This is the finding that changes a runbook the most. When a background migration fails, the manager marks it with failedAt and moves on. And the predicate it uses to pick the next one excludes exactly the rows carrying failedAt, so a failed migration is skipped and does not block the ones after it.

The code acknowledges the gap and explains the patch (worker/src/backgroundMigrations/utils/backfillBase.ts:107):

The BackgroundMigrationManager has no dependency model — it runs every eligible row in name order and marks each finished/failed independently, so a failed upstream step does NOT by itself stop a downstream step from running on partial data. Each downstream step calls this from its validate() […] Because the next step in turn guards on its own predecessor, a single failure transitively halts the rest of the chain.

The guard exists, but only in steps 3 and 4. Steps 1 and 2 declare no predecessor. A failure of step 1 does not stop step 2 from starting; what saves the situation is that step 3 does check step 2.

Practical consequence: the only reliable signal that the backfill is going well is to look at the background_migrations table in Postgres and check that all five rows have finishedAt and none has failedAt. And to restart a failed one you clear failedAt by hand and relaunch with --retry-failed, as the error message itself says.

Two concrete failure modes that deserve a place in the runbook:

Step 3 breaks if ClickHouse merges a part midway. The message says it all: “Part … no longer active after processing — its rows are in a merged successor part that is not in this run’s todo list. Clear failedAt and re-run with state.chunksLoaded=false”. Recovery is manual, editing the JSONB.
Step 4 stops dead on one giant trace, if it exceeds 50,000 descendants. The three ways out the error offers are raising the cap, skipping the item (and losing that trace entirely) or fixing it by hand in SQL.

And an asymmetry worth knowing before comparing data: the historic backfill does not copy trace metadata down to child spans, deliberately, whereas the live path does concatenate them. History and new data are not identical.

Where the ten minutes come from

The v3 API warns that its data runs about ten minutes behind (deprecations.ts:16):

Data on this API is delayed by about 10 minutes. Real-time is only OpenTelemetry writes plus v2 observations and v2 metrics reads.

That delay is the sum of four things, all visible in the code:

The staging table partitions on three-minute intervals.
A partition is treated as closed four minutes after creation: “We ’lock’ partitions 4min after their creation, i.e. the 15:00:00 partition should stop receiving updates at 15:04:00.”
The propagation cron runs every minute and only looks at partitions older than LANGFUSE_EXPERIMENT_EVENT_PROPAGATION_PARTITION_DELAY_MINUTES, which defaults to 10.
That cron has a global concurrency of one across the cluster and processes a single partition per invocation, so a backlog makes the delay grow rather than recover.

The way out of the delay is not to lower that variable, it is to have clients write over OTLP with direct writes, which is the real-time route. The header that enables it is x-langfuse-ingestion-version: 4, sent automatically by the Python SDK from 4.0.0 and the JavaScript one from 5.0.0, and a value above 4 is rejected with a 400.

The forty-eight-hour TTL

This is the frightening one, and it appears in no migration guide.

The staging table carries TTL toDateTime(s3_first_seen_timestamp) + INTERVAL 48 HOUR with ttl_only_drop_parts = 1. If the propagator stops for more than forty-eight hours, a long bank holiday for instance, those partitions delete themselves. The data is already in the legacy tables, so an installation in dual keeps working, but the hole in events_full is permanent, and no code detects it and none repairs it. The health probe detects stuck, not lost.

The defence is an alert, and one exists for this. The worker health endpoint accepts ?failIfEventPropagationStuck=true and returns 503 if the propagator’s heartbeat has not been refreshed for more than LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES, which defaults to 35. The comment carries a condition to read before putting it in a Kubernetes probe:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

Another detail of the same mechanism: the propagation cursor lives in Redis, not in Postgres. If Redis is flushed, the cursor resets and every live partition of the staging table is reprocessed.

What can and cannot be undone

The ClickHouse down migrations exist, and they are honest about what they do. The one for 0039 warns in capitals:

WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this table may predate this migration […] Rolling back drops that data regardless of which path created the table.

The ones for the dropped tables recreate the schema and nothing else: “Any data held in event_log before the up migration dropped it is NOT restored.”

But the real point of no return is not in ClickHouse, it is in Postgres. There is a Prisma migration called 20260723150000_drop_legacy_tracing_tables that does exactly what it says:

DROP TABLE IF EXISTS "traces";
DROP TABLE IF EXISTS "observations";
DROP TABLE IF EXISTS "scores";

Prisma Migrate has no down migrations. It applies, and that is that.

Summarising the real reversibility: the write mode can go down from events_only to dual whenever you want, and that is the panic button. What does not come back is what was not written while you were up there: the legacy tables receive nothing retroactively. And step 5 of the chain, the one that drops the scratch tables, ships disabled precisely so you can do forensics if something went wrong.

One last note on the ground you are standing on. The comment in the export-source policy drops something worth reading twice: “legacy-writes-disabled, which in practice only surfaces on self-hosted (Cloud does not run events_only)”. The events_only path is the one you will use and not the one the vendor’s managed service uses. It is less battle-tested in production than dual.

How this looks in code

The minimum runbook, in variables:

# Step 2: upgrade to v4 behaving like v3
LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy
LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=dual_write # "direct" will not start here
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=false

# Step 3: third rail. Both services, web and worker, together.
LANGFUSE_MIGRATION_V4_WRITE_MODE=dual
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=true
LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES=35

# Step 4: only once all five background_migrations rows have finishedAt
LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only
LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true # without this it will not start
LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES=true # last of all

And the query that answers the only question that matters for weeks:

SELECT name, started_at, finished_at, failed_at, failed_reason,
 state->>'chunksLoaded' AS chunks
FROM background_migrations
WHERE name LIKE '20260701_v4_%'
ORDER BY name;

Migration checklist

Every v3 background migration is finished before the image is upgraded.
There is a Postgres and a ClickHouse backup from before the first DROP TABLE.
The write mode has the same value in the worker and in the web, and both change together.
The upgrade went to legacy first, and the service was verified unchanged.
There is an alert on ?failIfEventPropagationStuck=true, with initialDelaySeconds of at least 60 seconds.
Somebody looks at the background_migrations table every day while the backfill runs.
The in-house integrations calling the twenty-four routes that will return 404 have been inventoried.
Client SDKs are on Python 4.0.0 or JavaScript 5.0.0 at minimum before moving to events_only.
Step 5 of the chain is enabled last, and only after verifying the backfill.

Traps

The default is the end state. A v3 installation booting v4 without touching the variable lands in events_only and stops writing the old tables from the first second.

A failed background migration does not block the next one. And the first two steps of the chain declare no predecessor.

More worker replicas do not speed up the backfill. One migration at a time across the cluster, by design.

Forty-eight hours of stopped propagator is a permanent hole. No detection and no repair.

New indexes do not cover old data. Historical searches scan until somebody materialises the indexes by hand.

On a cluster, every ALTER waits for all replicas. One down replica hangs the migration.

Classic ingestion rejects per event. A mixed batch returns 207 and half of it is lost without anyone reading the response body.

History and the live path do not produce quite the same thing. Trace metadata does not descend to children in the backfill.

The route you will take is not the one the vendor takes. Cloud does not run events_only.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window (this article): the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/env.ts, worker/src/env.ts, web/src/env.mjs, web/src/features/public-api/server/deprecations.ts, web/src/pages/api/public/ingestion.ts, web/src/features/public-api/server/createAuthedProjectAPIRoute.ts, worker/src/backgroundMigrations/, worker/src/features/eventPropagation/, packages/shared/clickhouse/migrations/canonical/, packages/shared/prisma/migrations/.
Langfuse compatibility and migration, checked on 14 September 2026.
OpenTelemetry integration migration to v4, checked on 14 September 2026.

Langfuse v4, day 2 (3 of 10): two well-known agents instrumented, Open Deep Research and GPT Researcher

Mon, 14 Sep 2026 07:00:00 +0200

Third article in a series of ten about operating Langfuse v4 in production. The first walked through what goes into a trace; the second measured what an agent turn costs over a minimal graph. This one applies the same to two agents people deploy. Code read on 14 September 2026: Open Deep Research at commit 1b7d2e8 and GPT Researcher at 6f99857, with Langfuse 4.15.2 and LangGraph 1.2.11.

TL;DR

One Open Deep Research request with the default configuration is 1,416 observations and 8.1 MB. Measured over a faithful replica of its topology with five concurrent units, six supervisor iterations and ten react calls per researcher, which are its defaults. It all lands in a single trace. And it is a floor, because on the bench the search results are three hundred characters and real ones are much larger.

None of those 1,416 is of type AGENT. Its nine nodes are called clarify_with_user, write_research_brief, research_supervisor, final_report_generation, supervisor, supervisor_tools, researcher, researcher_tools and compress_research. None contains the string the handler looks for.

The parallel fan-out does not break the trace, and the fear that it does is unfounded. Open Deep Research forwards the config to its subgraphs and GPT Researcher passes a fresh one carrying only tags. I tested both patterns: one trace in both cases, because LangChain rebuilds the config from the context variable.

GPT Researcher is two products in one repository. The multi-agent mode is LangGraph and traces like any graph. The standard mode has no graph, and there every model call opens its own trace: eight calls, eight traces, measured.

Its human node does not use interrupt(), it waits on a websocket. The node’s observation stays open while the person thinks, so its duration measures human latency. The 95th percentile of that graph means nothing unless that node is excluded.

Instrumenting both requires no change to their repositories. For Open Deep Research it is enough to import the compiled graph and pass the handler through config. For GPT Researcher’s standard mode you have to wrap with the SDK, because there is nowhere to hook callbacks.

You are here: OBSERVE, day 2

The previous article produced a formula over a toy graph: 4 + 5n observations per turn, with the byte volume growing with the square of the steps. A formula over a toy is good for understanding the mechanism and no good for sizing anything.

This article applies it to two real agents, chosen because they are well known and because they break in different ways. The series grows to ten articles and the migration one moves along one place.

The analogy: two factories, one audit

A traceability audit in a factory that runs the whole process on its own lines is tedious but complete: every station leaves a record, and the auditor reconstructs the part from beginning to end. The volume of paper is enormous and that is the only problem.

The second factory subcontracts half the process. The auditor sees material going into a building and finished product coming out, with times and quantities, and has nothing about what happens inside. There is little paper. The problem is not the paper either.

Open Deep Research is the first factory and the problem is volume. GPT Researcher, in its standard mode, is the second, and the problem is that the trace does not carry what you need to know. Both are fixable, with different tools, and confusing them leads to buying disk when what is missing is instrumentation.

The two agents, and why these

Open Deep Research (langchain-ai/open_deep_research, 12.7 k stars) is LangChain’s reference research agent. All its work goes through LangChain, so callback instrumentation sees it whole.

GPT Researcher (assafelovic/gpt-researcher, 28.4 k stars) is the best-known open source research agent and it is not from LangChain. Its multi-agent mode uses LangGraph; its standard mode, which is what the API and the web interface run, does not.

One is in-house and the other is not, which for a comparison is what matters.

Open Deep Research: the shape of the graph

Three graphs, two of them nested as subgraphs (src/open_deep_research/deep_researcher.py):

The root graph, deep_researcher (line 701), with four nodes: clarify_with_user, write_research_brief, research_supervisor and final_report_generation.
The supervisor subgraph (line 353), with supervisor and supervisor_tools, iterating between them.
The researcher subgraph (line 589), with researcher, researcher_tools and compress_research, which is a classic react loop.

The multiplication lives in supervisor_tools. When the supervisor decides to delegate, that node launches one researcher subgraph per unit and awaits them together (line 305):

research_tasks = [
 researcher_subgraph.ainvoke({
 "researcher_messages": [HumanMessage(content=tool_call["args"]["research_topic"])],
 "research_topic": tool_call["args"]["research_topic"]
 }, config)
 for tool_call in allowed_conduct_research_calls
]

tool_results = await asyncio.gather(*research_tasks)

The detail that decides whether the trace comes out whole is that config at the end: the node receives LangGraph’s configuration and forwards it to the subgraphs, so the callbacks travel and every researcher hangs off the right place.

The three ceilings in the default configuration (src/open_deep_research/configuration.py) are max_concurrent_research_units = 5, max_researcher_iterations = 6 and max_react_tool_calls = 10. They multiply each other.

What one request costs, measured

I replicated that topology with fake models, the same technique as the previous article: three graphs, the same nesting, the same fan-out with asyncio.gather, three-hundred-character search results and two-hundred-character model replies. No network and no GPU, so it reproduces anywhere.

Units	Iterations	React calls	Observations	generation	chain	Attribute KB	Largest observation
1	1	1	26	8	18	56.8 KB	3.7 KB
2	2	3	92	26	66	257.3 KB	4.6 KB
5	6	10	1,416	370	1,046	8,108 KB	11.0 KB

The last row is the repository’s default configuration. One research request, one person pressing a button once, leaves 1,416 observations and eight megabytes in the observability backend. In a single trace, under a single identifier.

Those eight megabytes are a floor, not a ceiling. On the bench each search result takes three hundred characters; a real Tavily result carrying page content takes between ten and a hundred times more, and that text enters the history and gets re-serialised in every later observation of the same researcher, which is the previous article’s quadratic growth acting on much larger text.

Put into capacity arithmetic: a hundred research requests a day, which for a small team is nothing, is 141,600 observations and on the order of a gigabyte a day uncompressed, with toy searches. With real searches, tens of gigabytes. It is another order of magnitude compared with a chat assistant, and it is the reason this article comes before the capacity one.

Zero AGENT observations

Among those 1,416 observations there is not one of type AGENT. The previous article explained why: the handler decides that type by looking for the string agent in the class path or the run name. Open Deep Research’s nine nodes are clarify_with_user, write_research_brief, research_supervisor, final_report_generation, supervisor, supervisor_tools, researcher, researcher_tools and compress_research.

They are good names. They describe what each node does and they leave out a word that adds nothing for whoever reads the code. The result is that LangChain’s reference research agent, seen from Langfuse, is a tree of chains.

It is fixable from outside, without touching the repository, by naming the subgraph invocation or renaming the two nodes that represent a model decision, which are supervisor and researcher. Without that, any dashboard grouping by observation type will not tell decision work apart from plumbing.

Instrumenting it without touching the repository

Open Deep Research ships as a LangGraph Server graph, with its langgraph.json pointing at deep_researcher. There are two ways to get the handler in, depending on how you run it.

If you invoke it from your own code, the compiled graph is importable and the config is enough:

from langfuse import Langfuse, propagate_attributes
from langfuse.langchain import CallbackHandler
from open_deep_research.deep_researcher import deep_researcher

trace_id = Langfuse.create_trace_id(seed=f"{request_id}")
handler = CallbackHandler(trace_context={"trace_id": trace_id})

with propagate_attributes(
 trace_name="deep-research",
 user_id=user,
 session_id=request_id,
 tags=["odr", "research"],
):
 result = await deep_researcher.ainvoke(
 {"messages": [HumanMessage(question)]},
 config={
 "callbacks": [handler],
 "configurable": {
 "thread_id": request_id,
 "max_concurrent_research_units": 3,
 "max_researcher_iterations": 4,
 },
 },
 )

Lowering those two ceilings from 5 and 6 to 3 and 4 is not an observability recommendation, it is a product decision that also divides the trace volume by three. Better taken at the same time.

If you deploy it on LangGraph Server there is no invocation of yours to put the config into, so the handler gets bound at compile time. The pattern is to wrap the exported graph:

# instrumented_graph.py, and point langgraph.json here
from langfuse.langchain import CallbackHandler
from open_deep_research.deep_researcher import deep_researcher

graph = deep_researcher.with_config({"callbacks": [CallbackHandler()]})

That route loses the per-request seeded trace identifier, because the handler is created once. If the agent uses interrupts, that loss matters, for the reason the previous article measured.

GPT Researcher: two products in one repository

Here is the interesting part, and it is a trap you fall into with any agent that has grown in layers.

The repository holds the gpt_researcher package, which is the engine, and the multi_agents package, which is an orchestration on top. Only the second uses LangGraph.

The orchestrator (multi_agents/agents/orchestrator.py:66) has eight nodes: browser, planner, human, researcher, writer, fact_checker, visualizer and publisher, with two conditional loops, one for human review of the plan and one for fact checking. The researcher node launches in parallel one editor subgraph per section (multi_agents/agents/editor.py:134), and that subgraph has its own revision loop between reviewer and reviser.

All of that traces like any graph. And the models are called with self.llm.ainvoke(messages, **kwargs) from the generic provider (gpt_researcher/llm_provider/generic/base.py:365), which is a LangChain chat model with no explicit config, so it picks up the ambient configuration while running inside a node. The generations show up.

The standard mode is another thing. The GPTResearcher class has no graph, no nodes and does not go through LangGraph at any point. It is the mode the API and the web interface use, that is, the one most people run.

The myth of the fresh config

Before going on, something that looks like a bug and is not has to be dismantled, because it shows up in many agents.

When the researcher node launches the section subgraphs, it does it like this (multi_agents/agents/editor.py:74):

final_drafts = [
 chain.ainvoke(self._create_task_input(research_state, query, title),
 config={"tags": ["gpt-researcher"]})
 for query in queries
]
research_results = [result["draft"] for result in await asyncio.gather(*final_drafts)]

That config is fresh and carries only tags. The method does not even receive the node’s configuration, so it could not forward it. The intuitive conclusion is that the subgraphs lose the callbacks and every section ends up in its own trace.

It does not happen. I set both patterns side by side, Open Deep Research’s forwarding the config and GPT Researcher’s passing a fresh one, and both give one trace with one root span. The reason is that LangChain does not take the config you pass as is: it starts from the ambient configuration held in a context variable and overlays the keys you bring. The callbacks are still there, and context variables are copied into the tasks asyncio.gather creates.

What does get lost are the parent’s tags, replaced by the new list. That is a filtering nuisance, not a broken trace.

The standard mode: eight calls, eight traces

With no graph there is no root run, and with no root run every model call is its own tree. Measured with eight standalone calls to a chat model, with the handler passed by hand on each one:


Observations	8
Distinct traces	8
Root spans	8
Types	8 generation

Eight traces of one observation each, with no relation between them. None of them knows it belongs to the same piece of research. And that is the favourable case, because the standard mode offers no public place to put that handler: the **kwargs reach ainvoke from create_chat_completion (gpt_researcher/utils/llm.py:120), but the path crosses several internal layers that are not meant as an extension point.

The route that works without patching the repository is to wrap from outside with the SDK and let the inner calls hang off that:

from langfuse import get_client, observe, propagate_attributes
from gpt_researcher import GPTResearcher

@observe(name="gpt-researcher", as_type="agent")
async def research(question: str, request_id: str, user: str) -> str:
 with propagate_attributes(user_id=user, session_id=request_id, tags=["gptr"]):
 researcher = GPTResearcher(query=question, report_type="research_report")
 await researcher.conduct_research()
 report = await researcher.write_report()

 client = get_client()
 client.update_current_span(
 output={"length": len(report), "sources": len(researcher.get_source_urls())},
 metadata={"gptr_costs": researcher.get_costs()},
 )
 return report

With that there is one trace per piece of research, with an agent observation at the root and the generations hanging inside, in the same tree. Searches and document chunking do not appear, because they do not go through LangChain. If they are needed, they get added with start_as_current_observation(as_type="tool") around your own calls, which is the first of the three ways out of the span filter the previous article described.

Worth noting researcher.get_costs(). GPT Researcher keeps its own cost accounting, and putting it in as metadata is the only thing that lets you reconcile it later with what Langfuse computes. When the two figures disagree, it is almost always the gateway’s model alias.

The human node that stays open

The multi-agent mode has a human node that asks for an opinion on the plan before researching. It does not use LangGraph’s interrupt(). It waits inside the node, reading from the websocket (multi_agents/agents/human.py).

The two approaches have opposite consequences for the trace. With interrupt(), the graph hands control back and the trace splits in two, which is what the previous article measured. Waiting inside the node, the trace is single but that node’s observation stays open for as long as the person takes, minutes or hours.

Neither is wrong. What is wrong is the dashboard that averages durations without knowing which of the two it has in front of it. With the wait inside the node, the 95th percentile of graph duration measures how long a human takes to answer a message, not how long the agent takes. Alert on that metric and the alert fires at lunchtime.

The practical rule is to exclude human-wait nodes by name from any latency metric, and to measure that wait separately, which tends to be an interesting product metric in its own right.

What to look at in the trace once it arrives

With both agents instrumented, and before building any dashboard, there are four checks that are done once and save weeks:

Count the observations of a typical request. If it comes out above a thousand, this platform’s problem is going to be volume and sampling has to be decided before anything else.
Look for more than one root span. More than one means there is work being traced outside the tree, and you want to know which before there are thousands.
Check that generations carry cost. If they come in at zero, it is the model alias, fixed by registering that name with its price.
Sort observations by duration and remove the ones waiting on a person. What stays at the top is where the time really is.

Checklist

The graph is invoked with a trace identifier seeded per request, not a random one.
The agent’s concurrency and iteration ceilings are set deliberately, knowing they multiply trace volume.
Nodes representing a model decision carry agent in the name, or it has been accepted in writing that there will be no observations of that type.
Work that does not go through LangChain is wrapped with the SDK, or it has been decided that it need not be visible.
Human-wait nodes are identified and excluded from latency metrics.
The agent’s own cost accounting, if it has one, goes in as metadata so it can be reconciled.
There is a measured figure of observations per request, taken from your own installation and not from this article.

Traps

A research agent’s defaults are demo values, not production ones. Five units times six iterations times ten react calls multiply, and whoever chose them was thinking about report quality, not about your ClickHouse.

One repository can hold two agents with two different instrumentation stories. Instrumenting the mode the README documents and not the one the API runs is a mistake with no symptom: the traces arrive, just of something else.

Passing a fresh config to a subgraph does not break the trace. It breaks the tags. The tree survives because the callbacks travel through a context variable.

A node waiting on a person contaminates any duration statistic. And not obviously, because the node finishes cleanly and flags no error.

Good node names and good observation classification are conflicting goals. The winner is whoever writes the agent’s code, who is normally not you.

Eight megabytes per request with three-hundred-character searches. With real results, the number your installation produces will not look like this article’s, and it will be larger.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented (this article): Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Open Deep Research source, commit 1b7d2e8, read on 14 September 2026: src/open_deep_research/deep_researcher.py and configuration.py.
GPT Researcher source, commit 6f99857, read on 14 September 2026: multi_agents/agents/orchestrator.py, multi_agents/agents/editor.py, multi_agents/agents/human.py, gpt_researcher/llm_provider/generic/base.py and gpt_researcher/utils/llm.py.
Own measurement of 14 September 2026 over replicas of both topologies, with LangGraph 1.2.11, langchain-core 1.6.3, Langfuse SDK 4.15.2, InMemorySpanExporter and fake models, no network.
GitHub stars for both repositories, checked on 14 September 2026.

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

Sun, 13 Sep 2026 07:00:00 +0200

Second article in a series of ten about operating Langfuse v4 in production. The first one, what really goes into a trace, walked through the data model by reading the server code. This one measures the other side. Verified against Langfuse 4.15.2 (Python SDK), LangGraph 1.2.11 and langchain-core 1.6.3, with figures taken on 13 September 2026 on a bench with no network.

TL;DR

One agent turn is four observations plus five per tool call. Measured with an in-memory exporter over a minimal ReAct graph: zero tools, 4 observations; one, 9; five, 29; ten, 54; forty, 204. The formula holds exactly across the whole range.

The byte volume grows quadratically, not linearly. The same turn goes from 12.2 KB with one tool call to 1.87 MB with forty. 88 % of those bytes is the input field, because every observation re-serialises the full message history as it stands at that moment.

Tool definitions travel attached to every model call. The handler appends them to the input field as messages with role tool. Measured: 364 bytes per tool. With twenty tools that is 7.3 KB on every single step of the loop.

The AGENT observation type is assigned by string match on the name. A node called planificador comes out as CHAIN; the same node called agente_planificador comes out as AGENT. The logic is literally looking for agent in the class path or in the node name (langfuse/langchain/CallbackHandler.py:397).

An interrupt with a resume produces two separate traces. Verified with invoke, with stream and with subgraphs, and also reusing the same handler object. The SDK ships machinery to reattach the trace, but it only fires if the root run finishes with a control-flow exception, and in LangGraph 1.2.11 the interrupt comes back through the return value.

The fix fits in one line and is measured. Passing trace_context={"trace_id": Langfuse.create_trace_id(seed=thread_id)} when building the handler leaves the interrupt and the resume in the same trace, even with different handlers and therefore with different replicas.

The span processor drops by default whatever it does not recognise. It only exports spans from the SDK’s own tracer, spans carrying some gen_ai.* attribute, or spans from a closed list of thirty-five instrumentation prefixes (langfuse/_client/span_filter.py:11). Your own code’s spans do not arrive unless they come through one of those three doors.

You are here: OBSERVE, day 2

The series was planned with eight articles and has grown to ten. The reason is ordering: the next two are about migrating and about sizing ClickHouse, and both need an input figure that nobody publishes, which is how many observations and how many bytes a real unit of work produces. A chat produces a trace with two observations. An agent produces two hundred. Sizing with the chat figure leads to a cluster that falls over the day someone connects the agent.

So before migrating, measure. And to measure there has to be in front what people actually deploy.

The analogy: the file that gets photocopied whole at every signature

A procurement file passes across six desks. At each desk somebody reads what is there, adds a sheet and passes it on.

There are two ways to leave a record of that route. The first is to write in a register which desk touched it, when, and which sheet it added. Six short entries. The second is to photocopy the whole file on the way into each desk and on the way out, and to archive the twelve copies. The register takes one sheet. The copies take, at the sixth desk, six times what they took at the first, and in total they take something on the order of the square of the number of desks.

LangChain’s callback instrumentation does the second thing. It does not record the delta of each step, it records the complete state on the way into and out of each step. With a chat that makes no difference, because the state is short and there is one step. With a forty-step agent, the archive weighs almost two megabytes per file processed.

This is not a defect that needs fixing. It is what lets you open a trace and see exactly what the model saw at step 17 without reconstructing anything. But it drives the sizing, and the number has to be known before signing the contract.

Which agentic platform is the most used

The question has two answers depending on the metric, and both are defensible.

By PyPI downloads over the last thirty days, checked on 13 September 2026:

Package	Downloads / 30 days
`langgraph`	52.3 M
`strands-agents`	35.2 M
`openai-agents`	22.9 M
`crewai`	19.1 M
`google-adk`	14.5 M
`pydantic-ai`	8.0 M
`llama-index`	5.0 M
`agno`	1.9 M
`autogen-agentchat`	0.6 M

By GitHub stars the order flips at the top: CrewAI is at 58.4 k, LangGraph at 38.1 k and the OpenAI Agents SDK at 28.6 k.

PyPI downloads are inflated by continuous integration and by images that reinstall on every build, so they indicate presence in pipelines rather than agents in production. Stars measure interest, not deployment. With both together, LangGraph is the one that shows up in more pipelines and the one with the most worked-through instrumentation path in Langfuse, so it is the one instrumented here. What follows about volume applies equally to any platform instrumented through LangChain callbacks, because the cost comes from the callback model, not from the graph.

How LangGraph gets into Langfuse

The route is worth stating, because the previous article left two open and only one is used here.

langchain-core 1.6.3 does not import OpenTelemetry anywhere. The check is direct against the installed wheel and returns not one file. LangGraph does not emit OTLP spans on its own either. What exists is LangChain’s callback system, and the Langfuse CallbackHandler hooks into it.

That handler does not talk to the classic ingestion API. It creates observations with the v4 SDK, which underneath is OpenTelemetry: LangfuseSpanProcessor sets up an OTLPSpanExporter against {base_url}/api/public/otel/v1/traces (langfuse/_client/span_processor.py:123). So it is OTLP, but generated by the SDK, not by third-party auto-instrumentation.

The difference matters for one concrete reason left hanging in the previous article. Because the data passes through the SDK calls, the classic masking function does apply: input, output and metadata all go through _process_media_and_apply_mask before becoming attributes (langfuse/_client/span.py:534). With auto-instrumentation of the OpenInference or OpenLLMetry kind, the messages travel in span events and masking does not touch them. For an agent handling customer data inside its tools, that difference decides the route.

The trade-off is that a callback handler only sees what goes through LangChain. HTTP calls a tool makes on its own, the database query, the subprocess, none of that shows up. And if you instrument your code with OpenTelemetry so that it does show up, you run into the filter described further down.

The observation type is decided by the node name

The first article insisted that v4 has ten observation types and that using the right one changes what the interface knows how to group. The LangChain handler emits six of those ten: tool, retriever, generation, agent, chain and span. It never emits event, embedding, evaluator or guardrail.

The assignment is direct for four of them: tool callback, type tool; retriever callback, type retriever; model callback, type generation. For chain callbacks, which is what every node of a LangGraph graph is, the criterion is this (langfuse/langchain/CallbackHandler.py:393):

elif callback_type == "chain":
 # Detect if it's an agent by examining class path or name
 if serialized and "id" in serialized:
 class_path = serialized["id"]
 if any("agent" in part.lower() for part in class_path):
 return "agent"

 name = self.get_langchain_run_name(serialized, **kwargs)
 if "agent" in name.lower():
 return "agent"

 return "chain"

Measured over the same graph with the node renamed and everything else identical:

Node name	Tool calls	Observations	Types
`planificador`	10	54	11 generation, 33 chain, 10 tool
`agente_planificador`	10	54	11 generation, 22 chain, 11 agent, 10 tool

Eleven observations change type because of eleven characters in a function name. This has two practical consequences. The first is that any dashboard counting AGENT observations is counting naming conventions. The second is that the interface draws the agent graph when the trace contains any observation of a type other than span, event or generation, and chain qualifies, so the drawing comes out either way. What changes is what you can filter and aggregate afterwards.

The operational recommendation is boring and it works: name the nodes that represent a model decision with a name containing agent, and leave the rest as chains. It is a naming convention promoted to data schema, which is not pretty, but it is what there is and it is stable.

What a turn costs, measured

The setup is a minimal ReAct graph: one node that calls the model, a ToolNode with one tool, a conditional edge with tools_condition and an in-memory checkpointer. The model is langchain-core’s GenericFakeChatModel, which returns a fixed list of messages, so neither network nor GPU is needed and the experiment reproduces on any laptop. The exporter is OpenTelemetry’s InMemorySpanExporter, passed to the client through the span_exporter parameter, so observations are counted with no Langfuse server in front.

Result, with the node named planificador:

Tool calls	Observations	generation	chain	tool	Attribute bytes
0	4	1	3	0	4.2 KB
1	9	2	6	1	12.2 KB
5	29	6	18	5	64.1 KB
10	54	11	33	10	173.5 KB
20	104	21	63	20	541.4 KB
30	154	31	93	30	1,108 KB
40	204	41	123	40	1,874 KB

The observation count is exactly 4 + 5n. The five per loop iteration are: the model node, the model call, the conditional edge, the tools node and the tool. The four fixed ones are the root graph, the first partial pass and the close.

The bytes do not follow that line. From 10 to 20 steps the volume multiplies by 3.1; from 20 to 40, by 3.5. That is the quadratic growth expected when each of the n steps drags along a history whose length is proportional to n.

The breakdown by field confirms it. With forty steps, out of 1,709 KB measured in that run, 1,502 KB sit in langfuse.observation.input and only 56 KB in the output. 88 % of what is ingested is the same history repeated at different stages of progress. The largest single observation in that turn is 22.2 KB, well below the 9.5 MB per OTLP span ceiling, so the problem is not the size limit: it is the aggregate.

And inside the input there is a second multiplier. The handler appends each tool’s definition to the generation’s input field, as messages with role tool (langfuse/langchain/CallbackHandler.py:1190). Measured with three-parameter tools and a one-line description:

Bound tools	Input field bytes of one generation
0	34
1	398
5	1,854
10	3,674
20	7,334

That is 364 bytes per tool, on every generation, always. An agent with twenty tools and forty steps ingests 7.3 KB × 41 generations = 300 KB in tool schemas alone, which never change over the course of the turn.

What this means for ClickHouse

A honest caveat belongs here. I compressed the full payload of those runs with ZSTD level 3, which is what events_full uses, and got ratios between 29× and 86×, improving the longer the turn. That number is inflated by how synthetic the bench is: my tool always returns the same string, so the redundancy is higher than in production.

What does hold is the shape. Repeating the history is exactly the pattern columnar compression absorbs well, so the ClickHouse disk is not where it hurts. Where it hurts is earlier: in the bandwidth towards the OTLP endpoint, in the worker CPU that decompresses and flattens, and in the ingestion queue, all of which see uncompressed bytes. The first article noted that the queue that saturates first is langfuse.queue.ingestion.depth, and this is why it saturates.

Quick sizing arithmetic, with the measured figures and an average ten-step turn:

1,000 turns a day is 54,000 observations and around 170 MB uncompressed a day.
10,000 turns a day is 540,000 observations and 1.7 GB uncompressed a day.
If the average turn goes from ten steps to twenty, observations double and bytes triple.

Sensitivity to the number of steps is the parameter to watch, more than the number of users. A prompt change that makes the agent take three more turns around the loop multiplies the observability bill with nobody touching the infrastructure. Article 6 of the series comes back to this with ClickHouse system tables to measure the real bytes per observation in your own installation, which is the only thing that replaces this estimate.

Interrupts: two traces where there should be one

A production agent with human approval uses interrupt(), hands control back, waits for a person and resumes with Command(resume=...). The operational question is whether the trace survives that pause.

The SDK ships explicit machinery for the case. There is a store of pending trace contexts keyed by thread_id, capped at 1,024 entries with eviction of the oldest (MAX_PENDING_RESUME_TRACE_CONTEXTS), and a function that recognises a Command carrying resume and recovers the stored context.

Measured, it does not fire in this combination of versions:

Case	Same trace?
Same handler object, `invoke`	No
Same handler object, `stream`	No
Interrupt inside a subgraph	No
Fresh handler for the resume	No

The reason is in the code and it is consistent. The pending context is only stored from on_chain_error when the failing run is the root one and the resulting level is DEFAULT, which is what happens with LangGraph’s control-flow exceptions (CallbackHandler.py:866). In LangGraph 1.2.11 the interrupt does not leave the root as an exception: the graph returns normally, with the interrupt in the return value. The root run ends through on_chain_end and the branch that stores the context never runs. The interrupted node’s observation does come out right: level DEFAULT and the Interrupt in the status message, not marked as an error, which is the correct behaviour.

The fix is verified and does not depend on the SDK changing:

from langfuse import Langfuse
from langfuse.langchain import CallbackHandler

trace_id = Langfuse.create_trace_id(seed=thread_id)
handler = CallbackHandler(trace_context={"trace_id": trace_id})

With that, the interrupt and the resume land in the same trace even if the handler is a different object and even if another replica serves the resume, because the identifier derives from the thread_id and not from in-memory state. It is the same technique the documentation recommends for joining several agents into one trace, applied to joining an agent with itself.

With one nuance to decide deliberately. Seeding with the bare thread_id puts every turn of that conversation into a single trace, which grows without bound while the conversation stays alive. If what you want is one trace per turn with the resume attached to the right turn, seed with thread_id plus the turn number, and leave conversation grouping to the session identifier, which is what it is for. Since v4 has no trace table, a huge trace is not a huge row: it is a filter returning many rows, and what suffers is the interface when opening it.

Thread, session, user, and what is only read at the root

The handler recognises five metadata keys: langfuse_session_id, langfuse_user_id, langfuse_trace_name, langfuse_tags and langfuse_prompt. The first four are read only when parent_run_id is None, that is, only on the root run (CallbackHandler.py:587). Putting them in the config used to invoke a subgraph does nothing.

The recommended route, and the one that works without surprises, is the context manager:

from langfuse import propagate_attributes

with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["production", "support-l1"],
):
 graph.invoke(payload, config=cfg)

What LangGraph contributes on its own arrives anyway. In the measured run these show up as observation metadata: langgraph_node, langgraph_step, langgraph_triggers, langgraph_path, langgraph_checkpoint_ns, checkpoint_ns and thread_id, plus ls_provider, ls_model_type and ls_integration. The thread_id is also promoted to trace metadata.

That is fine, but keeping in mind the first article: in events_core what is indexed is metadata_names, that is the key names, not the values. Filtering by langgraph_node = "planificador" is a scan. If you are going to segment by node routinely, the right place is the observation name, which is where the node already appears, or a tag, assuming that tags are not indexed either. The cheap field for slicing by node in v4 is the observation name itself.

The session identifier deserves an explicit decision. LangGraph’s thread_id and Langfuse’s session_id are the same concept and they do not connect on their own. Mapping them one to one is the reasonable thing, and it is also what makes the whole conversation navigable in the interface even when each turn is a separate trace.

The filter that drops your own spans

This is the finding that surprises most people while setting it up. The Langfuse span processor does not export everything that passes through the TracerProvider. It exports what satisfies one of these three conditions (langfuse/_client/span_filter.py:104):

It was created by the Langfuse SDK’s own tracer.
It carries at least one attribute starting with gen_ai.
Its instrumentation scope matches one of thirty-five prefixes in a closed list.

The list covers the expected and a bit more: openinference, litellm, haystack, langsmith, strands-agents, pydantic-ai, autogen-core, vllm and a couple of dozen opentelemetry.instrumentation.* entries.

The consequence for an agent is direct. If you instrument your tools with OpenTelemetry by hand so that the ERP call or the Postgres query shows up inside the agent’s trace, those spans are dropped silently, with no log and no error. There are three ways out, in order of cleanliness:

Create those observations with the Langfuse SDK, with start_as_current_observation(as_type="tool"), which is what the rest of this article assumes.
Put some gen_ai.* attribute on the span, which is abusing the convention to smuggle the span through, so only if the span does describe an interaction with a model.
Send those spans to your normal OpenTelemetry collector instead of Langfuse, and correlate by trace identifier. For a sovereign platform with its own collector, this is the sensible option: Langfuse keeps the model side, the collector keeps the system side, and the trace identifier joins them.

The behaviour being silent is what makes this take an afternoon to find. It belongs in the runbook.

The model and the cost with a gateway in the middle

One detail that affects anyone putting LiteLLM in front, which is the setup described in the operational pair and in GPU multi-tenancy.

The model name the handler puts on the generation comes first from the ls_model_name metadata and, failing that, from the component’s serialisation (CallbackHandler.py:1297). With a ChatOpenAI pointed at the gateway, that name is whatever alias you defined in LiteLLM, for example qwen-32b-internal. Langfuse computes cost from its own model table, which does not know that alias, so the generation arrives with tokens and without cost.

It is fixed by defining the model in Langfuse under that same name with its price per million tokens, which is also the only way for the cost to reflect what your GPU costs rather than what somebody else’s API costs. It is an afternoon of work and it stops the agentic platform’s cost panel from sitting at zero forever.

How this looks in code

The minimum to have in place for a LangGraph agent to produce useful traces in Langfuse v4:

import os
from typing import Annotated, TypedDict

from langfuse import Langfuse, get_client, propagate_attributes
from langfuse.langchain import CallbackHandler
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition

# 1. Client. Masking applies because the data goes through the SDK.
def mask(*, data, **kwargs):
 return redact_pii(data)

Langfuse(
 public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
 secret_key=os.environ["LANGFUSE_SECRET_KEY"],
 host=os.environ["LANGFUSE_HOST"],
 mask=mask,
 environment="prod", # lowercase, dashes, 40 chars, must not start with "langfuse"
 sample_rate=float(os.getenv("LANGFUSE_SAMPLE_RATE", "1.0")),
)

# 2. Graph. The decision node carries "agent" in its name on purpose.
class State(TypedDict):
 messages: Annotated[list, add_messages]

graph = StateGraph(State)
graph.add_node("agent_planner", model_node)
graph.add_node("tools", ToolNode(TOOLS))
graph.add_edge(START, "agent_planner")
graph.add_conditional_edges("agent_planner", tools_condition)
graph.add_edge("tools", "agent_planner")
app = graph.compile(checkpointer=checkpointer)

# 3. Per turn: deterministic trace seeded with thread and turn, session = thread.
def run_turn(thread_id: str, turn: int, text: str, user: str):
 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {
 "callbacks": [handler],
 "configurable": {"thread_id": thread_id},
 }
 with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["agent", "support"],
 ):
 out = app.invoke({"messages": [HumanMessage(text)]}, config=cfg)
 return trace_id, out

# 4. Resume after human approval: same seed, same trace.
def resume_turn(thread_id: str, turn: int, decision):
 from langgraph.types import Command

 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {"callbacks": [handler], "configurable": {"thread_id": thread_id}}
 return app.invoke(Command(resume=decision), config=cfg)

# 5. Score the turn. Scores do not travel over OTLP: they have their own route.
get_client().create_score(
 trace_id=trace_id,
 name="resolved",
 value=1,
 data_type="BOOLEAN",
)

Two notes on what is not in the example. sample_rate is head sampling and per trace, so it samples whole turns, which is what you want: half an agent trace is no use to anybody. And if the process is a web server, the client is built once at startup, not per request, whereas the handler can be per turn with no problem, because with the seeded identifier it no longer holds state worth preserving.

Agent instrumentation checklist

Nodes representing a model decision carry agent in the name. The rest do not.
Every turn has a deterministic trace identifier seeded with thread and turn number.
LangGraph’s thread_id goes as Langfuse’s session_id, always.
Trace attributes are set with propagate_attributes, not as metadata on a subgraph.
The gateway’s model alias is registered in the Langfuse model table, with a price.
Tools doing work outside LangChain are instrumented with the Langfuse SDK, not with bare OpenTelemetry.
There is an alert on ingestion queue depth, and the average number of steps per turn is known.
Someone has measured, on the actual installation, how many observations and how many bytes a real turn produces. The figures in this article come from a synthetic bench.
Sampling is set through an environment variable, so it can be lowered without deploying code.

Traps

Type AGENT does not mean there is an agent. It means somebody wrote agent in the name. And the other way around: a whole graph without that string produces not one agent-type observation.

Putting langfuse_session_id in a subgraph’s metadata does nothing. It is only read on the root run.

An interrupt splits the trace in two. Unless you seed the identifier. And with several replicas, the handler’s in-memory state is never going to save you, because the resume can land on another pod.

Your OpenTelemetry spans do not arrive. The default filter drops them without warning if the scope is not in the list and there are no gen_ai.* attributes.

The number to watch is not users, it is steps per turn. Volume grows with the square of the steps, and the steps are decided by the prompt, not by the infrastructure.

You store megabytes and the interface shows you a thousand characters. The materialised view’s truncation to two hundred characters and the thousand-character read ceiling are still there. You pay for the full ingestion and see a clipping, unless you open the specific observation.

Cost comes out at zero with the gateway in front. Model alias unknown to the price table.

flush_at above the queue size blows up when constructing the client. With an OpenTelemetry ValueError that does not mention Langfuse anywhere.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front (this article): instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Own measurement of 13 September 2026: LangGraph 1.2.11, langchain-core 1.6.3 and Langfuse Python SDK 4.15.2, with OpenTelemetry’s InMemorySpanExporter and GenericFakeChatModel, no network.
Langfuse SDK 4.15.2 source: langfuse/langchain/CallbackHandler.py, langfuse/_client/span_filter.py, langfuse/_client/span_processor.py, langfuse/_client/span.py.
PyPI downloads over the last thirty days via pypistats, checked on 13 September 2026.
GitHub stars for LangGraph, CrewAI and the OpenAI Agents SDK, checked on 13 September 2026.

Langfuse v4, day 2 (1 of 10): the data model changed underneath, and what really goes into a trace

Sat, 12 Sep 2026 07:00:00 +0200

First article in a series of ten about operating Langfuse v4 in production. The architecture of version 3 and its ten performance levers were already covered in Langfuse from the inside; the seam with the gateway, in the operational pair. This series is about day 2. Verified against Langfuse 4.35.0, commit of 11 September 2026.

TL;DR

Version 4 is observations first. There is no trace table. A trace is a set of observations that share an identifier, and the trace attributes live denormalised in every row: name, user, session, tags, version, release and environment (clickhouse/migrations/canonical/0039_create_events_full.up.sql). The whole intuition of “first I create the trace and then I hang things off it” stops describing the storage.

There are ten observation types, not three. Alongside the classic SPAN, EVENT and GENERATION come AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (packages/shared/src/domain/observations.ts:5). Using the right type changes what the interface knows how to group.

The real limits are in the code and they do not match what people assume. The body of a request to the classic ingestion route is capped at 4.5 MB (api/public/ingestion.ts:40). The OTLP body, at 512 MiB, compressed and decompressed, with a 413 when it is exceeded. An OTLP span above 9.5 MB is only written to the log, not rejected. Identifiers accept 800 characters. The trace name, 1,000.

The ingester looks for the user in five places and the session in six. With strict precedence order, starting from Langfuse’s own attributes and ending in the metadata of other frameworks. Among the session ones sits gen_ai.conversation.id, which is the OpenTelemetry standard, so a conformant instrumentation works without translation.

Scores do not travel over OTLP. Zero occurrences of the word across the 3,864 lines of the OTLP ingestion processor. They come in through the classic ingestion route, through the scores endpoint or through the interface. Any design that assumes an evaluator can write its score down the same channel as the trace is wrong.

The classic masking function does not cover what almost everyone thinks it covers. It only acts on data that passes through the SDK calls. The attributes produced by a third-party automatic instrumentation come out unmasked. There is a new function that does act on the batch attributes, but it leaves out span events, which is where the recent conventions put the messages.

Filtering by tags or by status message is a scan. In the listings table, the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the dates, the model and the metadata names. The name, the tags, the version and the status message have no index.

You are here: OBSERVE, day 2

The previous series about this pair covered selection and setup. The six-service architecture explained version 3; tracing with OpenTelemetry explained the pipeline; the pair with the gateway explained where the trace gets lost between the two pieces.

What is left uncovered is the work of the next three years: migrating, sizing ClickHouse, retaining and deleting, taking backups, alerting, getting the data out. This series is about that, and it starts with the question that conditions everything else, which is what gets stored.

The analogy: the archive that changed system

An archive with a folder per case file and documents inside each folder is easy to understand. You open the folder, you look at what is there. The problem shows up when you have to find every document of one kind over the last two years: you have to open every folder.

Large archives solve that the other way around. There is no folder: each document carries, copied into its header, the case reference, the client name and the date. Searching by client is immediate. Reconstructing a case file is grouping by reference. In exchange, each document weighs a little more, and what gets written into that header matters a great deal, because everything you can search for comes from there.

Langfuse v4 made exactly that change. What used to be a folder with attributes of its own now travels repeated in the header of every document. The two operational consequences of this article come from there: which fields are worth filling in and which fields are not worth filling with content.

The data model, as it is in the code

The tables

Three new pieces in ClickHouse:

events_full (migration 0039): the table of record, with the full input and output, compressed with ZSTD level 3 and indexed with full-text indexes over the lowercase versions. ReplacingMergeTree engine ordered by event timestamp and deletion flag, partitioned by month of the start time.
events_core (migration 0040): the listings table, lighter.
events_core_mv (migration 0041): the materialised view that fills the previous one, and which truncates to two hundred characters the input, the output and each metadata value.

That truncation to two hundred characters is the first thing to know. What you see in a listing is not what is stored; it is a clipping. And when reading a full trace a second cap applies on top, LANGFUSE_SERVER_SIDE_IO_CHAR_LIMIT, with a default value of a thousand (packages/shared/src/env.ts:513).

In the events_full table there is a configuration line that says a lot: index_granularity_bytes = '64Mi', with the comment that it avoids very small granules because of large rows. It is the schema itself acknowledging that the rows of this system are bulky.

The ten observation types

SPAN, EVENT, GENERATION, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (domain/observations.ts:5), with their corresponding ingestion events (server/ingestion/types.ts:279).

For a platform with agents, the difference between marking a tool call as a generic SPAN or as a TOOL is the difference between being able to count tool calls per session and having to reconstruct it by hand. It is worth reviewing the instrumentation with this list in front of you.

The level is still DEBUG, DEFAULT, WARNING and ERROR (domain/observations.ts:31), with a free-form status message.

The limits, with their source

Field	Limit	File
Trace, span and observation identifiers	800 characters, no carriage return	`ingestion/types.ts:10`
Trace name	1,000 characters	`ingestion/types.ts:428`
Environment	40 characters, pattern of lowercase, digits, hyphen and underscore	`ingestion/types.ts:226`
Text-type score	500 characters	`domain/scores.ts:44`
Score config name	35 characters	`domain/score-configs.ts:11`
Classic ingestion body	4.5 MB per request	`api/public/ingestion.ts:40`
OTLP body	512 MiB, adjustable	`web/src/env.mjs:583`
OTLP span considered large	9.5 MB, only logged	`packages/shared/src/env.ts:193`
Name, status message, version, release, tags	no declared limit	`ingestion/types.ts:424`
Metadata	no declared limit	`ingestion/types.ts:425`

The last two are the dangerous ones, because they have no schema cap and the real ceiling ends up being the one on the request body. A three-megabyte metadata value passes validation and reaches ClickHouse.

A note on the langfuse prefix in the environment field: it is reserved for the product’s own internal traces and is stripped idempotently from public inputs (ingestion/types.ts:228), including the case of someone trying to sneak it in duplicated. The default value is default.

On rate limits: in the cloud there are per-plan quotas, but in self-hosted they are not applied. The service returns directly that there is no limiting (features/public-api/server/RateLimitService.ts). It is good news for bulk exports and bad news for anyone relying on that mechanism as protection.

The two instrumentation routes

What gets lost over OTLP, and it is one specific thing

The check is simple and the result is blunt: there is not a single occurrence of the word score in the OTLP ingestion processor, a file of 3,864 lines. Scores only come in through three places: the score-create event of the classic ingestion route, the POST /api/public/scores endpoint, or the interface.

This has a design consequence worth being clear about before building evaluators: the trace can travel through the OpenTelemetry collector with everything good that brings, on-disk queue, retries and sampling, but the score an evaluator puts on it has to go down a different channel and with the project keys.

Everything else does travel over OTLP. Whatever is not in the mapping lists is kept as metadata converted to a string, visible but outside the indexed fields.

The precedence chain, which is the map you need to have

Verified in packages/shared/src/server/otel/OtelIngestionProcessor.ts:

User identifier (line 2545), in this order: langfuse.user.id, user.id, the observation metadata, the trace metadata, and lastly the Vercel SDK metadata.

Session identifier (line 2565): langfuse.session.id, session.id, gen_ai.conversation.id, observation metadata, trace metadata, Vercel metadata.

Environment (line 2383): langfuse.environment, deployment.environment.name, deployment.environment; for each key the span attributes are checked first and the resource attributes after.

Model (line 2753): langfuse.observation.model.name, gen_ai.response.model, ai.model.id, gen_ai.request.model, llm.response.model, llm.model_name, model.

Level (line 1250): the native attribute, with accepted aliases where SUCCESS and OK become DEFAULT, WARN becomes WARNING and FATAL or CRITICAL become ERROR; if there is no attribute and the span status is an error, the result is ERROR.

Tags (line 3227): langfuse.trace.tags, langfuse.tags, observation and trace metadata, Vercel metadata, and tag.tags.

Input and output (line 1790): a chain of some twenty frameworks in strict order, starting from Langfuse’s own attributes and going through Genkit, the Vercel SDK, the standard inference operation details event of the conventions from 1.37 onwards, the per-role message events, the prompt and completion attributes, Logfire, MLflow, TraceLoop, OpenInference and the gen_ai.input.messages and gen_ai.output.messages attributes.

That this chain has twenty entries says something about the real state of the conventions: it was already noted that the generative AI conventions were extracted into their own repository without any published version. While that remains the case, the backend has to guess, and it guesses with a precedence list.

The practical part: if Langfuse’s own attributes are emitted, they always win. If you want backend-neutral instrumentation, gen_ai.conversation.id and deployment.environment.name are the two standard attributes that Langfuse does understand.

Metadata, and how it ends up in the table

A metadata value is flattened into two parallel arrays, one of names and one of values, with dot-separated paths and everything converted to a string (otel/utils.ts:239, applied in worker/src/services/IngestionService/index.ts:363). A nested object with twenty keys is twenty entries in each array.

There is a cap of 10,001 positions when reconstructing arrays from OTel attributes (OtelIngestionProcessor.ts:271), which is high but it exists.

Sessions, users and cardinality

A session is a text column with a Bloom filter index at 1 %. The same goes for the user. The Postgres table that accompanies sessions stores only whether it is bookmarked, whether it is public and its environment; there is no counter and no cap. There is no limit on traces per session in the code, and the documentation recommends session identifiers below 200 ASCII characters.

The cardinality difference matters and it is not intuitive: user and session are plain text, so high cardinality does not break the schema, it only inflates the Bloom filters. The environment, by contrast, is declared low cardinality. Generating a dynamic environment value per tenant degrades ClickHouse measurably.

Aggregated cost comes out of a cost details map with materialised input, output and total columns. The classic sessions endpoint is deprecated; the replacement is the version 2 observations one with a filter.

Scores: five types and three sources

The types are NUMERIC, CATEGORICAL, BOOLEAN, CORRECTION and TEXT (domain/scores.ts:45). The boolean requires a value of zero or one. The correction does not accept an associated config. The text one is capped at 500 characters.

The sources are API, EVAL and ANNOTATION. The EVAL source is reserved for the internal evaluators: the public API only accepts the other two. Annotation scores require an associated config except for the correction one.

On what costs money, a clarification that corrects a widespread belief: in 4.35.0 the entitlement for the number of model-based evaluators is unlimited on every plan, including the open edition (features/entitlements/constants/entitlements.ts:60). What is limited by plan is the number of annotation queues and the days of data access. And there is a version nuance: trace-level evaluators are discontinued in version 4.

Masking, and what it does not cover

This is the part to read twice if the deployment has to comply with anything.

The classic Python SDK function has the signature masking_function(*, data, **kwargs) and covers only the data that passes through the SDK calls: observation creation, update, and setting the trace input and output. It does not cover the raw OTel attributes produced by a third-party automatic instrumentation.

The replacement is a function that acts on the attributes and the resource attributes in the export batch. That one does reach automatic instrumentation, with a limitation declared in the documentation: it cannot change the span name, the identifiers, the parent relationship, the resource attributes, the events or the links.

And there is the gap that matters: the generative AI conventions from 1.37 onwards put messages and responses into span events. That is, the route by which prompts arrive in a modern instrumentation is precisely the one the new masking does not touch.

For a deployment with personal data, the conclusion is that client-side masking is not enough and a redaction processor has to go into the collector. It is the same reasoning that led in the previous article to recommending the collector in the middle, and it adds to the already documented problem of MCP tool arguments being written in clear, bypassing the message redaction switch.

On the server side there is no safety net: Langfuse has no server-side personal data masking. The encryption key that gets configured protects secrets, that is, model provider credentials, integration headers and SSO configurations. Trace inputs and outputs are not encrypted at application level.

Volume control

Three levers, in order of how high up they act:

In the SDK: sample_rate, or the LANGFUSE_SAMPLE_RATE variable, with head sampling at trace level. If the trace is not sampled, neither its observations nor its scores travel. LANGFUSE_TRACING_ENABLED set to false switches the whole thing off.

In the collector: probabilistic sampling and tail sampling. It is the only layer where you can decide based on the outcome, that is, keep every error and a fraction of the rest.

In the server: a deterministic per-project sampling, with a SHA-256 hash of the trace identifier (server/ingestion/sampling.ts). Deterministic per trace, so it does not cut traces in half.

What not to put into a trace

With the schema in front of you, the list is concrete.

Do not filter by tags or by status message. In the listings table the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the creation and update dates, the model name, the experiment identifier and the metadata names. The name, the tags, the version, the release and the status message are not. Filtering by them scans.

Do not use metadata as a store. It gets flattened into string arrays, truncated to two hundred characters in the listings table but stored whole in the full table, and on top of that it is indexed with a full-text index segmented on non-alphabetic characters. Bulky metadata multiplies the inverted index, not just the data.

Do not store full prompts without thinking about it. The input and the output carry ZSTD level 3 compression and two full-text indexes over their lowercase versions. Storing the entire context of a 142,000-token agentic call on every turn inflates the inverted index disproportionately. Client-side truncation is the cheap lever.

Do not use the environment to separate tenants. It is low cardinality, it has a 40-character cap and it isolates nothing: same project, same keys, same limits, same retention. It is there to segment dashboards. Real isolation means separate projects, as was already argued in the article on the seams.

Version 3 and the calendar date

The constant is in the code: V3_SUNSET_DATE = "2026-11-16" (features/public-api/server/deprecations.ts:11).

It is worth being precise about what it means, because the public documentation and the code do not say exactly the same thing. The code comment is literal: the classic ingestion route is never switched off and continues to accept score events; trace and observation events fail only in version 4’s exclusive write mode, not in the mixed or classic modes. And when they fail, they fail per event, not per batch.

The x-langfuse-ingestion-version: 4 header activates direct writing to the events table, which is the real-time route. Without it the deferred path is taken. Recent SDKs send it by themselves. Any value above four is rejected.

As for names, trace-level input and output attributes are deprecated in favour of the observation ones on the root span. The classic read endpoints carry a deprecation marker in the response body, and their replacements are version 2 of observations, version 3 of scores and version 2 of metrics. And dataset runs are renamed experiments.

There is a curious piece that deserves a mention because it shows up in the Redis dashboards and is misleading: a queue called version 4 classic API usage, which ingests nothing. It is a periodic task every fifteen minutes that scans the ClickHouse query log to detect which projects are still using the old API, and materialises the result into Redis for the interface’s migration notice.

How this looks in code

Two pieces, the client one and the collector one.

On the application side, what changes compared with the usual instrumentation is the observation type and where the identity comes from:

from langfuse import get_client, propagate_attributes

lf = get_client()

# identity propagates by context, not observation by observation
with propagate_attributes(
 session_id=id_conversacion, # the same across every turn
 user_id=sujeto_del_token, # from the claim, not from the body
 environment="produccion",
 tags=["soporte", "n2"],
):
 # the type matters: the interface groups by it
 with lf.start_as_current_observation(
 as_type="agent", name="resolver-incidencia"
 ) as raiz:

 with lf.start_as_current_observation(
 as_type="tool", name="buscar_en_inventario"
 ) as herramienta:
 resultado = mcp.call("buscar", args)
 # truncate here, not on the server
 herramienta.update(output=str(resultado)[:4000])

 # the score goes down its own channel, never over OTLP
 lf.score_current_trace(name="resuelta", value=1, data_type="BOOLEAN")

The as_type parameter accepts nine literals: span, generation, embedding, agent, tool, chain, retriever, evaluator and guardrail. The tenth type in the model, the event, has its own constructor because it does not open a scope.

On the collector side, the redaction processor that compensates for what the SDK masking does not reach:

processors:
 redaction:
 allow_all_keys: true
 blocked_values:
 - '\b\d{8}[A-HJ-NP-TV-Z]\b' # national ID number
 - '\b[\w.+-]+@[\w-]+\.[\w.]+\b' # email
 summary: silent
 tail_sampling:
 policies:
 - name: errores
 type: status_code
 status_code: {status_codes: [ERROR]}
 - name: resto
 type: probabilistic
 probabilistic: {sampling_percentage: 5}

exporters:
 otlphttp/langfuse:
 endpoint: http://langfuse-web.observabilidad.svc:3000/api/public/otel
 headers:
 Authorization: "Basic ${LANGFUSE_BASIC}"
 x-langfuse-ingestion-version: "4"
 sending_queue:
 enabled: true
 storage: file_storage
 retry_on_failure:
 enabled: true

The endpoint carries the base path without the signal suffix: the exporter adds its own. And the on-disk queue is the main reason for putting the collector in the middle, because the SDK’s batch processor drops when it fills up.

Instrumentation checklist

Review the type of every observation against the list of ten. Tool calls as TOOL, agents as AGENT, guardrails as GUARDRAIL.
Decide where the user identifier comes from and set it in a single place, high up the precedence chain.
Emit gen_ai.conversation.id if you want neutral instrumentation, or Langfuse’s own attribute if coupling does not matter.
Set the environment with deployment.environment.name, with few distinct values and never one per tenant.
Get scores out through their own channel, with the project keys, and do not expect them to travel with the trace.
Truncate input and output on the client before sending them, especially with long agentic contexts.
Put a redaction processor in the collector, because the SDK masking leaves out span events.
Do not filter by tags in dashboards that get queried often.
Send the ingestion version header, or use recent SDKs that send it by themselves.
Review the size of the metadata: there is no schema cap, the cap is the one on the request body.

Traps and things that are not what they look like

What you see in a listing is truncated to two hundred characters by the materialised view, and the full read has another cap of a thousand. The data is whole in the full events table.

Metadata has no size limit in the schema. A blob of megabytes passes validation.

An OTLP span over 9.5 MB is not rejected, it is only logged. There is no signal back to the client.

The environment is declared low cardinality. One value per tenant degrades ClickHouse.

Scores do not travel over OTLP. It is by absence in the code, not by configuration.

The classic masking does not cover automatic instrumentation, and the new one does not cover span events, which is where the recent conventions put the messages.

Langfuse does not encrypt trace inputs or outputs. The encryption key protects integration credentials.

In self-hosted there are no rate limits on the API. It is no use as protection.

The classic ingestion route is not switched off on 16 November, it changes behaviour depending on the write mode, and it fails per event instead of per batch.

Model-based evaluators are not limited by plan in this version, contrary to what is usually assumed. What is limited are the annotation queues and the days of data access.

The series: the ten articles

This is the first. The order is designed so that each one can be read on its own and so that together they cover the whole of day 2.

What goes into a trace (this article): version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Anuncio de Langfuse v4, 17 de agosto de 2026.
Compatibilidad y migración, consultado el 12 de septiembre de 2026.
Sesiones, muestreo y enmascarado, consultados el 12 de septiembre de 2026.
Integración nativa con OpenTelemetry, consultado el 12 de septiembre de 2026.

The gateway does not live alone: the JWT that does not validate the audience, the user who never reaches the trace, and the seams you have to write by hand

Sat, 12 Sep 2026 06:30:00 +0200

Eighth article in the operational track of the control layer. The previous seven deal with the gateway from the inside: the pair with Langfuse, day 2, virtual keys, humans and agents, prefix routing, the MCP gateway and sizing for agents. This one deals with what surrounds it. Verified against LiteLLM 1.102.0 and Langfuse 4.35.0, both with a commit of 11 September 2026.

TL;DR

The gateway has five neighbours: a database, a cache, an identity provider, a trace backend and a collector. Each neighbourhood has its own physics, and these are the seven things it decides.

The proxy starts without Postgres, but then only the master key works. The DATABASE_URL variable is optional (proxy_server.py:1163); without it, any key other than the master one returns a 400 for a missing connection (user_api_key_auth.py:1886). With a database configured but down, the default behaviour is to return 503, unless general_settings.allow_requests_on_db_unavailable is switched on.

Redis is never mandatory, and that is precisely the trap. There is a circuit breaker with a threshold of five failures and a recovery of sixty seconds (constants.py:456). When it opens, the proxy fails no request: it degrades to local memory. What is lost silently is that rate limits stop being global and become per pod.

No observability backend can bring a request down. All callbacks are launched with asyncio.create_task and their exceptions are caught and logged (litellm_logging.py:3235). If Langfuse does not respond, events are discarded as the queue fills up. There is no added latency, and there is no reliable record either: the pair’s observability is a statistical figure, not an audit record.

JWT authentication validates neither the audience nor the issuer unless two variables are defined. In _build_decode_kwargs (handle_jwt.py:1003) the audience comes from JWT_AUDIENCE and the issuer from JWT_ISSUER; if they are empty, the code sets verify_aud=False and verify_iss=False and emits a warning once only. Any valid token signed by that provider, issued for another service, is accepted.

Role-based access control ships disabled. enforce_rbac has a default value of False (_types.py:4848 onwards). With that value, a token whose role resolves to nothing does not produce a 403: it carries on through the flow. Real authorisation, then, comes from team membership, if there is a team claim configured at all.

The user identifier that reaches Langfuse is not the one from Keycloak. The user_id field of the trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604), that is, with the user field of the OpenAI body, which the client sets. The subject of the token and the owner of the key do not travel. Anyone who wants per-person traceability needs to configure end_user_id_jwt_field or make the client send that field.

Langfuse provisions nothing from a claim. There is no variable that maps an OIDC claim to an organisation or to a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID and LANGFUSE_DEFAULT_PROJECT_ID, the same for everyone. On top of that, per-project access control sits behind the Enterprise licence in the self-hosted edition, as do audit logs and per-project retention.

You are here: between the four pieces

The LiteLLM and Langfuse pair walked through the seam between the gateway and the trace backend: integration routes, cost that arrives as zero, correlation and queues with discarding. This article widens the focus to the other three neighbourhoods, with identity at the centre, because that is the one that orders everything else and the one an auditor asks about.

The identity piece itself, that is, Keycloak, is dealt with in the next article. Here we deal with the gateway side.

The analogy: the building site with five trades

On a small building site, the problem is rarely inside each trade’s work, it is at the joins: where the plumber leaves the pipe sleeve and the bricklayer closes up the partition wall, where the electrician wants to run through the same gap. Each trade does its part well and the join ends up badly resolved because it belongs to nobody.

An inference platform has five trades. The gateway knows about tokens, the identity provider knows about people, the trace backend knows about events, the database knows about spend and the cache knows about counters. Each one has good documentation. The joins have nobody’s documentation, and that is where the user’s identity gets lost between the token and the trace.

There is one more detail of the analogy that is useful. On a building site, the part that causes the most problems is not the one that looks badly finished, it is the one that gets covered up. The same happens here: the five seams in this article fail silently.

Seam 1. What brings things down and what degrades them

It pays to have written down what happens when each neighbour falls over, because intuition fails in almost every case.

Neighbour	Does it start without it?	If it falls over
Postgres	Yes, master key only	503 by default; degraded with `allow_requests_on_db_unavailable`
Redis	Yes	Circuit breaker after 5 failures; limits become per pod
Langfuse	Yes	Events discarded as the queue fills; no impact on latency
OTel collector	Yes	Same as the previous one
Identity provider	Yes	Access to the interface and the JWT flow goes down; virtual keys carry on

Postgres

Without DATABASE_URL, _setup_prisma_client returns None without error (proxy_server.py:10375). The proxy serves /chat/completions with the master key. Any other key gets a 400. There is no global budget either, and start-up itself warns about it: Redis does not replace the database (proxy_server.py:9240).

With a database configured and down, the exception handler re-raises unless the degradation flag is active (db/exception_handler.py:56). There is also a DISABLE_PRISMA_HEALTH_CHECK_ON_STARTUP and a watcher that reconnects. The detail that matters for diagnosis: a transient database failure returns 503, not 401. If during an incident you see 401s, the problem is credentials; if you see 503s, it is the database or admission.

Redis

The circuit breaker is enabled by default, with a threshold of five failures, a recovery of sixty seconds and a minimum duration of five (constants.py:456, logic in caching/redis_cache.py:153). The socket has a timeout of 0.1 seconds (constants.py:453). With the circuit open, DualCache serves from memory without propagating an exception (dual_cache.py:215).

The consequence has to be measured, not assumed: with four workers and three replicas, a limit of one hundred requests per minute becomes one thousand two hundred. It is a failure mode that generates no errors and that can last for days without anyone noticing, until the bill arrives or the engine saturates.

The right alert is not about Redis, it is about litellm_service_latency labelled by service (integrations/prometheus_services.py:119), which exposes Redis and database latency separately.

The callbacks

Everything behind, that is, Langfuse and the collector, is launched outside the response path and its failures are caught. The classic integration uses the v2 SDK with its own background thread and a flush interval of one second, adjustable with LANGFUSE_FLUSH_INTERVAL (langfuse.py:198). There is a cap of fifty instantiated Langfuse clients (constants.py:554), because each client is a thread.

The OTLP integration uses the batch processor of the OpenTelemetry SDK (opentelemetry.py:3060) with its default values. In both cases, saturation means discarding, and discarding generates no error towards the client.

Seam 2. Identity: the three doors and what each one validates

Everything goes through a single authentication constructor (user_api_key_auth.py:1245), in this order:

Custom authentication from the enterprise edition.
Public routes.
Opaque OAuth2, if enabled, with a licence check.
JWT, if general_settings.enable_jwt_auth is active and the token has three parts.
Master key, with constant-time comparison.
Virtual key, with SHA-256 hash and lookup in the verification table.

A JWT works for /chat/completions, not just for administration: the routes allowed by default for a team include openai_routes (_types.py:4886), which contains the chat endpoint. Administration, on the other hand, is only reachable by a token with an administrator role.

The first thing to know: it is paid for

The JWT branch carries an explicit licence check, with the literal message that JWT authentication is an enterprise-only feature (user_api_key_auth.py:1416). SSO for the administration interface is free up to five billable users and requires a licence above that (ui_sso.py:976).

This conditions the architecture of any sovereign deployment that does not want to pay for a licence: the Keycloak integration is left for the administration interface and for provisioning, while inference traffic authenticates with virtual keys. Virtual keys are not worse, but they are a different thing, and it has to be said in the risk analysis: they are credentials decoupled from the identity provider, so removing a person from Keycloak does not invalidate their key.

The `litellm_jwtauth` fields that decide security

From the LiteLLM_JWTAuth class (_types.py:4848 onwards), these are the ones that matter, with their default values:

Field	Default	What it does
`enforce_rbac`	`False`	If false, an unresolved role does not produce a 403
`enforce_scope_based_access`	`False`	Without this, `scope_mappings` is not applied
`enforce_team_based_model_access`	`False`	Model access by team
`team_id_jwt_field`	`None`	Claim the team comes from
`team_id_upsert`	`False`	If false, a team that does not exist gives a 404
`user_id_upsert`	`False`	Same with the user
`public_key_ttl`	`600`	Cache of the public keys
`public_key_stale_ttl`	`3600`	Extra margin if the provider is down
`admin_jwt_scope`	`litellm_proxy_admin`	Scope that grants administration
`user_allowed_email_domain`	`None`	Restriction by email domain

The accepted algorithms are RS256/384/512, PS256/384/512, ES256/384/512 and EdDSA (handle_jwt.py:163). There is no symmetric one, which eliminates the algorithm confusion class of attack. It is a good design decision and it deserves recognition.

Key download supports several comma-separated URLs in JWT_PUBLIC_KEY_URL, resolves discovery if the URL points to the well-known OpenID configuration, retries three times with increasing waits and memoises the failure for thirty seconds (handle_jwt.py:680, :751, :88). If the provider is down, it serves the stale copy up to the sum of the two TTLs, that is, up to seventy minutes. For availability that is fine; for revocation it is not, and it has to be written down.

One detail of key rotation: with more than one key in the set, an exact match of the key identifier is required, but with a single key and no identifier it is accepted without checking (handle_jwt.py:913).

The validation that does not happen

This is the part to fix on day one. In _build_decode_kwargs (handle_jwt.py:1003), the audience is taken from the JWT_AUDIENCE environment variable and the issuer from JWT_ISSUER. If they are not defined, the code sets verify_aud=False and verify_iss=False, and emits a warning once only.

What that means in a real deployment with Keycloak: a token issued for the Grafana client, or for the Backstage one, signed by the same realm, is accepted by the gateway as a valid credential. Expiry is always validated, with zero margin. So is the signature. The audience is not.

There is a second route, issuers, with per-issuer configuration, which does validate issuer and audience unless explicitly disabled (handle_jwt.py:1186). That is the one to use when there is more than one provider.

The minimum fix is two lines of environment:

JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma

And on the Keycloak side, an audience mapper in a dedicated client scope that injects that audience. The detail is in the Keycloak article, because the standard that would resolve this cleanly, resource indicators, is not supported.

From claim to permission

The administrator role comes from admin_jwt_scope appearing in the scope claim, and it returns a result without going through a team (handle_jwt.py:309, :1373). The rest is resolved by team.

If the team in the claim does not exist in the database, with team_id_upsert false a 404 is returned with the message that the team has to be created (auth_checks.py:3035). With the flag active, it is created by invoking team creation with a synthetic administrator identity (auth_checks.py:2874). That is the only automatic provisioning route that exists from Keycloak towards the gateway, and it works well. It pays to know that it creates teams with no budget and no limits, so a default value is needed in the configuration.

The vulnerability worth knowing about

CVE-2026-35030, scoring 9.1 on CVSS 3.1 and 9.4 on CVSS 4.0, advisory GHSA-jjhc-v7c2-5hh6. With JWT authentication active, the cache of the OIDC user info endpoint used the first twenty characters of the token as the key, so that two tokens with the same prefix were confused with each other. Fixed in 1.83.0.

In 1.102.0 the code uses the full SHA-256 hash (handle_jwt.py:962). A search for similar patterns across the tree finds no authentication cache indexed by token prefix: the truncations that remain are masked for logging. And it is worth recalling what has already been published about the MCP gateway: below 1.83.14 there is unauthenticated remote execution or SQL injection.

Seam 3. Identity does not reach the trace

Here lies the most expensive design flaw in the arrangement, because it is discovered at the first audit.

The user_id field of a Langfuse trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604 and :724), which comes from the user field of the OpenAI request body or from end_user_id_jwt_field (litellm_pre_call_utils.py:1584). It is not the subject of the token nor the owner of the virtual key.

The knock-on consequences:

If the client does not send user, the trace has no user.
If the client does send user, the trace has whatever value the client cares to put there, unverified.
An agent calling with a service key leaves traces with no person behind them.

There are three ways to fix it, in order of robustness:

Configure end_user_id_jwt_field so that the end user comes from a token claim and not from the body. It is the only option in which the value is not controlled by the client, and it requires JWT authentication, that is, a licence.
Force the trace_user_id metadata field, which overwrites the field (langfuse.py:726; on the OTLP route, langfuse_otel.py:96). It can be set per team.
Contractually require the client to send user and validate it in a guardrail. It works, but it is a client declaration, not a proof.

Something similar happens towards the engine. The caller’s identity does not travel to vLLM unless litellm.add_user_information_to_llm_headers is switched on, whose default value is None (litellm/__init__.py:223). With it, x-litellm-* headers are injected with user identifier, team identifier and key hash (litellm_pre_call_utils.py:1356). Forwarding client headers requires general_settings.forward_client_headers_to_llm_api, and the authorization header is always filtered out, which is correct.

Towards MCP servers, by default the server’s own credential is used, not the user’s token. Only two modes forward the caller’s token, and doing so has a name in the MCP specification: token passthrough, which is explicitly forbidden. It comes up again in the Keycloak article.

Seam 4. Langfuse as a neighbour

What can be automated and what cannot

The exact list of variables for SSO in Langfuse 4.x, taken from web/src/env.mjs:

With the Keycloak provider: AUTH_KEYCLOAK_CLIENT_ID, AUTH_KEYCLOAK_CLIENT_SECRET, AUTH_KEYCLOAK_ISSUER, AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING, AUTH_KEYCLOAK_CLIENT_AUTH_METHOD, AUTH_KEYCLOAK_CHECKS, AUTH_KEYCLOAK_ID_TOKEN_SIGNED_RESPONSE_ALG, AUTH_KEYCLOAK_SCOPE, AUTH_KEYCLOAK_ID_TOKEN, AUTH_KEYCLOAK_NAME.

With generic OIDC: the same set with the AUTH_CUSTOM_ prefix, plus AUTH_CUSTOM_FETCH_USERINFO and claim mapping with LANGFUSE_CUSTOM_SSO_SUB_CLAIM, LANGFUSE_CUSTOM_SSO_EMAIL_CLAIM, LANGFUSE_CUSTOM_SSO_NAME_CLAIM and LANGFUSE_CUSTOM_SSO_IMAGE_CLAIM.

Global ones: AUTH_DISABLE_USERNAME_PASSWORD, AUTH_DISABLE_SIGNUP, AUTH_DOMAINS_WITH_SSO_ENFORCEMENT, AUTH_SESSION_MAX_AGE.

And now what is missing. There is no variable that maps a claim to an organisation or a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID, LANGFUSE_DEFAULT_ORG_ROLE with a default value of VIEWER, LANGFUSE_DEFAULT_PROJECT_ID and LANGFUSE_DEFAULT_PROJECT_ROLE (features/auth/lib/createProjectMembershipsOnSignup.ts:61). The documentation itself confirms it: enterprise SSO does not provision roles automatically at sign-up.

On top of that, in the per-edition permissions table (features/entitlements/constants/entitlements.ts:139), the open and self-hosted professional editions do not include per-project access control or the administration API. They belong to the enterprise edition, as do audit logs and per-project retention. In the open edition, every user inherits the role of their organisation.

The declarative bootstrap, which does exist

For on-premise without a licence, the workable route is bootstrapping through variables (web/src/initialize.ts), which is idempotent: LANGFUSE_INIT_ORG_ID (mandatory, if missing the rest is ignored with a warning), LANGFUSE_INIT_ORG_NAME, LANGFUSE_INIT_PROJECT_ID, LANGFUSE_INIT_PROJECT_NAME, LANGFUSE_INIT_PROJECT_RETENTION, LANGFUSE_INIT_PROJECT_PUBLIC_KEY, LANGFUSE_INIT_PROJECT_SECRET_KEY, LANGFUSE_INIT_USER_EMAIL, LANGFUSE_INIT_USER_NAME, LANGFUSE_INIT_USER_PASSWORD.

With that, one project per tenant is created from GitOps with one initialisation container per tenant and the keys in Kubernetes secrets.

Langfuse credentials per team

This part is well resolved on the gateway side and few people use it. TeamCallbackMetadata.callback_vars (_types.py:2152) accepts langfuse_public_key, langfuse_secret_key, langfuse_host and langfuse_environment, with an allow list in initialize_dynamic_callback_params.py:67. The gateway caches one logger per credential set (langfuse_handler.py:24) and, on the OTLP route, builds a full exporter per key (langfuse_otel.py:400).

That is to say: one Langfuse project per LiteLLM team is possible today, without a licence, and it is the correct way to isolate traces between tenants. What does not exist is automatic project creation, nor the correspondence between the team and the project. That is glue you have to write.

The ingestion route

Langfuse accepts only OTLP over HTTP, in http/protobuf or http/json. gRPC is not supported. The base route is /api/public/otel and the signal lives at /api/public/otel/v1/traces, verified in the repository. Authentication is Basic with the project’s key pair, plus the x-langfuse-ingestion-version: 4 header for the version 4 route.

The gateway builds it exactly like that (langfuse_otel.py:328, :347, :359) and then normalises the endpoint by adding the signal suffix (opentelemetry.py:3237). The result matches the real route, so the integration works without touching anything. The detail matters when a collector is put in the middle and somebody copies the endpoint by hand.

And the calendar warning that already appeared in the operational pair still stands: the classic callback is tied to the v2 SDK and the date marked is 16 November 2026. With a nuance worth reading before scheduling an emergency migration, which is detailed in the data model article: that date applies to Langfuse Cloud, and in the code of the self-hosted version the old route is not switched off, it changes behaviour according to the write mode.

When to put a collector in

Direct if Langfuse is the only trace destination. Via a collector if there is more than one consumer, or if any of these three things is needed that the proxy’s batch processor does not provide:

Retries with increasing waits and an on-disk queue, with the sending_queue and file_storage processors. With that, a Langfuse maintenance window stops losing traces.
Tail sampling, which lets you keep 100 % of errors and 1 % of the rest. The proxy cannot sample by outcome, because at the point it decides it does not yet know how things ended.
Attribute redaction before they leave the namespace. It is the only layer where you can delete the MCP tool arguments which, as has already been documented, are written in the clear, bypassing the message redaction switch.

Seam 5. Postgres, Redis and the memory policy

Separate Postgres, always. Each piece applies its own Prisma migrations over the public schema of the database it is given. Two Prisma clients migrating the same schema collide over table names and over migration history. On top of that, Langfuse 4.x requires Postgres 15 as a minimum, recommends 16 and wants UTC by default. The load profile is also the opposite: the gateway writes spend records at high frequency, while Langfuse does light transactional work because trace data goes to ClickHouse.

Separate Redis, and the argument is not the one it looks like. There is no key collision by design: the gateway uses hash tags of the form {api_key:...}:requests and an optional namespace, and Langfuse uses the BullMQ prefix plus its own prefix for its key cache. The decisive argument is another one: Langfuse requires maxmemory-policy=noeviction because its queues are data, not cache, while the gateway assumes a disposable cache. Sharing an instance means that a spike of rate limit keys can fill the memory and cause write errors in the ingestion queues instead of a harmless eviction.

If it is shared anyway, two minimum measures: REDIS_KEY_PREFIX in Langfuse, namespace in the gateway, and different logical databases.

And a reminder about surface area: Langfuse 4.x does not start without ClickHouse or without an S3-compatible bucket. CLICKHOUSE_URL and LANGFUSE_S3_EVENT_UPLOAD_BUCKET are mandatory and have no default value (packages/shared/src/env.ts:120 and :271). Its failure surface is far larger than the gateway’s, and that changes where to put the high availability effort.

Ports, for the NetworkPolicy

Piece	Port
LiteLLM (API, interface and metrics)	4000
Langfuse web	3000
Langfuse worker	3030
ClickHouse	8123 HTTP, 9000 native
S3-compatible storage	9000
Redis	6379
Postgres	5432
OTel collector	4317 gRPC, 4318 HTTP

The pairs to allow under a default-deny policy, following the method from hardening the stack:

litellm → postgres:5432, redis:6379
litellm → collector:4318 (or langfuse-web:3000 if going direct)
litellm → vllm:8000
litellm → keycloak:8443 (interface and JWT flow only)
collector → langfuse-web:3000
langfuse-web → postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-worker→ postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-web → keycloak:8443
ingress → litellm:4000, langfuse-web:3000
prometheus → litellm:4000

The Langfuse worker needs no inbound traffic beyond the health probe.

The glue you have to write

Put into a table, the real state of the tenant chain:

Segment	Automatic?	How
Keycloak → LiteLLM team	Yes	`team_ids_jwt_field` plus `team_id_upsert: true`
Keycloak → LiteLLM user	Yes	`user_id_jwt_field` plus `user_id_upsert: true`
Keycloak → Langfuse organisation	No	Static assignment at sign-up only
Keycloak → Langfuse project	No	Same
LiteLLM team → Langfuse project	No	`callback_vars` by hand, or an operator of your own
Keycloak user → trace `user_id`	No	`end_user_id_jwt_field`, or a contract with the client

The minimum viable architecture without a licence, then, is this:

One Langfuse project per tenant, created with the initialisation variables from GitOps.
That project’s keys in a Kubernetes secret.
A periodic job of your own that reads the teams through the gateway’s administration API and writes callback_vars with the corresponding keys.
People entering Langfuse through Keycloak, but assigned by hand to their organisation.
The end user identifier resolved by contract with the clients, and validated.

That is about one hundred and fifty lines of your own code. It pays to budget for them from the start, because the gap does not close by itself.

Reference configuration

general_settings:
 # only with a licence; without one, virtual keys
 enable_jwt_auth: true
 litellm_jwtauth:
 team_ids_jwt_field: "groups"
 team_id_upsert: true
 user_id_jwt_field: "sub"
 user_id_upsert: true
 end_user_id_jwt_field: "preferred_username"
 user_allowed_email_domain: "ejemplo.es"
 admin_jwt_scope: "litellm_proxy_admin"
 # the three that ship switched off
 enforce_rbac: true
 enforce_scope_based_access: true
 enforce_team_based_model_access: true
 allow_requests_on_db_unavailable: false
 proxy_batch_write_at: 60

litellm_settings:
 callbacks: ["langfuse_otel", "prometheus"]
 # without this, the engine does not know who is calling
 add_user_information_to_llm_headers: true
 turn_off_message_logging: true
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

# the two that enable the validation that by default does not happen
JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma
JWT_PUBLIC_KEY_URL=https://sso.ejemplo.es/realms/plataforma/.well-known/openid-configuration

LANGFUSE_HOST=http://langfuse-web.observabilidad.svc:3000
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...

And on the Langfuse side:

AUTH_KEYCLOAK_CLIENT_ID=langfuse
AUTH_KEYCLOAK_ISSUER=https://sso.ejemplo.es/realms/plataforma
AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING=true
AUTH_DISABLE_USERNAME_PASSWORD=true
AUTH_DISABLE_SIGNUP=true
LANGFUSE_DEFAULT_ORG_ID=plataforma
LANGFUSE_DEFAULT_ORG_ROLE=VIEWER
ENCRYPTION_KEY=... # 64 hexadecimal characters

Checklist

Define JWT_AUDIENCE and JWT_ISSUER on day one, or use the per-issuer configuration. Without that, neither audience nor issuer is validated.
Set enforce_rbac, enforce_scope_based_access and enforce_team_based_model_access to true, and test that a token from another client of the same realm is rejected.
Decide where the end user of the traces comes from and write it into the design. If it is not decided, traces come out with no person.
Separate Postgres. Separate Redis. If Redis is shared, different prefix and logical database, and review the memory policy.
Check that the proxy starts with Postgres down only if that is what you want, and that the on-call team can tell an admission 503 from a credential 401.
Alert on litellm_service_latency per service, not on Redis availability, because the circuit breaker hides the problem.
One Langfuse project per tenant with the initialisation variables, and the per-team keys in callback_vars.
Put a collector in the middle if you need an on-disk queue, tail sampling or attribute redaction.
Check that the gateway version is above 1.83.14 because of the earlier critical vulnerabilities.
Write into the risk analysis that virtual keys are credentials decoupled from the identity provider, and define the offboarding procedure.

Traps and things that are not what they seem

A properly signed JWT is not a JWT addressed to this service. Without the two environment variables, the audience is not checked.

enforce_rbac set to false does not mean there is no authorisation, it means that authorisation comes only from team membership, if there is a team claim. With no team claim and without the flag, a valid token reaches /chat/completions.

The stale copy of the public keys lasts up to seventy minutes, adding the two TTLs. Revoking a key at the provider does not take effect immediately.

With a single key in the JWKS and no key identifier, the identifier is not checked.

The trace user is set by the client. Any per-person spend report built on that field is a declaration, not a measurement.

Redis going down generates no errors, it generates per-pod limits. It is the most expensive of the failures that give no warning.

Langfuse does not start without ClickHouse or without an S3 bucket. Its failure surface is bigger than the gateway’s, and that changes how the high availability effort is shared out.

Langfuse’s per-project access control is paid for in self-hosted, as are audit logs. For an ENS file, that gets decided before building, not after.

Sharing Redis does not break through key collision, it breaks through memory policy. It is a failure that shows up under load and gets diagnosed badly.

Langfuse’s ingestion route is OTLP over HTTP only. A collector configured with the gRPC exporter points at somewhere that does not exist.

Closing

The five seams share a common pattern: they fail without warning. The token from another service gets in, the trace user comes from the client, the rate limit stops being global, the spend queue discards on reaching 64 MB and the Langfuse project does not create itself. None of those five things produces a visible error.

Day 2 work, then, consists above all of turning silences into signals. Two environment variables so that the audience is validated, three flags so that authorisation is real, an alert on service latency instead of on Redis availability, and an explicit contract on where the end user’s identity comes from.

And one decision worth taking early, before the budget is closed. The gateway’s JWT authentication and Langfuse’s per-project access control are both behind a licence. A sovereign platform can live perfectly well without either, with virtual keys and one project per tenant, but it is a different architecture and it has to be drawn as such from the start.

Sources

Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Autenticación y SSO en Langfuse, consultado el 12 de septiembre de 2026.
SCIM y API de organización, consultado el 12 de septiembre de 2026.
Integración OpenTelemetry de Langfuse, consultado el 12 de septiembre de 2026.
Postgres y caché en la documentación de self-hosting, consultados el 12 de septiembre de 2026.
Autenticación por token en LiteLLM y ajustes de configuración, consultados el 12 de septiembre de 2026.
GHSA-jjhc-v7c2-5hh6 y registro de CVE-2026-35030 en NVD.

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

Sat, 12 Sep 2026 06:00:00 +0200

Seventh article in the operational track of the control layer. The fourth dealt with the coexistence of human traffic and agentic traffic on a fixed fleet. This one deals with the prior question: how big the fleet has to be, and how big the gateway in front of it. Verified against LiteLLM 1.102.0 (commit of 11 September 2026) and vLLM 0.29.0, with load data published between May and September 2026.

TL;DR

Until this year, sizing for agents meant extrapolating a chat profile and adding margin. In 2026 the first production-scale characterisations were published and the profile turned out to be something else. These are the seven conclusions.

The median prompt of an agentic call is 68,000 tokens, of which 63,000 arrive cached. The measurement covers 13.5 million sessions, 3.2 million users, 760.5 million model calls and 44.9 trillion prompt tokens in one week of June 2026. The median input-to-output ratio exceeds 275 to 1. Any calculation done on a profile of 800 prompt tokens and 250 output tokens describes a different system.

The load is prefill-dominant when counting tokens and decode-dominant when counting time. The genuinely new tokens that have to be prefilled on each turn are only between 1.5 and 7.3 times the output, not 275. With the prefix cache working, the split of execution time runs from 91 % to 98.6 % in decode. You size by memory bandwidth and by KV cache capacity, not by prefill FLOPs.

The sizing risk is not the average, it is the collapse of the prefix cache. The median hit rate within a turn is 98 %, it falls to 55 % at the boundary between turns and to 8 % when the client changes model. With dynamically retrieved context, prefix reuse drops to 1 %. One miss turns 142,000 prefix tokens into real prefill. Session affinity per replica stops being an optimisation and becomes a capacity requirement.

The intra-turn concurrency measured in production is 1.15 at the median and 1.4 at the 90th percentile. 93 % of tool batches contain a single invocation. The image of the agent opening twenty calls in parallel does not appear in the data. The real multiplier comes from the duty cycle and from the fact that 87 % of calls are started by the agent, not by the person.

The gateway is sized per worker, and each worker is a universe. One vCPU and 4 GiB per worker, with requests equal to limits. The limit counters, the caches and the cooldowns live in the worker’s memory unless there is Redis, so a limit of 100 requests per minute with four workers and three replicas allows 1,200. The Prisma pool is also per process: the number of connections open against Postgres is workers times connection_limit.

The CPU cost that grows with agentic load is the token counter. It serialises the entire tool catalogue on every request and tokenises it. The project itself measured 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens, and the operation passes through a concurrency limiter of four per worker. With agents sending long catalogues and six-figure contexts, that limit is the ceiling before the CPU is.

Autoscaling by memory does not work on this piece and the startup probe takes five minutes. The official chart deliberately leaves memory without a target because Prisma’s query engine never returns the maximum reached. That leaves CPU at 60 %, with a startupProbe of 300 seconds that makes any new replica arrive late. For agents there are two better metrics, and they are in /metrics.

You are here: the arithmetic that comes before buying

Classic capacity planning starts from a chat SLO and crosses two budgets, the VRAM one and the time one. That method is still correct. What changes with agents are the inputs: the prompt length, the number of calls per unit of human work, the split between prefill and decode, and the very definition of an SLO. This article replaces those inputs with measured figures and drags the result through to two numbers: gateway replicas and GPUs.

Out of scope is the splitting of traffic once it is built, which was already covered by the coexistence of humans and agents and prefix routing.

The analogy: the repair shop and the front desk

An appliance repair shop sizes its staff by counting faults, not phone calls. The difference matters because the person who brings in the washing machine occupies the front desk for three minutes, while the technician who takes it apart goes back to consult the machine’s data sheet fourteen times over the afternoon. If the shop manager sizes by visits to the front desk, they end up short of technicians and overstaffed at reception.

The inference platform has the same asymmetry. The person sends one turn. The agent sends fifteen calls for that turn, and each one drags along the complete file of everything done so far. The front desk is the gateway and the technicians are the GPUs, and the usual mistake is to size both with the same counter.

There is a second detail the analogy captures well. The technician does not read the whole data sheet each time: they have it open on the bench. That is the prefix cache. The day somebody moves them to a different bench between consultations, they have to fetch it from the archive again, and the work multiplies without the number of faults having increased.

Part 1. The real load profile

For two years, any capacity calculation for agents was done on assumptions. In 2026 three pieces of work appeared that measure the matter, and it is worth reading them together because they measure different scaffolds and their figures are not averageable.

The three sources

The characterisation at scale was published by a research team working on GitHub Copilot production telemetry (arXiv 2608.00101, 30 July 2026): 13.5 million sessions, 3.2 million users, 95.1 million user turns, 760.5 million model calls and 44.9 trillion prompt tokens, in one week of June 2026. It is the largest sample published to date.

The characterisation by benchmark suites comes from an architecture team (arXiv 2605.26297, 25 May 2026), which instruments five of them (ADE-Bench, DABStep, GAIA, SWE-bench Pro and Terminal-Bench 2.0) and measures the internal split of time.

The production measurement on a different scaffold was published by the vLLM project itself together with AgentX (vLLM blog, 8 September 2026).

The three disagree on turns per session: 15 calls at Copilot’s median, 43 turns in AgentX, 62.4 in SWE-bench Pro with Qwen. It is not a contradiction, they are scaffolds with different loops. The sizing lesson is that the number of calls per unit of human work depends on the client that connects, and that is why you have to measure it at home before fixing the size.

The figures you can actually use

Quantity	Measured value	Source
Median prompt per call	68K tokens, 63K of them cached	2608.00101
Input:output ratio (median)	above 275:1	2608.00101
Median session	3 user turns, 15 calls, 4.2 min	2608.00101
Mean session	6.1 turns, 40.6 calls, 62.6 min	2608.00101
Calls per turn (median)	4.5	2608.00101
Calls started by the agent	87 %	2608.00101
Range across user profiles	50× (23K to 1.1M tokens per turn)	2608.00101
Median input context	142K tokens; output 444 tokens per turn	AgentX
Accumulated context in SWE-bench Pro	mean 68.7 to 80.1K, maximum 146 to 166K	2605.26297
Reasoning tokens over the output	29.0 % to 67.6 %	2605.26297

Two operational observations come out of that and appear in no sizing guide.

The first: agents that fail accumulate up to 1.8 times more average context than those that finish well (2605.26297). The cost does not correlate with success, it correlates with difficulty and with getting stuck. A sizing done on the task that goes well falls short.

The second: the range across user profiles is 50 to 1. A user who only converses spends 23,000 tokens per turn; one who leaves the agent in a long loop spends 1.1 million. Segmenting by usage profile pays off more than any difficulty-based routing, because the variance is in the people, not in the requests.

What to measure before calculating

Three of your own figures, taken from a week of real traffic, are worth more than any table in this article:

Model calls per hour of human work, by user profile.
Prompt length distribution (median and 95th percentile), not the mean.
Aggregate prefix cache hit rate, with the time series, not the single number.

All three come from what the gateway already stores. In LiteLLM_SpendLogs you have the tokens per request and the key and team identifiers; in proxy_server_request you have the tools block of each call; and in vLLM’s /metrics you have vllm:prefix_cache_queries and vllm:prefix_cache_hits.

Part 2. Prefill, decode and the cache trap

Here is the finding that corrects almost everyone’s intuition, including mine when I started this article.

Counting tokens, agentic load is overwhelmingly input: 275 to 1. The apparently obvious conclusion would be to size by prefill capacity, that is by FLOPs. It is false.

What has to be prefilled on each turn is not the whole prompt, but what was added since the previous turn. That addition, measured, sits between 1.5 and 7.3 times the output length (2605.26297). The rest is served by the prefix cache. And the real split of the model’s execution time comes out like this: between 91.0 % and 98.6 % in decode, between 1.4 % and 9.0 % in prefill (2605.26297).

The hit rate that sustains that split is high and has been measured in three places: theoretical from 87.9 % to 99.3 % and empirical from 84.6 % to 99.5 % (2605.26297); above 96 % in production (AgentX); and a median of 98 % within a turn in Copilot (2608.00101).

Where it falls over

The same Copilot work measures the degradation, and it is the most actionable figure in the article:

98 % median hit rate within a turn. The detail matters: the first call of a turn hits around 45 %, the second rises to 86 % and from the third onwards it stabilises between 92 % and 94 %.
55 % at the boundary between turns, with idle gaps of 4.1 minutes at the median.
8 % when the client changes model.

And there is a fourth, worse regime: with dynamically retrieved context, that is when the scaffold injects search or RAG results that change on every iteration, prefix reuse falls to 1 % or less (arXiv 2608.15127, 15 August 2026).

A cache miss with a context of 142,000 tokens turns a prefill of a few thousand tokens into a six-figure one. The work multiplies by a factor of between 20 and 60 without the number of requests having changed. That is the difference between a fleet that meets its targets and one that seizes up mid-afternoon.

Two sizing rules come from that:

Without session affinity towards the same replica, the calculated sizing is worthless. LiteLLM’s concrete mechanism is session_affinity, which is not active by default and which was already covered in the prefix routing article. Without it, the calculation has to be redone assuming a prefill regime, which is between 20 and 60 times more expensive.
Changing model halfway through a session costs the entire cache. Any difficulty-based routing policy that moves a live conversation from one model group to another pays an 8 % hit rate on the next call. If you want to route by difficulty, do it at the start of the session.

Co-batching, which is the other effect nobody measures

Mixing a short request into the same batch as long-context requests degrades the short one’s inter-token latency. The measured figure: one long request in the batch raises time per token by 38.3 %; two raise it by 79.7 % (2608.15127). It is the technical reason why separating human and agent pools improves the human experience more than the capacity split suggests, and it is also the reason not to raise --max-num-seqs above what the KV budget allows.

Part 3. The KV budget, with the numbers of the agentic case

The formula does not change with respect to classic capacity planning, but the inputs do, and the result is surprising:

$$\text{KV}_{\text{bytes/token}} = 2 \times n_{\text{layers}} \times n_{\text{kv heads}} \times d_{\text{head}} \times \text{bytes}_{\text{elem}}$$

For a 70B-class geometry (80 layers, 8 KV heads with grouped attention, head dimension 128, 2-byte elements) that gives 0.3125 MiB per token. With the median agentic context of 142,000 tokens:

$$142{,}000 \times 0.3125\ \text{MiB} \approx 43.3\ \text{GiB per sequence}$$

For a 30B-class geometry (48 layers, everything else the same) that gives 0.1875 MiB per token, that is around 26.0 GiB per sequence.

It is worth pausing on what that means. An 80 GB H100 with the weights of a 70B model in FP8 already taking up space does not have 43 GiB free for a single sequence. The conclusion is harsh and has to be written down: serving real 70B-class agentic context demands tensor parallelism of at least 2 and a quantised KV cache. And the number of simultaneous sequences that fit, which is what sets --max-num-seqs, comes from dividing the available KV by that figure, not from a chat heuristic.

$$\text{sequences} \approx \frac{\text{VRAM}_{\text{KV available}}}{\text{KV}_{\text{bytes/token}} \times \text{max context}}$$

With FP8 in the KV, the divisor halves. It is the cheapest capacity lever there is in this load profile, and its cost in quality is validated with evals, not assumed.

Part 4. From people to concurrent requests

This is where most calculations overshoot, because it is assumed that an agent equals many simultaneous requests. The data says otherwise.

The concurrency of calls within a turn, measured in production, is 1.15 at the median and 1.4 at the 90th percentile, and 93 % of tool batches contain a single invocation (2608.00101). Intra-turn parallelism is small. Subagents appear in 44 % of AgentX sessions, with a median of 4 deployments among those that use them, so there is a long tail, but it is not the usual regime.

What does multiply is the duty cycle. The mean session lasts 62.6 minutes and contains 40.6 calls. At 444 output tokens and an interactivity of around 60 tokens per second, each call occupies around 7.4 seconds of decode, which gives between 5 and 7 active minutes out of those 62.6.

$$C = D \times \rho \times A$$

where $D$ is the people working, $\rho$ the duty cycle and $A$ the amplification. With $\rho \approx 0.10$ to $0.12$ derived from the previous figures, twenty developers give between 2 and 2.4 concurrent sequences in the steady state. It is the peak, not the average, that has to be sized for.

The amplification

Three factors published separately:

Tool failures: 9 % of turns trigger them, with a compute amplification of 4 times (2608.00101).
Subagents: 44 % of sessions, median of 4 deployments (AgentX).
Internal variance: the same task from the same agent varies by up to 30 times in cost within a single trace (2608.15127).

The product of the first two gives $(1 + 0.09 \times 3) \times (1 + 0.44 \times 3) \approx 2.9$. Nobody publishes that product as a single figure, so it is flagged as my own derivation, but an amplification factor of 3 is the reasonable design provision and it fits the measured variance.

For the twenty developers in the example: between 6 and 7 concurrent design sequences, with 142K contexts. At 43.3 GiB of KV per sequence in the 70B class, that is around 300 GiB of KV cache, that is four H100s dedicated to KV alone with the weights on top. With KV in FP8, half. With a 30B-class model, a third.

That calculation, done with the chat profile of 800 prompt tokens, would have given a single GPU. That is the distance between the two regimes.

Part 5. The gateway, which also has to be sized

The proxy consumes no GPU, but it has its own arithmetic and it is the part that produces the most surprises in production, because almost all of its state is per process.

A worker is a universe

The default value of --num_workers is 1 (litellm/constants.py:57). The usual startup is uvicorn directly with uvloop (proxy_cli.py:1477), and there are alternatives with gunicorn, hypercorn and granian.

What lives in each worker’s memory, and therefore multiplies:

The in-memory cache, limited to 200 elements with a TTL of 600 seconds (litellm/caching/in_memory_cache.py:27). With many deployments, the cooldowns and the counters compete for that same space and get evicted before they expire.
The key authentication cache, with a TTL of 60 seconds (proxy_server.py:1650), adjustable with general_settings.user_api_key_cache_ttl.
The limit and budget counters when there is no Redis (parallel_request_limiter_v3.py, local_only=True paths).
The router cooldowns, with DEFAULT_COOLDOWN_TIME_SECONDS = 5 (constants.py:75).
A periodic task scheduler of its own per worker (proxy_server.py:9674), coordinated between pods with a lock in Redis.

The practical consequence: a limit of 100 requests per minute with four workers and three replicas allows up to 1,200 without Redis. With Redis, the v3 limiter registers seven Lua scripts at startup (parallel_request_limiter_v3.py:608) and the limits become global.

One detail of the v3 limiter that matters with agentic load: LITELLM_TPM_TOKEN_RESERVATION_ENABLED comes enabled by default and reserves estimated tokens before the call, which adds a round trip to Redis per request. And there is an asyncio.Lock per instance that serialises the atomic batch checks (:665), documented by the code itself as a 99th-percentile risk under contention.

If Redis goes down, requests do not fail: there is a circuit breaker with a threshold of 5 failures and a recovery of 60 seconds (constants.py:456), and the proxy degrades to local memory. What is lost silently is the globality of the limits.

Postgres

The pool is also per process. The default values are database_connection_pool_limit = 10 and database_connection_pool_timeout = 60 (proxy_cli.py:62), injected as parameters of the Prisma URL. A pod with four workers opens forty connections. Three pods, one hundred and twenty. It is the fast route to exhausting Postgres’s max_connections without having served any traffic, and the reason the official chart offers PgBouncer inside the pod itself with maxDbConnections: 20 and maxClientConn: 1000.

On the request path there is only the authentication, which is cached for 60 seconds, and the rate limiting. Spend is deferred. The SpendLogs queue is capped at 64 MB and on overflow it discards the oldest elements with an error in the log (utils.py:6704). The write interval is proxy_batch_write_at, with a default value of 10 seconds plus a random spread of up to 5 (constants.py:1719, proxy_server.py:9704); the production guide recommends raising it to 60.

With a slow Postgres, the queue grows to those 64 MB and then loses billing data without returning any error to the client. It is the silent failure mode you have to alert on.

The token counter, which is the agentic trap

This is the piece that turns a correct sizing into an insufficient one, and it appears in no guide.

Before admitting a request, the proxy counts tokens. With chat load the cost is negligible. With agentic load it is not, for two reasons that add up.

The first is the prompt length. The project itself measured the admission counting cost at 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens (benchmarks, consulted on 12 September 2026), and published that with a counter written in Rust those figures fall to 4.9, 6.8 and 10.2 ms.

The second is the tool catalogue. The function that prepares the definitions serialises the entire catalogue to a pseudo-TypeScript and tokenises it (litellm_core_utils/token_counter.py:934), plus nine fixed tokens. The cost is linear in the size of the catalogue and it is paid on every request. That is exactly the pattern of an agent connected to an MCP gateway: a long catalogue, resent on every turn. The token bill of that catalogue was already covered in the MCP gateway article; here its other face appears, which is proxy CPU.

And on top of that there is an explicit ceiling. The counting runs in a thread with a capacity limiter whose default value is 4 per worker (token_counter.py:340, constants.py:407). With 100K prompts, four concurrent counts of 100 ms mean a theoretical ceiling of around 40 admissions per second per worker before the limiter starts queueing, and that is without having called the model yet.

What is cached is the tokeniser, with an LRU of 64 entries (utils.py:2206), not the result of the count.

Three levers, in order of return: raise TOKEN_COUNTER_MAX_CONCURRENT_COUNTS if CPU is spare, reduce the tool catalogue exposed per key, and watch the gateway overhead figure in /metrics to know whether this is happening.

Streaming and admission

There is no connection limit by default. --limit_concurrency exists but is optional, and it returns 503. The built-in admission control is only activated if general_settings.max_in_flight_requests_per_worker is defined (middleware/admission_control_middleware.py:285), with max_queued_requests_per_worker and admission_queue_timeout_seconds at 1.0 by default, rejecting with 503 and a retry-after: 1 header.

For agents, that header matters: an agentic client that retries without respecting it multiplies the load exactly when the gateway is saying it cannot take any more. The combination of gateway retries was already calculated by the fourth article and it reached 45 calls per turn in the worst case.

The default request timeout is 6,000 seconds (constants.py:512). The production guide recommends 600. With long agentic tasks, leaving it at 6,000 means a hung connection occupies a slot for an hour and a half.

Resources and autoscaling

The official chart recommends 1 vCPU and 4 GiB per worker, with requests equal to limits (helm/litellm-helm/values.yaml:235), and the production documentation suggests one worker per pod scaling horizontally. The project’s own high-performance benchmark, by contrast, uses four workers per pod with PgBouncer in front.

The HPA comes with targetCPUUtilizationPercentage: 60. And there is a comment in the chart worth reading twice: memory is deliberately left without a target because the resident consumption of Prisma’s query engine is a high-water mark that is never returned, so that an HPA on memory would add replicas and never remove them (values.yaml:260).

The other problem is startup. The startupProbe has failureThreshold: 30 with a period of 10 seconds, that is up to 300 seconds before the pod is considered ready (values.yaml:109). An HPA that reacts at 60 % CPU and takes five minutes to add capacity arrives late to any agentic burst.

The alternative, optional in the chart, is to scale by Prometheus metrics: targetRequestsPerSecond and targetTokensPerSecond over rate(litellm_proxy_total_requests_metric_total[1m]) and rate(litellm_total_tokens_metric_total[1m]). The chart itself warns that the token metric lags with long streaming responses, because tokens are accounted for on completion. For agents, that is almost always.

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

Metric	What for
`litellm_in_flight_requests`	Real saturation per worker. It is the scaling metric
`litellm_overhead_latency_metric`	Proxy cost per request; it rises with token counting
`litellm_overhead_with_guardrails_latency_metric`	The same including guardrails
`litellm_request_queue_time_seconds`	Arrival through to pre-call, includes authentication and ASGI queueing
`litellm_deployment_cooled_down`	Replicas out of the pool
`litellm_service_latency`	Redis and database latency, per service

The /health/backlog endpoint returns in_flight_requests, admitted_requests, queued_requests and rejected_requests (health_endpoints/_health_endpoints.py:1871).

What it does not expose: the depth of the SpendLogs queue or its bytes, the Prisma connections in use against the limit, nor the depth of the logging worker’s queue. All three are real failure modes and have to be instrumented from outside.

At the engine

Names verified in the vLLM 0.29.0 metrics documentation:

vllm:num_requests_running, vllm:num_requests_waiting
vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:inter_token_latency_seconds
vllm:prefix_cache_queries, vllm:prefix_cache_hits, vllm:kv_cache_usage_perc

And the ones that break dashboards on upgrade:

vllm:time_in_queue_requests is duplicated by vllm:request_queue_time_seconds.
The direct prefix cache hit rate indicator disappeared. It is now computed as rate(vllm:prefix_cache_hits) / rate(vllm:prefix_cache_queries).
vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are deprecated along with CPU swapping.
vllm:gpu_cache_usage_perc became vllm:kv_cache_usage_perc, which also breaks any HPA that uses it.

The SLO worth chasing

A 95th percentile of time to first token describes nobody’s experience when the unit of work is 43 turns. Coding tasks frequently exceed ten minutes and research tasks reach hours (2608.15127).

Two indicators serve better:

90th percentile of full-task latency, measured per session, not per request. It comes from grouping LiteLLM_SpendLogs by session identifier.
90th percentile of interactivity, in tokens per second per user. It is the one AgentX uses, with values between 58.3 and 74.2 in its runs.

To autoscale the engine, the official production-stack example uses vllm:num_requests_waiting with a threshold of 5, polling every 15 seconds and a cooldown of 360. With agentic load it is worth adding vllm:kv_cache_usage_perc as a second trigger, because with 26 to 43 GiB of KV per sequence the memory pressure arrives before the queue does.

Part 7. The engine levers, ordered by return

Verified against vLLM 0.29.0, released on 9 September 2026.

--long-prefill-token-threshold 512. The AgentX measurement attributes to it +93 % in tokens per GPU per second, with a 2.3-times improvement at the 90th percentile, by attacking the head-of-line blocking that a long prefill produces in front of short requests. It is the highest-return lever documented for this profile.

KV cache in FP8. It halves the memory budget per sequence, which is the limiting resource. Validate with evals.

--max-num-queued-reqs and --max-num-queued-tokens. New in 0.29. They are admission control in the engine, and they are the right mechanism to protect the SLO instead of letting the queue grow without limit.

max_num_batched_tokens downwards. High values improve time to first token; low values improve inter-token latency. In a regime where 91 % of the time is decode, inter-token latency is what is perceived. Chunked prefill is enabled by default whenever possible in the V1 engine.

Session affinity at the gateway. It is not an engine setting, but it is what holds up everything above. Without it, that 98 % hit rate turns into random distribution across replicas.

Two version warnings: Model Runner V2 becomes the default in 0.29 and V1 is deprecated with removal planned for 0.32; and starting with python -m vllm.entrypoints.openai.api_server is deprecated.

The full calculation, from start to finish

Case: twenty developers, a coding scaffold, a 30B-class model in the fleet, maximum context of 128K, KV in FP8.

1. Steady-state concurrency. $C = 20 \times 0.11 \times 3 \approx 6.6$ sequences.

2. KV budget per sequence. 0.1875 MiB per token in BF16, half in FP8, that is around 0.094 MiB. At 128K of context: around 12.0 GiB per sequence. For 6.6 sequences with margin, rounding to 8: 96 GiB of KV.

3. GPUs. The weights of a 30B in FP8 take up around 30 GB. On an 80 GB H100 with gpu_memory_utilization of 0.90 there are around 72 GB usable per card. With tensor parallelism of 2, the weights are split: around 15 GB per GPU, so around 57 GB of KV per GPU and around 114 GB in the replica. That covers the 96 GiB calculated, and TP=2 also adds memory bandwidth, which is the resource that dominates decode. Two H100s, one replica, TP=2. It is worth seeing the contrast: on a single H100 the full weights leave around 42 GB of KV, that is three and a half sequences, less than half of what is needed. With a second replica for availability and maintenance windows: four GPUs.

4. Requests per second at the gateway. 20 people, a mean session of 40.6 calls in 62.6 minutes, duty cycle included: of the order of 0.2 to 0.3 requests per second on average, with peaks of 2 to 3.

5. Gateway workers. The ceiling is not set by throughput but by token counting: four concurrent counts per worker, at 100 ms with long prompts. With peaks of 3 requests per second and six-figure prompts, two replicas of one worker each cover the case with margin for the rolling update, at 1 vCPU and 4 GiB each. Postgres will see 20 connections.

6. What needs switching on. Session affinity, admission control at the gateway and at the engine, --long-prefill-token-threshold, and Redis so that the limits are global.

The same calculation with the chat profile would have given one GPU and one worker. The distance is not in the safety margin, it is in the inputs.

A reference `config.yaml`

model_list:
 - model_name: agentes-30b
 litellm_params:
 model: hosted_vllm/qwen3-coder-30b
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 rpm: 600
 model_info:
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072

router_settings:
 routing_strategy: simple-shuffle
 # without this, the capacity calculation in this article is worthless
 optional_pre_call_checks: ["session_affinity"]
 num_retries: 2
 allowed_fails: 3
 cooldown_time: 30

general_settings:
 # admission control: below this the queue grows without limit
 max_in_flight_requests_per_worker: 24
 max_queued_requests_per_worker: 48
 admission_queue_timeout_seconds: 1.0
 # the spend queue discards at 64 MB; shorter interval, less backlog
 proxy_batch_write_at: 60
 enable_drain_endpoint: true
 drain_endpoint_token: os.environ/DRAIN_TOKEN
 database_connection_pool_limit: 10

litellm_settings:
 # 6000 s by default: a hung connection holds a slot for an hour and a half
 request_timeout: 600
 callbacks: ["langfuse_otel", "prometheus"]
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

Environment variables that go with it:

NUM_WORKERS=1
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS=8 # default 4; raise only if CPU is spare
LITELLM_TPM_TOKEN_RESERVATION_ENABLED=true
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS=30

And on the engine side:

vllm serve Qwen/Qwen3-Coder-30B \
 --tensor-parallel-size 2 \
 --max-model-len 131072 \
 --kv-cache-dtype fp8 \
 --long-prefill-token-threshold 512 \
 --max-num-queued-reqs 64 \
 --gpu-memory-utilization 0.90

Checklist

Measure three figures of your own over a week: calls per hour of human work and per profile, prompt length distribution, and the time series of prefix cache hit rate.
Calculate the KV per sequence with the model’s real geometry, taken from its config.json, not from a table.
Set --max-num-seqs from that budget, not from a chat heuristic.
Switch on session affinity before accepting the calculated number of GPUs.
Switch on admission control in both layers and check that the client respects retry-after.
Lower request_timeout from 6,000 to 600.
Raise proxy_batch_write_at to 60 and alert on the growth of the spend queue from outside, because there is no metric.
Leave the HPA on CPU at 60 %, never on memory, and assume the 300 seconds of the startupProbe when planning the margin.
Watch litellm_in_flight_requests and litellm_overhead_latency_metric; if the overhead rises with the size of the tool catalogue, the culprit is the token counter.
Go over the dashboards for the vLLM metric renames before upgrading.
Check that no routing policy moves a live session between model groups.
Redo the calculation when a new user profile arrives: the range across profiles is 50 to 1.

Traps and things that are not what they seem

The ratio of 275 to 1 does not mean the system is prefill-bound. The new tokens per turn are between 1.5 and 7.3 times the output. The time is between 91 % and 98.6 % in decode.

The aggregate prefix cache hit rate hides the problem. An aggregate 90 % may be 98 % intra-turn and 55 % at boundaries. You have to look at the series, not the number, and segment by request type.

vLLM’s hit rate counts the reprefill after an eviction as a hit. The counter exists internally and is not exposed, so the published number is optimistic under a memory-pressure regime.

Agents that fail cost more than those that succeed, up to 1.8 times more context. Sizing on the task that goes well leaves the calculation short.

The database pool is per process, not per pod. Four workers are forty connections.

The HPA on memory does not work with this piece, and not because of a configuration mistake: Prisma’s query engine does not return the memory.

The 300-second startupProbe turns any autoscaling into late reaction. For agentic peaks, it is better to over-provision than to trust the scaling.

Raising --max-num-seqs to “make use of the GPU” degrades perceived latency: two long requests in the batch raise time per token by 79.7 %.

The token counter has a limiter of four per worker and it appears in no sizing guide. With large tool catalogues and long contexts it is the first ceiling you hit.

The figures in this article come from three different scaffolds and are not averageable. Fifteen, forty-three and sixty-two turns per session are three systems, not three measurements of the same one.

Closing

The arithmetic of sizing has not changed. What has changed are the inputs, and now there are public measurements at scale instead of intuitions. Three figures sum up the change of regime: 68,000 tokens of median prompt, 87 % of calls started by the agent rather than by the person, and a time split that is decode for more than 90 % as long as the prefix cache hits.

From that comes an order of work that is not the usual one. First, session affinity, because without it the rest of the calculation describes a different system. Second, the KV budget with the model’s real geometry, which is what sets how many sequences fit. Third, the gateway, where the ceiling is not throughput but a token counter with four threads per worker.

And a warning about your own figures. The measured range across user profiles is 50 to 1. Any platform that does not segment its measurement by profile is averaging two different populations and producing a number that describes neither of them.

Sources

Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale, arXiv 2608.00101, 30 de julio de 2026.
Agentic AI Workload Characteristics, arXiv 2605.26297, 25 de mayo de 2026.
From LLM Inference to Agentic Workloads, arXiv 2608.15127, 15 de agosto de 2026.
vLLM x AgentX: serving agentic workloads, 8 de septiembre de 2026.
vLLM metrics design y optimization guide, consultados el 12 de septiembre de 2026.
LiteLLM benchmarks y production setup, consultados el 12 de septiembre de 2026.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Autoescalado con KEDA en vLLM production-stack, consultado el 12 de septiembre de 2026.

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

Fri, 11 Sep 2026 01:45:00 +0200

Sixth post in the operational track on the control layer. The previous five treat the gateway as the piece the tokens pass through: the pair with Langfuse, day 2, virtual keys, humans and agents and prefix routing. This one is about the other door. Verified against LiteLLM 1.102.0, commit of 10 September 2026.

TL;DR

The MCP gateway is not a module, it is a complete second surface. A native MCP endpoint with four route spellings, a proxy mode with three fixed tools, four REST routes, some thirty-five administration routes and a long set of OAuth discovery endpoints. It still lives under _experimental and it is nearly 20,000 lines.

I correct what I said in the previous post: access is NOT open by default. In 1.102.0, a key with no declared permissions and no team resolves zero servers. What does open things up is a per-server flag, allow_all_keys, and that flag overrides the key’s explicit scope unless you enable a setting that is off by default.

Permissions are enforced on execution, not just on the listing. Calling a tool that did not appear in tools/list returns 403. There are three chained checks before the call, including one on the allowed arguments. The same predicate governs listing and calling, by declared design.

The hierarchy is built out of intersections, with two escapes and a fail-open. The key’s access groups are additive on top of the key and team ceiling. The organisation ceiling replaces rather than intersects when there is nothing below it. And on an indeterminate error while resolving tool permissions, the code returns “unrestricted”. It is a deliberate decision and it has to be declared in the risk analysis.

Per-tool cost is worth zero and that is why its metric disappears. There are exactly two Prometheus metrics for MCP, and the spend one is only incremented if the cost is greater than zero. With the default configuration, that series never exists. There is no listing metric, no latency metric, no error metric, no server health metric.

The cost that does matter is not attributed to MCP anywhere. It is the tokens of the tool definitions, resent on every turn inside the prompt_tokens of the model call. Five typical MCP servers add up to 58 tools and some 55,000 tokens. The tool’s spend row, with its zero cost, has nothing to do with that.

And the gateway’s semantic filter does not act on the MCP door. It only applies when it is LiteLLM calling the model. A client connected to /mcp receives the whole catalogue, filter enabled or not. And even if it did act, filtering per request rewrites the cached prefix of the prompt, which is the opposite of what Anthropic and OpenAI do with their own tool searches.

The good observability ships switched off. Without LITELLM_OTEL_V2, a tool call comes out as a generic span with the model set to MCP: name. With it enabled there is a dedicated span and correct attributes, and the arguments and the result stay hidden unless you opt in. And in neither case is trace context propagated towards the upstream MCP server: the trace is cut at the gateway.

Tool arguments are written, and they are written in the clear. They go in full into the metadata column of the spend table, without passing through the switch that governs prompt storage nor through the redaction function, which only knows about messages and response. Whoever enabled redaction for GDPR probably believes they are covered.

The 1.83 branch concentrated four serious vulnerabilities, and one of them belongs to the MCP gateway itself. The endpoint for testing a connection before saving accepted the stdio transport’s command, args and environment in the request body. A low-privilege key got command execution on the proxy host.

And the specification moved underneath. The revision in force since July 2026 removed protocol sessions, the initialisation handshake and the session header. LiteLLM 1.102.0 advertises the June 2025 revision and its version enumeration does not even contemplate the two later ones.

You are here: the door nobody chose

The gateway was chosen for the reasons in the switchboard post: licence, fit inside the cluster, maturity. What was evaluated was the inference traffic.

Later, somebody connects an agent, the agent needs tools, and the same process that is already authenticated and deployed turns out to speak MCP too. The decision to turn it into the tool door is almost never taken: it is inherited. And that door has neither the same controls, nor the same observability, nor the same cost model.

When MCP grows covered the identity of your own MCP servers. MCP from the inside covered the protocol and its instrumentation. This post covers the piece in between.

The analogy: the switchboard that also hands out keys

An office building’s telephone switchboard. Its original job is to route calls: it decides who can call outside, how much each department spends, and it leaves a record of every conference call.

Over time, the switchboard operator is also given the key cabinet. They are already at reception, they already know who everybody is, it looks like the natural place. And from that moment there are two services at the same desk with very different rules.

Calls are priced by the minute and show up on the bill. Keys have no tariff, so on the department’s bill they come out at zero, and somebody might conclude that the key cabinet costs nothing. It does cost: it costs the operator’s time, it costs having to read the whole key catalogue every time somebody asks what is available, and it costs that nobody checks that the label on a key still says what it said the day its loan was authorised.

The analogy ends at the interesting spot. When somebody takes a key, the record says which key it was and what they said they wanted it for. That record is simultaneously the evidence of the action and a piece of data that perhaps should not be stored in the clear.

The real routes

The first thing that surprises you when you look at the code is how many spellings get you to the same place.

The native endpoint

The MCP sub-application is mounted at /mcp and inside it has four mounts: the root as a wildcard, /mcp, /{name}/mcp and /sse.

And there is also a hand-declared route for bare /mcp on the main application, with a comment that explains why: the mount cannot match its own bare prefix, and the resulting 307 redirect breaks MCP clients sitting behind a proxy that terminates TLS. It is the kind of detail you discover in production on a Friday.

/mcp/sse holds a surprise. It does not use the classic SSE transport: it uses the same HTTP session manager in stateless mode. The SseServerTransport object is constructed and is not referenced on any other line of the file. It is dead code, and the /mcp/sse/messages endpoint that this object advertises does not exist as a working route.

The /{name}/mcp route resolves in a documented order: server alias, comma-separated list with a maximum of 16 elements, tool set, access group. If nothing matches, 404.

And /mcp/proxy exposes a fixed surface of three tools. It is the progressive disclosure pattern, and it will come up again further down because it is the most important mitigation in this post.

The REST routes

Four, under the /mcp-rest prefix:

Method	Route	Authentication
GET	`/mcp-rest/tools/list`	Virtual key
POST	`/mcp-rest/tools/call`	Virtual key
POST	`/mcp-rest/test/connection`	Key plus administrator role
POST	`/mcp-rest/test/tools/list`	Key plus administrator role

The variants without -rest that appear in the internal types have no router registering them. The /mcp/* wildcard absorbs them as JSON-RPC.

The last two are the stars of the security section.

Administration and discovery

Some thirty-five routes under /v1/mcp, almost all of them with virtual key authentication. Three details worth knowing:

GET /v1/mcp/registry.json carries no authentication dependency. It returns 404 if the public registry is not enabled, which is the real protection.

POST /v1/mcp/server requires the proxy administrator role. POST /v1/mcp/server/register does not: it is the proposal path, which requires a key with a team, explicitly forbids the stdio transport and leaves the entry pending review.

And on top of that there is a long set of OAuth .well-known endpoints, with dynamic client registration, authorisation, token, revocation and introspection, plus their per-server variants.

Configuration and transport

The mcp_servers schema in config.yaml has fifty-odd fields, all of them optional. A warning for anyone coming from earlier versions: spec_version no longer exists in 1.102.0.

The default transport when omitted is http, that is, streamable HTTP. The other two values are sse and stdio. The string streamable_http is not valid in the configuration file, although it is accepted as a synonym when importing connector files.

Tool prefixing uses the format {server}{separator}{tool} with - as the default separator, configurable through an environment variable read at import time. The prefix chosen is the alias, or the name, or the identifier, in that order. There is an optional short mode that derives three characters from the hash of the server identifier and resolves collisions by rehashing.

Access control, and the correction to the previous post

In the humans and agents post I closed the list of traps with this one: “access to MCP servers is open by default if no level defines a list”. Verified in 1.102.0, that is false.

The function that resolves the allowed servers does the opposite. If the team declares nothing, it inherits from the key; if the key declares nothing either, the base set is empty and the scope is marked as unbounded with zero servers. There is no branch that expands to “the whole registry” other than the administration role one.

What does open things up, and this is the important part, is something else.

`allow_all_keys`

It is a per-server flag, in configuration, in the API and in the database, defaulting to false. When set to true, that server joins the set of open servers and any key can see it.

The detail to know: by default it overrides the key’s explicit scope. A key that declares access only to server A will also see server B if B has the flag. The key’s scope is only respected if you enable general_settings.mcp_allow_all_keys_respects_mcp_scope, which is off by default.

There is a second opening case: anonymous subjects, with no user identifier, role or key, additionally receive the servers configured with authentication delegation towards the upstream and the pass-through ones. It is consistent with the design of those two modes, and it is a surface you have to know about.

`require_key_mcp_access_defined`

It lives in general_settings, defaults to false, and needs no licence. What it inverts is only the empty-key inheritance towards the team: with the flag on, a key with no list of its own gets an empty set instead of inheriting the team’s.

It does not affect keys that do declare servers, which still intersect with the team. And it does not close off the access group grants, which are additive and are the escape route you have to audit separately.

There is a sibling, require_end_user_mcp_access_defined, with the same shape and applied to the end user.

The hierarchy

The function’s docstring is normative and says that all the rules are intersections. In detail:

Key and team: intersection when both declare. Inheritance in one direction when one is empty.
Key access groups: additive union on top of the previous result.
End user: intersection if it declares anything.
Agent: intersection.
Internal user: ceiling that only narrows.
Organisation: ceiling that intersects if there are restrictions below, and that replaces by becoming the ceiling when there are none.

For subjects with no key, admitted by SSO or by the gateway, the model changes to a per-source union instead of an intersection.

The fail-open

This is the point you have to take to the risk analysis and not leave in a technical note.

On an indeterminate failure while resolving the tool permissions of a key or a JWT, the code returns “unrestricted”, that is, all tools allowed. The organisation ceiling is skipped in the same situation. For subjects with no key the behaviour is the opposite, fail-closed.

There is a well-resolved intermediate case: when a permission is named but cannot be read, a specific error is raised and it is denied in both cases.

The doctrine is documented in the class docstring, which is appreciated. And it remains a behaviour that, in a system under the ENS, you have to declare in writing instead of discovering it during an audit.

What is done well

Permissions are enforced on the call, not just on the listing. The execution path chains three checks before contacting the server: allowed or forbidden tools, per key and team tool permission, and validation of the arguments against the allowed ones. All three return 403.

Listing and calling share the same predicate, and the docstring declares it as an explicit invariant. A client that calls a tool directly when it did not appear in its listing receives 403, it does not execute. The same over REST and over the responses API route.

The x-mcp-servers and x-mcp-access-groups headers only narrow, and they fail closed if they do not resolve.

Header forwarding towards the upstream is an explicit per-server allowlist, with a centralised decision about when to strip the caller’s Authorization, and a protection against cross-forwarding: in a multi-server listing, the global header is withheld if more than one server would consume it. That is RFC 9700 applied.

The bill that never shows up

Here is, in my opinion, the most important thing in this post, and it has nothing to do with security.

Every tool the gateway aggregates takes up space in the model’s context. Name, description and parameter schema travel in the request’s tools block, on every turn, and they are paid for as input tokens of the real model.

The figures published by Anthropic in November 2025 are the best public reference there is. Five connected MCP servers add up to 58 tools and some 55,000 tokens. The breakdown: GitHub, 35 tools and some 26,000 tokens. Slack, 11 tools and some 21,000. Sentry, five and some 3,000. Grafana, five and some 3,000. Splunk, two and some 2,000.

With tool search enabled, that set drops from some 72,000 to some 8,700 tokens, and the usable context goes from 122,800 to 191,300.

The second cost of a large catalogue is not economic and is not billed: it is that the model chooses worse. It has its own section further down, because of the two it is the one that decides whether the agent is any good.

How to measure it with what is already there

The gateway stores enough data, even if it does not cross-reference it for you.

In the listing’s spend row, with call type list_mcp_tools, it enriches the metadata with allowed_server_count, tool_count_total, per_server_tool_counts and per_server_list_outcomes.

In the model call rows, the proxy_server_request column keeps the request body with the tools block inside it.

The practical method: pair the listing rows with the LLM rows of the same session by session_id, read the total number of tools from the first, and pass the tools array of the second through litellm.token_counter(model=..., tools=...), which accepts that argument. Multiply by the number of turns, because the block is resent on all of them.

For the relative cost, a quicker comparison: mean prompt_tokens of keys with MCP enabled against those without, same model and same period.

Tool selection

The previous section treats the catalogue as a bill. This one treats it as what determines whether the agent gets it right. This is ground where the talk runs well ahead of the measurement, so it is worth separating what has been measured from what gets repeated.

How many tools a model can take

Anthropic’s official figure is that Claude’s ability to pick the right tool degrades when going from 30 to 50 available tools. The same documentation gives the inverse criterion, which is the more useful one: the full catalogue with nothing else is the right option with fewer than ten tools, when all of them are used on every request, or when the definitions add up to fewer than a hundred tokens.

There is also an administrative ceiling before the semantic one. VS Code with Copilot cuts off at 128 tools per request and returns an error when you exceed it.

The full curve gets cited far more than it has been measured. The soundest primary data I found is from May 2025: varying the number of candidates from 1 to 11,100 over a registry of more than 4,400 servers, below some 30 tools the success rate is above 90 %, between 31 and 70 intermittent failures appear, and past the hundred mark the degradation becomes dominated by retrieval. It is a qualitative heat map, not a numeric table.

The figure that circulates most (78 % with ten tools, 40 % with a hundred, 13.6 % with seven hundred) comes from a conference talk with no paper, no repository and no published methodology. Do not take it to a client presentation.

And there is a warning that affects everything else. An audit of 496 tasks from four tool calling benchmarks found 18.5 % misalignment between the label and reality. In the BFCL v4 subset, 80 % of the failures assigned to the agent came from fragile state comparisons. In one of the benchmarks with an automatic judge, 23 identical re-runs gave a range from 57.9 % to 76.8 %, almost 19 points of spread, enough to reorder the ranking. Any absolute figure in this section deserves a generous margin of error.

How it fails

The work with the cleanest taxonomy evaluates 36 servers, 220 tools and 1,000 tasks against twenty frontier models, with an average of 15.2 tools exposed per task of which 4.1 are relevant.

Failure mode	Share
Not using a tool when one was needed	10.5 %
Choosing the wrong tool	9.0 %
Malformed parameters	6.9 %
Not recovering from an error	10.3 %
Cognitive failures, not tool failures	63.3 %

The interesting part is in the per-model breakdown: the dominant failure mode changes with the model, not with the size of the catalogue. In one of them, 40.1 % of the failures are simply not calling any tool. In others the malformed parameter dominates. In another, the wrong tool. There is no canonical failure to attack.

On tools that do the same thing, which is the real case of a gateway with fifteen servers, there is a good measurement. A study with ten groups of five functionally equivalent tools and a thousand query pairs puts the selection bias between 0.3 and 0.4, that is, you would have to redistribute between 30 and 40 % of the probability mass for equivalent tools to be chosen equally. The same work shows that a biased continued training takes the selection of a specific endpoint from 0.6 % to 12.8 %.

That study contributes another figure that reorders priorities: shuffling a tool’s description moves the selection distribution substantially, whereas changing only the name has minimal and inconsistent effects. Applied to the gateway’s server-tool prefixing: it costs tokens and probably does not change which tool the model goes to. There is no study isolating the effect of name spacing, and it is a real gap in the literature.

If the description is what carries weight, the state of the estate is the problem. An audit of 856 tools from 103 MCP servers found undeclared limitations in 89.8 %, missing usage guidance in 89.3 % and opaque parameters in 84.3 %. Only 2.9 % were free of problems across all their components.

The optimum is not a number

The May 2026 work I cited above proposes a chance-corrected metric and uses it to size the list adaptively. Over 370 functions it reaches 90.3 % coverage while seeing an average of 7.4 tools, against 90.8 % seeing a fixed 50.

But the same paper publishes its own counterexample, and it is honest to cite it: over another benchmark of 3,251 tools, the fixed K of 5 wins in aggregate, 64.7 % against 61.9 %. The adaptive advantage concentrates in the hard queries, where the right tool sits between sixth and twentieth by similarity: there it finds 16.7 % of the cases where the fixed K finds 0 %.

The March 2026 work over 121 tools puts the plateau at K=3, with 97.1 % that does not improve going up to K=5. With the caveat that it measures retrieval against labels, not model accuracy.

And a July 2026 work gives the formulation that seems to me the right one: ranking by score is inconsistent with optimal acquisition when costs are heterogeneous, so the optimal K is a function of cost, not a constant. Trimming from 7 to 4.4 exposed tools while keeping success is possible; fixing a universal number is not.

Filtering can make the result worse

This is the part that appears in no product blog. A June 2026 benchmark with three registry sizes (25, 100 and 250 tools), six filtering methods, seven models and 26,460 runs:

Strategy	Success	Visible tools	Tokens
Expose the whole catalogue	32.1 %	125.00	56,062
Keyword, top-5	22.1 %	4.80	3,200
Keyword, top-10	22.4 %	9.54	5,356
State-aware	24.0 %	25.95	13,522
Full causal path	24.0 %	26.24	13,697

Four of the five filtering strategies perform worse than not filtering. The fifth, which does win comfortably, exposes 0.99 tools per step and requires hand-written precondition and effect contracts for every tool, plus a search of the causal path to the goal state. It is a planner with the goal given, not a retrieval filter, and it is not comparable with anything a gateway offers.

There is more evidence in the same direction. A work over a corpus of 43,000 tools measured that the agent’s completion rate drops when using retrieved sets against hand-annotated sets: the filter introduces its own recall ceiling. And top-K breaks the tasks that need several tools: on a benchmark of 16,464 endpoints, a fine-tuned retriever reaches a Recall@3 of 68.6 % but only 39.7 % of the queries have their complete tool set within the first three. Almost 29 points of gap between “the one I needed is there” and “all the ones I needed are there”.

Filtering is not free on security either. A March 2026 work attacks the retrieval layer by injecting adversarial tools that saturate the top-k: with injection rates of 1.2 % it manages to dominate the top-k between 91 % and 97 % of the time. It is an attack surface that exposing the whole catalogue does not have.

The most elegant counterpoint comes from Anthropic’s own numbers. With the same catalogue of more than fifty tools, the gain from tool search drops from 25 points on one model to 8.6 on the next, because the unfiltered baseline rises from 49 % to 79.5 %. The margin left by filtering shrank to a third in one generation of models. Any architecture decision taken here has a short shelf life.

The prompt cache trap

Here is what makes a homemade top-K filter a bad idea even if it worked.

Tool definitions do not go just anywhere in the prompt: they go at the beginning. Anthropic documents the order of the cacheable prefix as tools, then system, then messages, with a hierarchy where each level rests on the previous one. And the invalidation rule is explicit: modifying tool definitions, be it names, descriptions or parameters, invalidates the entire cache, and a change at one level invalidates that level and all subsequent ones.

OpenAI says the same and adds a detail: the prefix must remain unchanged, and order counts. The same set of tools served in a different order is a cache miss.

With vLLM serving an open model there is no tool block of its own. The tools parameter is serialised by the chat template inside the first system message, behind the system prompt text. For automatic prefix caching the effect is identical, because it is still near the absolute start of the prompt.

The consequence is the one that matters. A filter that returns a different set of tools on the next turn does not just invalidate the tool block: it invalidates everything behind it, which is the system prompt and the whole conversation, which is exactly the part that grows with the turns. At list prices, a prefix token goes from costing 0.1 times when read from cache to 1.25 times when written. An agent with sixty thousand tokens of stable prefix goes from some six thousand equivalent tokens per turn to some seventy-five thousand, plus the prefill that gets paid again in latency.

The two big providers have separately arrived at the same solution, and it is the opposite of filtering in the prefix. Anthropic excludes deferred tools from the prefix and, when the model discovers one through search, injects it inline inside the conversation: “the prefix stays intact, so prompt caching is preserved”. OpenAI: “all tools are loaded at the end of the model’s context window (…) this allows the cache to be preserved from one request to the next”. Anthropic’s operational recommendation is the stable core pattern: leave the three to five most used tools undeferred.

The closest academic work, from January 2026, states the problem without measuring it and recommends exactly that, keeping a fixed set of reusable general-purpose functions and implementing dynamic capability through code generation.

The July 2026 MCP specification picks up both halves of the matter. Servers “should return the tools from tools/list in a deterministic order to allow caching in the client and improve the hit rate of the model’s prompt cache”. And it adds mandatory ttlMs and cacheScope in listing results, so that the client can freeze the catalogue for a known window instead of re-querying it and propagating any variation to the prompt.

Nobody has publicly measured the cost in euros of this effect. The only attempt I found is a self-published audit of ten thousand turns that puts it at 2.4 % to 3.5 % of cache misses, with non-reproducible methodology. It is a gap in the literature, and the order of magnitude in the previous paragraph comes from applying list prices, not from a measurement.

And now, LiteLLM

With all of the above in mind, the gateway’s levers read differently.

The semantic filter does not act on the MCP endpoint. It is registered as a generic LiteLLM callback, and its only dispatcher is the proxy’s precall hook, which additionally discards anything that is not completion, acompletion or aresponses. The tools/list path does not invoke that hook at any point; there is a comment in the code that admits it in passing, when explaining why the listing’s JWT signature is done separately. An external MCP client connected to /mcp receives the whole catalogue, filter enabled or not.

Lever	Serves external MCP clients	Breaks the prompt cache
Per-server `allowed_tools`	Yes	No, it is static
Name and description overrides	Yes	No, it is static
Selection by route or header	Yes	No, if it is stable per agent
`/mcp/proxy` and virtual tools	Yes	No, the surface is fixed
Semantic filter	No	Yes, it rewrites the block per request

When it does act, the filter has details you need to know before enabling it. It embeds only each tool’s description, with the name as a fallback if it is missing; the parameter schema does not go in. Of the query it embeds only the last user message: on turn twelve of an agent conversation, that might be “yes, go ahead”. The tool embeddings are cached and indexed at start-up; the query’s is computed on every request, with a synchronous call inside a coroutine.

Its failure modes are almost all open, which is the reasonable thing: if the embedding model does not respond, if nothing passes the threshold, if the names do not match the catalogue, it returns all the tools. The exception is context overflow, which fails closed with a 400, and which if it happens while building the index during start-up is memoised and makes all subsequent requests fail until you restart. The defaults are text-embedding-3-small, top_k of 10 and a threshold of 0.3, under litellm_settings.mcp_semantic_tool_filter.

/mcp/proxy does serve external clients, and that is where its real value lies. It replaces the catalogue with three tools with opaque identifiers. How search_tools searches deserves a line, because it is not what you would assume: by default it is neither a regular expression nor a semantic match, it is a keyword counter, how many tokens of the query appear as a substring in the concatenated name and description. It only moves to embeddings if you configure a model in mcp_tool_search. The result cap is hard-coded at five in this mode, and the client cannot ask for more. In exchange, call_tool goes through the same permission checks as a normal call plus a validation of the argument schema that the normal route does not do at that point.

There is a related variant: with mcp_tool_search_enabled in the key’s permissions, the catalogue is replaced by four virtual tools, and there the search one does expose top_k to the client.

The catalogue order is not deterministic, and that breaks the client’s cache without anybody filtering anything. The list of allowed servers comes from iterating a Python set of strings, so with the random hash seed the order differs between processes, between uvicorn workers and between restarts. There is no sorting anywhere in the listing path, and there is no tool cache: every tools/list queries the upstreams live, so the order within each server also depends on what the upstream returns that time. The gateway does not offer today the guarantee that the July specification recommends. The fix, sorting the catalogue before serving it, fits on one line.

Nor is there any pagination towards the client. The catalogue is returned whole, always.

And the lever almost nobody uses is the static one. Every MCP server accepts tool_name_to_display_name and tool_name_to_description, and those overrides really do replace what is sent to the client in tools/list, not just what the administrator sees in the interface. The tools/call routing translates from the new name to the real one before any permission check. With that you can trim a four-hundred-token description written by a third party, rename an ambiguous tool so it does not compete with a similar one, and do it without touching the prompt cache, because it is static and holds for all turns.

It is also the rug pull mitigation left pending from the security section: if the description the model sees is the one the operator wrote, whatever the upstream server changes stops reaching the context. Two limits. They are configured through the database or the management API, not through config.yaml. And the reverse name translation is by exact equality and is not disambiguated between servers, so two servers with the same display name route unstably, made worse by the non-deterministic order of the previous paragraph.

Alongside that, per-server allowed_tools remains the blunt and effective instrument: it filters the listing the client sees, reduces tokens, and is static.

The ladder, in order

Trim. Per-server allowed_tools, plus a per-key allowlist. Cheap, static, does not break the cache and reduces the permission surface at the same time.
Rewrite the descriptions with the overrides. That is where the evidence is, because the description carries weight and the name almost none, and along the way it pins the contract against the rug pull.
Split by route. Each agent to /{server}/mcp or to a tool set, never to the aggregated endpoint. Stable per agent, so the cache survives.
Sort the catalogue before serving it if you control the deployment, while the gateway does not do it.
/mcp/proxy when the catalogue goes past thirty or fifty tools. It is progressive disclosure and it works with external clients, which is more than you can say for the semantic filter.
The semantic filter, only when the one calling the model is LiteLLM. With a generous top_k, knowing that it embeds only the last user message and that context overflow is a closed and sticky failure.
What not to do: a home-grown top-K filter that rewrites the tools array on every turn. It costs the whole prefix and, according to four of the five measured strategies, probably makes selection worse.

The declared cost, which is worth zero

The calculation has four precedence steps: cost set by a post-call hook, per-tool cost in tool_name_to_cost_per_query, server default cost, and 0.0.

Without configuring mcp_server_cost_info, all MCP traffic is accounted at zero and consumes neither key nor team budget.

There are exactly two Prometheus metrics for MCP:

litellm_mcp_tool_calls_total
litellm_mcp_tool_call_spend_metric

Both with the same eight labels: tool name, server name, key hash, key alias, team, team alias, user and end user.

And the second is only incremented if the cost is greater than zero. With the default configuration, that time series never appears in Prometheus. It is not that it is worth zero: it is that it does not exist.

Nor is there a tools/list metric, or MCP latency, or MCP errors, or upstream server health. For that you are left with the generic request metrics.

In the reports, /spend/calculate has no MCP branch: it accepts a model and messages, or a completion response, and it always calls the LLM cost calculation. The per-team report does include MCP calls, grouped by the model column, which for these rows is worth MCP: tool_name. That is: every tool appears as if it were a model, with zero input and output tokens. Only the sessions endpoint separates the two call types explicitly.

Observability, and the trace that gets cut

There are two OpenTelemetry stacks in LiteLLM and which one runs depends on an environment variable.

Without LITELLM_OTEL_V2=true, the old logger runs, and it has no MCP handling: the only appearance of MCP in the whole file is the metadata key in a list. The tool call comes out as a generic litellm_request span with the model set to MCP: name.

With v2 enabled there is a dedicated span, with a dedicated role, client kind and a parent in the proxy’s request span. The span name is tools/call tool-name. The attributes are correct and follow the conventions:

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`mcp.method.name`	`tools/call`
`mcp.session.id`	MCP session identifier
`gen_ai.tool.name`	Tool name
`gen_ai.tool.call.arguments`	Only with content capture enabled
`gen_ai.tool.call.result`	Only with content capture enabled
`server.address`, `server.port`	From the upstream URL, redacted
`litellm.cost.total`	The cost that is probably worth zero

Content capture is off by default. Without it you see neither the arguments nor the result in the trace.

A note for anyone working with semantic conventions: mcp.tool.name does not exist. The MCP registry has exactly four attributes (mcp.method.name, mcp.session.id, mcp.protocol.version and mcp.resource.uri), all four in development, and the tool name goes in gen_ai.tool.name. LiteLLM gets it right. And the GenAI conventions were extracted into their own repository which as of today has no release at all, so there is no version number to cite.

In Langfuse, the vendor mapper returns an empty dictionary for any span that is not an LLM call. An MCP call appears as a raw span with its attributes, with no native input, output or model fields.

The trace is cut at the gateway

Incoming context is read. The traceparent the client puts in params._meta, following the specification’s propagation proposal, is extracted and turned into a link on the span, never into its parent. The reason is documented in the code: parenting to the remote trace would leave the span hanging off a trace whose root never reaches the backend. The client’s baggage is discarded on purpose to avoid forging identity attributes.

Outgoing context is not propagated. There is no propagator injection call anywhere in the tree. The headers that go out towards the upstream MCP server are built with authentication and the authorised extra headers, and nothing else.

The operational consequence: there is no end-to-end trace. If the upstream MCP server is instrumented, its spans live in another trace and there is no way to correlate them automatically.

Auditing and the conflict with privacy

A tool call leaves a normal spend row, with call type call_mcp_tool, spend equal to the cost per query, model MCP: name, zero tokens in the three token columns, a dedicated mcp_namespaced_tool_name column, and the usual key, team, user and session fields.

And in the metadata column, in JSON, the complete structure of the call: name, arguments, result, server, namespaced name, MCP session identifier, authentication mode and server resource with the URL redacted.

The problem

That structure is written into the metadata column without passing through the privacy switch. The assignment is direct and is not conditioned by the function that governs whether prompts and responses are stored, which does control the messages and response columns.

And the redaction function does not cover it either: it only touches messages and response. A search for MCP in the redaction file returns nothing.

The practical result: an operator who enables turn_off_message_logging for GDPR keeps sending the full arguments of every tool to every logging destination they have configured. To the SIEM, to Langfuse, to S3.

If an argument carries a national ID number, an address, a case number or a medical record, it has been exported.

There is an aggravating factor that does not depend on LiteLLM. The revision of the specification in force introduces an extension that allows mirroring tool arguments into HTTP headers visible to load balancers, proxies and web application firewalls. The specification recommends not marking sensitive parameters that way, and it is a recommendation, and the upstream server decides.

The fit with the ENS

The conflict is real and has a familiar shape: activity logging (op.exp.8) requires the call to be traced and non-repudiable; protection of personal data (mp.info.1) requires that log not to become a secondary repository of personal data with a different purpose. The tool argument is simultaneously the evidence of the action and the data. Redacting it destroys non-repudiation; storing it in the clear creates the repository.

The defensible way out you can build today: a hash of the argument in the audit log, the argument in the clear with encryption (mp.info.3), separate retention and its own access control. The hash sustains non-repudiation and the clear text is only opened under procedure.

There are two other measures this material touches directly. Change management (op.exp.5) because registering an MCP server and, above all, changing the description of an already approved tool are changes to the security configuration. And maintenance and updates (op.exp.4) because, as you can see in the next section, an old version of the gateway is a non-compliance with a name and a number.

Two honesty warnings. The codes follow the numbering of Annex II of RD 311/2022 and you have to check them against the text in force before citing them in a compliance document. And on NIS2: as of the date of this post the directive is still not transposed in Spain, with the draft bill on cybersecurity coordination and governance going through the process and a reasoned opinion from the Commission over the delay. What is enforceable today in the public sector and its suppliers is the ENS.

What is not logged

An administrative gap: registering, modifying and deleting MCP servers generates no audit log. There are three TODO comments in the management code that say so, one of them with an if is_audit_logging_enabled(): pass. In the same deletion block there are two more TODOs about not cleaning up orphaned permissions on keys and teams.

Security

The vulnerability you need to know about

CVE-2026-42271, published on 8 May 2026. Affects LiteLLM from 1.74.2 up to versions prior to 1.83.7. CVSS 3.1 of 8.8.

The POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list endpoints, which exist to preview an MCP server before saving it, accepted the stdio transport’s command, args and env in the request body. A low-privilege key executed arbitrary commands on the proxy host. It also affects Red Hat OpenShift AI on several branches.

It is the story of this post condensed into one line: the gateway you put in to contain MCP was the hole.

It does not come alone. The 1.83 branch concentrates three more serious ones:

CVE	Component	CVSS	Fixed in
`CVE-2026-42208`	SQL injection in key validation, with no prior authentication	9.8	1.83.7
`CVE-2026-35030`	OIDC userinfo cache keyed on `token[:20]`, collision and impersonation	9.1	1.83.0
`CVE-2026-40217`	Remote execution through bytecode rewriting in the guardrails test endpoint	8.8	no version recorded

Plus two privilege escalations fixed in 1.83.10 and 1.83.14.

The operational conclusion is short: a deployment below 1.83.14 is exposed to remote execution or to an unauthenticated SQL injection. The current branch is 1.102.0.

And a note for context: the whole MCP ecosystem has the same pattern. The unauthenticated remote execution in MCP Inspector, the command injection in mcp-remote when connecting to an untrusted server, DNS rebinding disabled by default in the Python and TypeScript SDKs until late 2025, the directory escapes in the reference servers. Connecting to an untrusted MCP server compromises the client, not just the other way round.

The rug pull nobody covers

Documented since April 2025: a server can change a tool’s description after the client has approved it. The description is what the model reads to decide when and how to use it, so changing it is changing the behaviour without asking for permission again.

The specification in force does not address it. It defines list change notification and cache freshness fields, which are recency mechanisms, not integrity ones. The closest normative warning is that clients must treat tool annotations as untrusted unless they come from trusted servers.

LiteLLM does not cover it either. The only “pinning” in the code is of the configuration’s server identifier, so that it does not change when you edit the YAML. There is no hash or comparison of the description or the input schema between successive listings. A search for related terms in the MCP module: nothing functional.

The control exists outside the gateway. mcp-scan, from the same researchers who documented the attack, implements description hash pinning precisely to detect this.

In ENS terms it is an unauthorised change to the security configuration that today goes undetected. And the home-made fix fits in a cron job: store the hash of the description and schema of every approved tool, compare against tools/list periodically, alert on the difference.

Three more holes in the code

No URL validation towards the upstream. A new server’s url field has no validator. The SSRF protection utilities exist in the tree and are used in OAuth discovery and in downloading OpenAPI specifications, but not in the egress of tools/call. A server pointing at a private range or at the cloud metadata IP is accepted as is.

The stdio transport is local execution on the proxy host. There is a command allowlist: npx, uvx, python, python3, node, docker, deno, extendable through an environment variable. The code’s own comment admits the residual risk: the permitted runtimes can execute code through arguments. With docker and npx in there, it is de facto arbitrary execution for any proxy administrator.

Two server modes skip the gateway’s authentication. Pass-through and authentication delegation to the upstream completely omit virtual key validation. They are well hardened (fail-closed if the target is mixed or unresolvable, strict comparison against truthy values, explicit blocking of the client credentials flow so as not to lend out the proxy’s service account), and the result is still that those servers have no gateway access control.

The limit that truncates in silence

The tools/list pagination stops and returns what it has accumulated in three cases: repeated cursor, one thousand pages, or expiry of a global deadline. The docstring declares it: a slow or faulty upstream produces a partial catalogue instead of an error.

The first two log a warning. The third, the deadline, exits without logging anything at all. A slow server makes part of its tools disappear from the catalogue with no trace left behind.

Correction on the slot leak

In the previous post I took issue 34534 at face value, according to which every MCP tool call acquires a max_parallel_requests slot and does not release it. Tracing the 1.102.0 code, that diagnosis no longer holds: the release path exists, on both success and failure, and the limiter is registered as a litellm callback and not just as a proxy hook.

With one honest caveat: I found no regression test covering the full cycle for MCP, and the definitive confirmation is a real proxy with max_parallel_requests: 2 and five MCP calls in a row. If anybody reproduces it today, I would like to know.

The specification moved underneath

The MCP revision in force is 2026-07-28, published on 28 July 2026, and in between there was another one, 2025-11-25. The changes in the latest one are not cosmetic:

Protocol sessions and the session header are removed. Cross-request state becomes an explicit identifier in the tool arguments.
The initialisation handshake is removed. Every request carries version and capabilities in _meta. MCP becomes stateless.
server/discover appears as a mandatory call.
Elicitation changes shape: out go server-initiated requests, in comes a multi-round pattern where the server returns an “input required” result and the client retries with the answers.
ping, logging/setLevel and the roots change notification are removed. SSE stream resumption is removed.
Standard HTTP headers are mandatory on POST, which lets a gateway or a web application firewall decide without parsing the JSON.
On authorisation: mandatory issuer validation in the client, and dynamic client registration is deprecated in favour of client identifier metadata documents.
Roots, sampling and logging are deprecated, with a minimum window of twelve months.

LiteLLM 1.102.0 advertises 2025-06-18. Its version enumeration has three values and none of them is later than that date. The literal is sent in the initialize against the upstream.

One nuance that is not entirely bad: the version the gateway advertises to its clients is not hard-coded in LiteLLM, the SDK negotiates it. And there is partial implementation of the intermediate revision, because the elicitation and sampling handlers reference it in their docstrings. That is: partial implementation of one revision while advertising an earlier one.

In practical interoperability it does not hurt yet, because the client ecosystem is still mostly on the 2025 revisions. It hurts in planning: July’s stateless redesign invalidates the gateway’s session model, and that is work ahead.

Two ecosystem notes for anyone choosing. There are clients that speak remote streamable HTTP natively (Claude and Claude Code, ChatGPT with OAuth and dynamic registration, VS Code with Copilot, Cursor, Zed, Kiro) and others that still demand stdio or a local bridge with mcp-remote, among them Windsurf, Cline and Continue.

Tool execution from the proxy itself

Besides serving MCP outwards, the proxy can consume it itself. It is enabled with an mcp block in the body, with server_url equal to the litellm_proxy sentinel, and it works both on /v1/responses and on /chat/completions.

The flow: it separates the gateway’s tools, lists them, filters by allowed, deduplicates, translates them into OpenAI function format and concatenates them. It calls the model. If there are tool calls and auto-execution is permitted, it runs them against the MCP servers. It builds the follow-up and makes one second call.

Two limits you need to know before designing on top of this.

The loop is a single hop. There is no multi-step iteration. The calls from the first response are executed and the follow-up is returned. A plan that chains tools requires the client to call again.

Auto-execution fails closed and as a block. It requires all MCP references to have approval set to “never”; a single one requiring approval disables auto-execution for the whole request.

And a third one, which is the one that bites: the multi-turn state of /v1/responses is rehydrated from the spend table. The query is literal, it looks up the session of the previous request_id and retrieves all the rows of that session ordered by time.

With disable_spend_logs: true nothing is written, the query comes out empty and the history is lost in silence, with no 4xx error. The code itself acknowledges it: it does not expect retries because deployments that do not write spend logs have nothing to wait for. It is a coupling between a performance option and a conversation feature that nobody expects to find.

Health and day-to-day operation

Health checks exist, they are on demand and they are not periodic. GET /v1/mcp/server/health accepts repeated identifiers and returns per-server status. The implementation opens a client and runs an empty operation inside the session, with a ten-second timeout.

It skips the check entirely if the server requires per-user authentication, and those always return unknown status.

The surprising part: the status, last check and error columns have existed in the database since a May 2025 migration, and the check function returns an object without writing to the database. Status is recalculated on every request. There is no periodic MCP health job.

When an upstream server is down, the behaviour is to degrade and not block. The listing fans out in parallel and each failing server contributes an empty list plus a result classified into six categories. Those results reach the client in the response’s _meta under a vendor key, without filtering the upstream’s text or URL. The reason for the design is in the code: an empty contribution with no signal makes a broken upstream indistinguishable from a healthy server with no tools.

There is no cache of the previous list. A server that is down disappears from that turn’s tools.

On registering and deregistering without restarting, there are two worlds. Database servers can be created, edited and deleted through the API, and every operation reloads the registry with an atomic swap. Propagation between replicas goes through Redis publication on the configuration change channel, if there is Redis, and through polling every 30 seconds if there is not.

Servers declared in config.yaml are loaded once, at start-up. There is no re-read. Changing them requires a restart.

And there is a precedence trap between the two worlds, with two different warnings in the code. A database row with the same identifier as a configuration entry hides the configuration one entirely. And the other way round, a configuration identifier that matches the name or alias of a database server captures its permissions, because identifiers are resolved before names. Both cases emit a log warning, once.

A hardened `config.yaml`

general_settings:
 # Access is not open by default, but this closes the empty-key
 # inheritance towards the team, which is the one that surprises people.
 require_key_mcp_access_defined: true
 require_end_user_mcp_access_defined: true
 # Without this, allow_all_keys overrides the key's explicit scope.
 mcp_allow_all_keys_respects_mcp_scope: true

litellm_settings:
 # Does not cover MCP arguments. See the auditing section.
 turn_off_message_logging: true
 callbacks: ["langfuse_otel"]

mcp_servers:
 inventario:
 url: https://mcp-inventario.interna.svc/mcp
 transport: http
 auth_type: bearer_token
 auth_value: os.environ/MCP_INVENTARIO_TOKEN
 # Explicit allowlist. Also enforced on tools/call.
 allowed_tools:
 - buscar_articulo
 - consultar_stock
 # Never true on a server with data.
 allow_all_keys: false
 access_groups: ["operaciones"]
 timeout: 20
 max_concurrent_requests: 8
 mcp_info:
 description: "Inventario de almacen, solo lectura"
 # Without this, MCP spend is zero and its metric does not exist.
 mcp_server_cost_info:
 default_cost_per_query: 0.0005
 tool_name_to_cost_per_query:
 consultar_stock: 0.0002

And in the process environment:

# Without this there is no MCP span, only a generic litellm_request.
LITELLM_OTEL_V2=true
# Full block hashes, not truncated, if correlating
# with engine events.
LITELLM_MCP_CLIENT_TIMEOUT=45
LITELLM_MCP_TOOL_LISTING_TIMEOUT=20

For clients, the rule is not to use the aggregated endpoint. Each agent points at /{server}/mcp or at a tool set, or at the three-tool proxy mode if the catalogue is large.

Checklist

Check the version. Below 1.83.14 there is remote execution and an unauthenticated SQL injection with an assigned CVE.
Inventory which servers have allow_all_keys and enable the setting that makes it respect the key’s scope.
Audit the access groups separately, because they are additive on top of the key and team ceiling.
Declare in writing the fail-open behaviour on tool permission resolution errors.
Measure the real cost: number of tools per server from the listing row, and tools block tokens from the stored request body.
Choose a catalogue strategy before connecting the third server: three-tool proxy mode, semantic filter, or selection by route. Not the aggregated endpoint.
Declare mcp_server_cost_info even with a token value, or the spend metric will not exist.
Enable the OTel v2 logger, and decide deliberately whether to enable content capture.
Assume the trace is cut at the gateway and plan correlation with the upstream by other means.
Treat the spend table’s metadata column as a personal data repository: encryption, its own retention and separate access control.
Set up the rug pull check, because nobody does it: hash of the description and schema of every approved tool, periodic comparison against the listing.
Restrict egress towards MCP servers at the network level, because the gateway does not validate the destination URL.
Do not enable the stdio transport unless you need it, and if you do, review what is on the command allowlist.
If you use /v1/responses with multi-turn state, do not enable disable_spend_logs.

Traps and things that are not what they seem

MCP access is not open by default in 1.102.0. I got that wrong in the previous post. What opens things up is per-server allow_all_keys.
allow_all_keys overrides the key’s explicit scope unless you enable a setting that ships disabled.
require_key_mcp_access_defined only closes key-to-team inheritance. Access group grants remain additive.
On an indeterminate failure resolving tool permissions, the code fails open for keys and JWTs.
The default cost of an MCP call is 0.0, and the spend metric is only emitted if the cost is greater than zero, so the series does not exist.
MCP calls appear in the per-team report as if they were models, named MCP: tool and with zero tokens.
The OTel logger with MCP support is off by default. Without the environment variable there is no MCP span.
Trace context is not propagated towards the upstream MCP server. There is no end-to-end trace.
mcp.tool.name does not exist in the semantic conventions. The name goes in gen_ai.tool.name.
Message redaction does not cover tool arguments. They are written in the clear into the spend table.
Registering, modifying and deleting MCP servers generates no audit log. There are three TODOs in the code.
The semantic tool filter is not applied on tools/list. Only on /chat/completions and /v1/responses.
The catalogue order is not deterministic, because it comes from iterating a set of strings. It breaks the client’s prompt cache even though nobody filters anything.
search_tools in proxy mode counts keywords by default, it uses neither regular expressions nor embeddings, and its cap of five results is hard-coded.
Description overrides really do replace what the client sees, and they are the only catalogue lever that does not break the cache. They are only configurable through the database or the API.
spec_version no longer exists in the configuration schema.
/mcp/sse does not use the SSE transport, and the messages endpoint its object advertises is dead code.
Listing pagination truncates in silence when the deadline expires, without leaving a single log line.
A database row hides a configuration entry with the same identifier, and the other way round a configuration identifier can capture the permissions of a database server.
Servers declared in config.yaml require a restart. Only the database ones reload hot.
The health check does not write its result to the database even though the columns exist, and there is no periodic job.
The gateway does not validate the upstream MCP server’s URL. The anti-SSRF utilities exist and are not used on that path.
docker and npx are on the stdio command allowlist.
The /v1/responses tool loop is a single hop, and disable_spend_logs: true breaks its multi-turn behaviour with no error.
LiteLLM advertises the June 2025 revision of the protocol, two behind the one in force.

Closing

There is a thread running through the six posts in this track that is more visible here than anywhere else. The gateway is chosen for one job and ends up doing five, and each new job arrives with its own defaults, which are almost never the ones you would have chosen.

In the case of MCP the asymmetry is twofold. On one side, the part that looks dangerous is better built than you would assume: access control is enforced on execution and not just on the listing, the hierarchy is built out of intersections, header forwarding is an allowlist, and default access is closed, contrary to what I myself wrote two days ago. On the other, the part nobody looks at is worse: tool arguments come out in the clear bypassing the privacy switch, spend is worth zero and its metric never gets to exist, the trace is cut, and a change to the description of an approved tool goes undetected by anybody in the whole chain, neither the specification nor the gateway.

The architecture decision that comes out of this is that the MCP door has to be treated as what it is, an interconnection of systems with third parties, and not as one more function of the LLM proxy. That means its own inventory, formal registration, change control over every tool’s description, network isolation towards the upstream and a log designed knowing that it is going to contain personal data.

And it means accepting that the cost you really pay for connecting fifteen MCP servers is in no column of the spend table. It is spread across the prompt_tokens of every model call, on every turn, and across the accuracy you lose when the catalogue is too large for the model to choose well. That number can be computed with what the gateway already stores. The first step is wanting to compute it.

Sources

LiteLLM, MCP Gateway: https://docs.litellm.ai/docs/mcp.
LiteLLM, código: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, auth/user_api_key_auth_mcp.py, cost_calculator.py, faults/list_outcomes.py, litellm/proxy/management_endpoints/mcp_management_endpoints.py, litellm/proxy/spend_tracking/spend_tracking_utils.py, litellm/litellm_core_utils/redact_messages.py, litellm/integrations/otel/, litellm/responses/litellm_completion_transformation/session_handler.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en llamadas MCP: https://github.com/BerriAI/litellm/issues/34534.
NVD, CVE-2026-42271, ejecución de comandos por los endpoints de prueba de MCP: https://nvd.nist.gov/vuln/detail/CVE-2026-42271.
NVD, CVE-2026-42208, inyección SQL sin autenticación en la validación de clave: https://nvd.nist.gov/vuln/detail/CVE-2026-42208.
NVD, CVE-2026-35030, colisión de clave de caché de userinfo OIDC: https://nvd.nist.gov/vuln/detail/CVE-2026-35030.
NVD, CVE-2025-49596, ejecución remota en MCP Inspector: https://nvd.nist.gov/vuln/detail/CVE-2025-49596.
NVD, CVE-2025-6514, inyección de comandos en mcp-remote: https://nvd.nist.gov/vuln/detail/CVE-2025-6514.
NVD, CVE-2025-66416, protección de DNS rebinding desactivada por defecto en el SDK de Python: https://nvd.nist.gov/vuln/detail/CVE-2025-66416.
Invariant Labs, MCP Security Notification: Tool Poisoning Attacks (tool poisoning, rug pull, shadowing): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks.
Invariant Labs, Introducing mcp-scan (fijado de herramientas por hash): https://invariantlabs.ai/blog/introducing-mcp-scan.
Model Context Protocol, changelog de la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/changelog.
Model Context Protocol, changelog de la revisión 2025-11-25: https://modelcontextprotocol.io/specification/2025-11-25/changelog.
Model Context Protocol, Security Best Practices (confused deputy, indicadores de recurso, validación de audiencia): https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.
NSA, Model Context Protocol: Security Design Considerations for AI-Driven Automation, mayo de 2026: https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDF.
CISA y agencias aliadas, Careful Adoption of Agentic AI Services, mayo de 2026: https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services.
Anthropic, Advanced tool use (coste en tokens de las definiciones, búsqueda de herramientas, precisión de selección): https://www.anthropic.com/engineering/advanced-tool-use.
Anthropic, Code execution with MCP: https://www.anthropic.com/engineering/code-execution-with-mcp.
Anthropic, Prompt caching (jerarquía tools → system → messages e invalidación): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.
Anthropic, Tool search tool (herramientas diferidas fuera del prefijo, umbral de 30 a 50 herramientas): https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, Prompt caching (el orden de las herramientas cuenta): https://developers.openai.com/api/docs/guides/prompt-caching.
OpenAI, Tool search (las herramientas descubiertas se cargan al final del contexto): https://developers.openai.com/api/docs/guides/tools-tool-search.
vLLM, Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html.
Microsoft, Use tools in chat (límite de 128 herramientas por petición en VS Code): https://code.visualstudio.com/docs/copilot/agents/agent-tools.
RAG-MCP, prueba de estrés de 1 a 11.100 servidores candidatos, mayo de 2025: https://arxiv.org/abs/2505.03275.
Benchmarking the Benchmarks, desalineación del 18,5 % entre etiqueta y realidad, junio de 2026: https://arxiv.org/html/2607.02577v1.
MCP-Atlas, taxonomía de fallos sobre 220 herramientas y veinte modelos, febrero de 2026: https://arxiv.org/html/2602.00933v3.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307.
MCP Tool Descriptions Are Smelly, auditoría de 856 herramientas de 103 servidores, febrero de 2026: https://arxiv.org/html/2602.14878v1.
ToolMenuBench, seis estrategias de filtrado y 26.460 ejecuciones, junio de 2026: https://arxiv.org/html/2606.15508.
Retrieval Models Aren’t Tool-Savvy (ToolRet), corpus de 43.000 herramientas, marzo de 2025: https://arxiv.org/abs/2503.01763.
Tools Are Not Islands, brecha entre Recall@3 y conjunto completo, julio de 2026: https://arxiv.org/html/2607.25718.
ToolFlood, saturación adversaria de la capa de recuperación, marzo de 2026: https://arxiv.org/html/2603.13950.
Scores Are Not Decisions, el K óptimo como función del coste, julio de 2026: https://arxiv.org/html/2607.27083v1.
Don’t Break the Cache, caché de prompts en tareas agénticas de horizonte largo, enero de 2026: https://arxiv.org/html/2601.06007v2.
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer, mayo de 2026: https://arxiv.org/html/2605.24660v1.
Semantic Tool Discovery for LLMs, marzo de 2026: https://arxiv.org/abs/2603.20313.
OpenTelemetry, convenciones semánticas de GenAI y MCP: https://github.com/open-telemetry/semantic-conventions-genai.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
IBM ContextForge, gateway y registro MCP: https://github.com/IBM/mcp-context-forge.
Docker MCP Gateway: https://github.com/docker/mcp-gateway.
agentgateway: https://github.com/agentgateway/agentgateway.

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

Fri, 11 Sep 2026 00:30:00 +0200

Fifth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability, virtual keys covered governance and humans and agents covered the coexistence of two classes of traffic. Here a claim of mine from June gets taken apart. Everything that follows is verified against LiteLLM 1.102.0 (commit of 10 September 2026), vLLM 0.29.0 and the main branches of production-stack, llm-d and Dynamo from the second week of September 2026.

TL;DR

LiteLLM does not do prefix-aware routing, neither in the open version nor in the paid one. It is not a licensing question. A search across the whole tree for kv_aware, cache_aware, radix, prefix_cache and block_hash returns zero matches in the router. Nor is there any call from the router to /metrics or to any vLLM endpoint. In June I wrote the opposite in the inference router article and I was wrong.

What LiteLLM has is affinity, which is a different thing. Four pre-call checks pin a request to a replica by session identifier, by key hash, by previous response identifier or by exact hash of a marked prefix. None is active by default. Affinity distributes conversations; prefix routing distributes prefixes shared between different conversations, which is the case that multiplies throughput.

The only one that looks at the prompt is fragile in three ways. If the client does not send cache_control, the key comes out null and the check does nothing. If the cut point goes in the last turn, which is what the automatic injection does by default, the key changes on every turn and it never hits. And the pin’s TTL is 300 seconds hardcoded in two lines, while the provider offers one-hour caches.

The engine does not answer questions, it publishes events. There is no vLLM endpoint you can ask whether it has a prefix. What there is, is a ZMQ socket that emits stored and removed blocks in msgpack, switched off by default, with silent discarding when the consumer does not keep up. A router that wants to be exact has to replicate the engine’s hash in full: chained SHA256, deterministic seed, block granularity and the extra keys in their order.

And by default it does not publish the hits, only the additions. The index of a router that only listens drifts away from the engine’s real LRU without noticing. To see the hits you have to ask for it request by request with kv_cache_report_mode: full.

The three routers that do do it are very different from each other. The vLLM production-stack one indexes chunks of 128 raw characters in a trie that is never purged, models no eviction and dies on every restart. The llm-d one offers an approximate mode and an exact event-driven one, all its plugins are in alpha or beta, and its default tokeniser does not produce real tokens. NVIDIA Dynamo’s keeps an event-driven radix tree with a weighted cost function, and ships with the parameter that avoids starving a freshly scaled replica switched off.

The big improvements only appear in the best possible case. The two- and three-digit numbers are published by the projects themselves, with synthetic shared prefixes and a round-robin balancer as the reference. The most complete independent measurement, from Meta, obtains 2.3 times more throughput and also identifies load regimes where affinity leaves capacity at between half and two thirds of the reference. Another, on a multi-region deployment with 89 % reuse, measures single-digit improvements.

The correct architecture has two layers, not one. LiteLLM on top for identity, budget and audit, pointing at a single destination per model group. The KV-aware router underneath, inside the engine’s domain. Mixing it into a single piece is not an option that exists today.

You are here: between the gateway and the engine

This article occupies the gap between two layers that already have an article of their own. Above, the L7 switchboard and the gateway that has been operated since the first article in this track. Below, the fundamentals of the KV cache and hit rate engineering.

The gap is the piece that decides which replica each request goes to, knowing what each replica has in memory. It is the only function in the stack that cannot be resolved in the gateway or in the engine separately.

The analogy: the library with four counters

A library with four counters. Each librarian has in front of them a pile of the books they have consulted lately. When a reader comes in asking for the third volume of an encyclopaedia, if that volume is in the pile at counter two, serving them there takes seconds. At any other counter someone has to go down to the store room.

A normal balancer sends the reader to the counter with the shortest queue. It is the right decision if all the counters are interchangeable, and it is the wrong decision here, because they are not: what distinguishes them is not the queue, it is the pile.

Session affinity resolves one specific case of this. The returning reader is always sent to the same counter, because they will probably ask for the fourth volume of the same encyclopaedia. It works, and it does not resolve the general case: when a hundred different readers ask for the same third volume, session affinity spreads them across the four counters and four librarians go down to the store room for the same book.

Prefix routing is looking at what the reader asks for before assigning them a counter. It has two costs that the analogy makes visible straight away. The first is that somebody has to keep track of what is in each pile, and that bookkeeping takes work. The second is that if everybody asks for the third volume, counter two becomes the library’s only desk while the other three look on.

The correction: what I wrote in June

In the inference router article, on 2 June 2026, I closed the prefix routing section with this sentence: “vLLM Production Stack router implements it natively. NVIDIA Dynamo too. LiteLLM in its enterprise version has a beta.”

The first two hold up. The third is false and was no less false in June. While preparing the humans and agents article the first hint appeared, and full verification against 1.102.0 confirms it.

Case-insensitive search over litellm/, enterprise/ and gateway/, for the terms kv_aware, kv-aware, cache_aware, cache-aware, radix, prefix_cache, prefix-cache, kv_cache, block_hash, prefix_hash, prefix_score and longest_prefix: zero matches. Search for vllm inside litellm/router.py, litellm/router_strategy/ and litellm/router_utils/: zero. Search for num_requests_waiting, gpu_cache_usage or any scraping of the backends’ /metrics from the router: zero. LiteLLM’s router never asks the engine anything.

Search for premium_user in the files where those checks would live: a single result, and it is about something else. There is no paid version with this feature. The only routing piece that is behind a licence is per-tag budgets, in litellm/router_strategy/budget_limiter.py.

Where the error came from. LiteLLM’s documentation has an entry on dynamic TPM and RPM allocation marked as beta and enterprise, and a pre-call check called prompt_caching. Skim-read and added together, they give the impression of prefix routing in beta. Read in the code, prompt_caching is something else, and it deserves a section of its own because it is still recommended for this problem.

What LiteLLM does have: four affinities

The optional_pre_call_checks field accepts eight values. None is enabled by default: apply_default_settings passes an empty list. One of the eight, forward_client_headers_by_model_group, is declared in the type and in the interface schema and is consumed by no branch of the code. It is a dead value.

The four that pin a request to a replica are these.

Check	What it pins	Where it stores it	TTL
`session_affinity`	Session identifier	Redis with a Lua script, mirrored in memory	`deployment_affinity_ttl_seconds`, 3600 by default, refreshed
`deployment_affinity`	API key hash	Same	Same
`responses_api_deployment_check`	`previous_response_id`	Same	Same
`prompt_caching`	Exact hash of a marked prefix	`DualCache`	300 s, hardcoded

The other four are router_budget_limiting, encrypted_content_affinity, enforce_model_rate_limits and the dead value.

`prompt_caching`: why it is no use for this

The key is a SHA256 over the serialisation of the cacheable prefix, and “cacheable prefix” means everything up to and including the last block marked with cache_control: {"type": "ephemeral"}. Three consequences, checked by running the functions in isolation over the repository’s code.

If the request carries no cache_control, the extraction returns an empty list, the key comes out null and the check pins nothing. Automatic injection of cut points is only enabled with litellm.enable_anthropic_prompt_caching, which defaults to false, and only for Anthropic and Bedrock providers.

If the cut point is in the system prompt, the key does not change when turns are added and the pin works. This is the good case and it requires the client to mark it deliberately.

If the cut point is in the last message, the key changes on every turn and the pin never hits. And this is exactly the default case of the automatic injection, which places one point on the system and another at index -1. Since the second is the last, the cacheable prefix becomes the whole conversation.

On top of that there is the TTL. Two lines with ttl=300, one of them with the comment # store for 5 minutes. There is no parameter to change it. Issue 28427 has had it open for months with no response from the maintainers, and it points to the mismatch with the one-hour ephemeral caches that LiteLLM itself knows how to request via cache_control.ttl: "1h".

There is a nuance the issue does not capture and that makes diagnosis worse: in the default case the pin does not even last five minutes, because the key changes first.

`session_affinity`: the one to use

For conversations, this is the right piece and it is reasonably built. The session identifier is looked for in this exact order of headers:

x-litellm-trace-id
x-litellm-session-id
Any x-<vendor>-session-id, with the regular expression ^x-.+-session-id$ and a value of at least eight alphanumeric characters
Only if the User-Agent is Codex’s: session-id, session_id, thread-id, conversation_id
Plain x-session-id, for opencode

And if nothing matches, two fallbacks: a metadata.user_id in Anthropic format, and the W3C baggage with session.id=. The third point is the one that captures Claude Code with nothing configured.

Storage is a Lua script in Redis that does GET, SET NX EX and EXPIRE, so the first writer wins and the TTL is refreshed on every request that confirms the pin. It is an inactivity TTL, not a total session duration. Without Redis it degrades to a pod-local reservation, which is atomic but not shared.

A detail of the pipeline order that already came up in the previous article and still holds: the affinity checks run inside async_get_healthy_deployments before tag filtering. If the pin returns a single deployment and that deployment does not satisfy the request’s tag, the later filter removes it and the request fails. Affinity gets no vote on the tags.

The routers that do read the prompt, and why they are no good

LiteLLM has four pieces that read message content: AutoRouter with semantic matching, RequestComplexityRouter, AdaptiveRouter and QualityRouter. They all live in async_pre_routing_hook, which runs before looking for healthy deployments.

The reason they do not solve this problem is in what they choose. All four choose a model group, not a replica. They decide whether an easy question goes to the thirty-billion-parameter model or to the small one. None has access to the replica list or knows what is in each one’s memory.

What did land: admission control

A minor correction to the previous article, which placed this in the 1.101 branch as an upcoming feature. In 1.102.0 it is already there, in litellm/proxy/middleware/admission_control_middleware.py, with the three settings in general_settings: max_in_flight_requests_per_worker, max_queued_requests_per_worker and admission_queue_timeout_seconds, the last with a default value of 1.0 seconds.

It stays inactive as long as the first one is not declared. It exposes an authenticated /health/backlog and the three metrics litellm_admission_admitted_requests, litellm_admission_queued_requests and litellm_admission_rejected_requests_total, the last with a reason label. It is a per-process semaphore, not coordinated across pods.

What the engine exposes upwards

Before looking at the routers that do do this, you have to understand what raw material they work with. The natural question would be whether vLLM has an endpoint you can ask about a prefix. It does not.

The only cache router in the entry points is vllm/entrypoints/serve/dev/cache/api_router.py, and it only exposes destructive writes: POST /reset_prefix_cache, /reset_mm_cache and /reset_encoder_cache. All three are behind VLLM_SERVER_DEV_MODE=1 and do not exist in a normal deployment. There is no read.

What there is, is an event stream.

The ZMQ stream

It is enabled with --kv-events-config, and it is off by default. The configuration accepts a publisher (null or zmq), an endpoint (by default tcp://*:5557, which binds), an optional replay endpoint, the number of batches retained for that replay (10,000), and a high water mark limit of 100,000.

That limit is the one that matters operationally: above it, ZMQ discards events if the consumer does not keep up. Silently.

The transport is a PUB socket with msgpack serialisation and three-part frames: topic, eight-byte sequence number and payload. The sequence number lets a subscriber ask for replay from where it left off through the optional ROUTER socket. With data parallelism the port shifts by rank, so you have to subscribe to 5557+rank.

Three event types. BlockStored carries the hashes, the parent hash, the token identifiers, the block size, the LoRA adapter name, the extra keys, the medium (GPU, CPU or storage) and a session identifier. BlockRemoved carries only hashes. AllBlocksCleared carries no payload and is emitted when the prefix cache is reset.

They are published once per scheduler step, only if there are events. With the feature switched off the cost is zero. With it switched on, the full token identifiers of each block travel over the wire, so bandwidth scales with tokens per second and not with requests per second.

An example in vLLM’s own repository declares itself experimental on its eighth line.

The hash that has to be replicated

An exact router does not receive the engine’s state, it reconstructs it. For that it has to compute the same hashes, and the engine computes them like this:

# vllm/v1/core/kv_cache_utils.py
def hash_block_tokens(hash_function, parent_block_hash, curr_block_token_ids, extra_keys):
 if not parent_block_hash:
 parent_block_hash = NONE_HASH
 return hash_function((parent_block_hash, tuple(curr_block_token_ids), extra_keys))

It is chained: each hash identifies the complete prefix up to that boundary, not the isolated block. The default algorithm is SHA256, configurable with --prefix-caching-hash-algo among four values.

Five things a router has to get right for its hashes to match the engine’s:

The seed. Since 0.29, NONE_HASH is deterministic from a fixed seed, so two different processes produce the same hashes with nothing configured. It is a silent change of semantics from earlier versions, which required setting PYTHONHASHSEED. With the xxhash algorithms it is still needed.

The block size. By default 16 tokens, but the attention backend can override it if the user did not pass --block-size: 256 with sliding-window sparse attention, 64 with ROCm’s AITER, 64 or more with FlashAttention on XPU. A router must not assume 16, it has to read it from the event.

The hashing granularity, which is not necessarily the physical block size. There is --prefix-match-unit, which with several cache groups resolves to the greatest common divisor of the cacheable groups’ sizes.

The extra keys, in their order: LoRA adapter name (by name, not by identifier), multimodal identifiers with their relative offset inside the block, cache_salt only in block zero, and the hash of the prompt embeddings. Sampling parameters do not go in, which is correct: the prefix KV does not depend on the temperature.

And truncation. By default the hashes travel over the wire as 64-bit integers truncated from a SHA256, not as the full bytes. It can be disabled with VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=0. With the default value, a router has a theoretical collision risk that the engine does not have.

The hits are invisible by default

This is the detail that takes the most effort to discover and has the most consequences. The events are incremental: only what gets newly cached is announced. A reused block generates no event.

A router that only listens sees the additions and the removals, but does not see the hits. Its notion of what is recent and what is cold drifts away from the engine’s real LRU with nothing to warn it.

A full mode exists, and it is per request. sampling_params.extra_args["kv_cache_report_mode"] = "full" makes BlockStored events be emitted for blocks that were hits too. The function’s docstring says so plainly: it generates events so that external consumers, “for example a gateway”, know which blocks were reused. That is to say: for this to work, the router has to inject a parameter into every request it forwards.

The metrics, and what broke

For aggregate measurement two counters remain: vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, both in tokens and not in requests, with model and engine labels. There are two equivalents for the external connector, which allow a hit in HBM to be distinguished from one brought in from CPU or NVMe.

And there are two removals that break existing configurations. vllm:gpu_prefix_cache_hit_rate no longer exists: zero appearances in the repository. It was deprecated in 0.8, hidden in 0.9 and removed in 0.10. And vllm:gpu_cache_usage_perc was renamed to vllm:kv_cache_usage_perc, which breaks Grafana panels and autoscalers that depended on the old name.

The hit rate has to be derived in PromQL with rate() over the two counters. The one that appears in the text log is something else: it uses a window of the last thousand requests.

And it has a bias you need to know about before taking decisions with it. When a request is evicted, its blocks return to the free queue keeping the hash, so the later reprefill finds them again and counts as a hit. A high hit rate can mean useful reuse between users or it can mean the engine is evicting and recomputing in a loop. The internal statistics separate the two things with a preempted field, which is not exposed as a Prometheus metric.

The last token

A small detail with an effect on long conversations: the maximum hit length is the request’s token count minus one. There is always something to recompute in order to get logits. Since the reservation requires alignment to the block size, in practice that can force recomputing the whole final block. With blocks of 16 it is noise. With a backend that prefers 256, it is up to 256 tokens recomputed per turn.

The ones that do do it

vLLM production-stack

Eight routing logics in a single --routing-logic selector: roundrobin, session, kvaware, loadaware, prefixaware, disaggregated_prefill, disaggregated_prefill_orchestrated and priority. Two of them, loadaware and priority, are not in the Helm chart’s enum and are only enabled through extraArgs.

The naming trap is that prefixaware and kvaware are radically different things.

prefixaware is a hash trie in the router’s process. It chunks the prompt string every 128 characters, not tokens, and hashes each chunk with 64-bit xxhash. Its own docstring says so: the chunk size is “in number of characters”. That 128 is hardcoded, there is no flag or Helm value to change it.

Four consequences come out of that. --prefix-min-match-length is measured in characters and is quantised to multiples of 128. A shared prefix that differs by one character inside the first chunk breaks the whole match, because there is no partial match inside a chunk. For chat, it concatenates the content of all the messages with newlines and without applying the chat template, so the indexed string is not what the engine tokenises. And the tie-break between tied replicas is random.choice, without looking at load.

The trie has no notion of saturation. A popular prefix concentrates all the traffic on the same replica indefinitely. The docstring assumes the simplified model and declares it: “we assume there is no prefix cache eviction”.

It has no pruning either. HashTrie only has insertion and longest prefix lookup; there is no deletion method. When a replica disappears, its URLs stay forever in every node’s sets, and filtering happens at query time. It works, and the trie grows monotonically: each distinct prompt adds its length divided by 128 permanent nodes, with no bound, no TTL and no LRU. The router’s memory limit in the chart is 1,000 MiB. By contrast, the round-robin router in the same file does bound its caches to 1,024 entries.

And it dies on every restart: it is in-process memory state, with no snapshot and no initial load.

kvaware and loadaware are better and have a different price. They do not index themselves: they ask the LMCache controller, which knows about real tokens. That requires lmcache==0.3.11 and vllm==0.13.0 pinned exactly, and the official router image drags in the whole of vLLM. The limitation to read before planning anything: they only look at request_json.get("prompt", ""), with an open TODO for chat completions. With /v1/chat/completions, both tokenise an empty string.

loadaware is the only one of the three that breaks affinity on saturation, with the formula score = relative_match - beta * relative_load and beta = 1.0 by default. The authors’ own reading: a replica at twice the average load loses the equivalent of a full cache hit. The guide suggests 0.25 to favour cache and 2.0 to favour balancing.

On published results, the documentation’s benchmarking file literally says that the measurement platform is coming soon. The project blog’s two posts claim between three and ten times lower response latency, and only one declares a partial configuration: Llama 3.1 70B on four nodes with tensor parallelism 2 over 80 GB A100s, inputs of 9K tokens and outputs of 10, against AIBrix 0.2.0 and a flat deployment. Charts with no labelled points, no absolute figures and no declaration of which routing logic was used in each.

llm-d

There has been a reorganisation here that invalidates a good part of the third-party documentation. The Endpoint Picker code left the Gateway API Inference Extension repository and was merged into llm-d’s, which has been renamed to llm-d Router. The Kubernetes project keeps the InferencePool API and a lightweight reference EPP that does nothing about prefixes. If you look for KV-aware routing in the extension’s repository, today it is not there.

The InferencePool is at v1 and is GA. The other two APIs moved to the llm-d.ai group at v1alpha2, and when they coexist with the old ones the EPP prefers the new ones and ignores the old.

Scoring is a weighted sum. The default weights, when no configuration is passed: queue 2, KV utilisation 2, prefix 3. Any scorer referenced without a weight counts as 1.0. There is no configuration through environment variables: the old ENABLE_*_SCORER no longer exist, everything goes in an EndpointPickerConfig in YAML.

The prefix scorer consumes an attribute published by a producer, and there are two producers.

The approximate one computes the hashes in the EPP itself without talking to the engine, and keeps a local LRU per pod and block pair that is populated after the decision. It is an approximation by construction. It has a hard floor: a configured block size below 64 tokens is raised to 64 at request time, because the index stores one entry per pod and block and going down to 16 would multiply memory by four.

The exact one builds the real index from the engine’s ZMQ events, with one subscriber per pod that is installed and torn down with the endpoint’s registration and deregistration. It has adapters for vLLM and SGLang, deduplicates evictions by reference counting, and the replay buffer that allows the index to be rebuilt after an EPP restart requires vLLM 0.26.0 or above.

The trap is in the tokeniser. The EPP tokenises through a plugin, and the default backend is estimate: byte packing with no tokeniser, roughly 4 bytes per token, with identifiers that do not correspond to real engine tokens. The exact scorer needs real tokens, and for that you have to explicitly configure the vllm backend, which does HTTP against the engine’s /v1/completions/render endpoints. If it is omitted, the auto-created estimate satisfies the dependency and the system degrades silently.

On maturity, the file itself declares it: all in-tree plugins are in alpha or beta, and they will be promoted to stable when the project approaches 1.0. An alpha plugin configured without the corresponding flag makes the EPP fail to start.

And on replicas, the warning is explicit: active-active mode should be avoided with approximate prefix routing, because the EPP replicas do not share state and each one only sees the traffic it has served itself, which significantly degrades the hit rate. The exact index is safe in high availability, and even so the reference guides fix a single replica because the accounting of in-flight requests is local to the process.

The cost is the best documented in the whole field, and it deserves a table. Simulator with Qwen3 8B, 100 pods, 100K input tokens:

Requests/s	Cap on tokens to match	Peak CPU	Peak memory
5.0	4,096	1.19 cores	0.26 GiB
5.0	100,000	3.82 cores	0.65 GiB
98.7	4,096	35.17 cores	2.46 GiB
98.8	100,000	46.50 cores	3.41 GiB

With outputs of 10K tokens at 50 requests per second, 32.53 cores and 12.54 GiB. The maintainers’ sizing rule is half a core to one core per request per second on large agentic loads. And idle consumption scales with the number of pods because of continuous scraping: with 100 pods, about 7.5 cores with no traffic.

NVIDIA Dynamo

A precise event-driven index, with a global radix tree in each frontend’s process. Since all the replicas consume the same event plane, there is no need to synchronise routers with each other for the prefix state.

The cost function is explicit and weighted. It discounts from the prefill the blocks that overlap in GPU, host and disk with different weights (1.0 for the overlap credit, 0.75 for host, 0.25 for disk), and adds the potential decode blocks and the active requests.

There is a parameter to look at before deploying: --router-kv-overlap-score-credit-decay is at 0 by default, that is, disabled. Its description says what it is for: preventing loaded, cache-rich replicas from winning again and again while the freshly autoscaled ones receive too little traffic. With the default value, a new pod has no cache, therefore never wins the scoring, therefore never generates cache.

The approximate mode exists (--no-router-kv-events, with a TTL of 120 seconds) and the documentation says it is not the recommended route in production.

Dynamo is the most honest project in its measurement methodology: its comparison guide recommends contrasting --router-mode random against kv over the public Mooncake trace from FAST'25, with a declared cache ratio of 59 %, and it generates the table locally instead of publishing a number.

The rest of the field

SGLang has renamed its router to sgl-model-gateway, version 0.3.2 within SGLang 0.5.19. Its cache_aware policy is an approximate radix tree over raw characters, with switching to shortest queue when the system is unbalanced according to two combined thresholds. One detail that can cost an afternoon: the Rust defaults and the Python CLI defaults do not match, and on the maximum tree size they differ by four orders of magnitude (10,000 nodes against 2^26).

AIBrix 0.7.0 offers a standard mode with a local hash table and a KV event synchronisation mode behind a flag and a build tag, with vLLM 0.7.0 or above.

Ray Serve LLM has PrefixCacheAffinityRouter with an explicit alpha API warning, a character-based prefix tree in a decoupled actor, and its imbalance threshold at infinity by default, that is, out of the box it never breaks affinity on load.

Envoy AI Gateway 1.1.0 implements none of this and delegates to InferencePool and the EPP. KServe the same, with its LLMInferenceService at v1alpha1. Mooncake Conductor is still an unimplemented proposal.

Comparison table

Project	Version and date	Index	Where it lives	Kubernetes	Declared maturity
production-stack `prefixaware`	main branch, 09-Sep-2026	Approximate, character trie, no eviction	Router process	Optional	WIP in the README
production-stack `kvaware`/`loadaware`	same	Exact via LMCache controller	LMCache controller	Optional	No chat completions
llm-d Router approximate	v0.10.0, 17-Aug-2026	Approximate, LRU per pod	EPP process	Yes	Beta, active-active discouraged
llm-d Router exact	same	Exact through ZMQ events	EPP process	Yes	Beta
NVIDIA Dynamo	v1.4.2, 27-Aug-2026	Exact, event-driven radix tree	Frontend process	No	GA with experimental subsystems
SGLang `sgl-model-gateway`	0.3.2, 03-Sep-2026	Approximate, character radix	Gateway process	No	Default policy
AIBrix	v0.7.0, 16-Jun-2026	Both	Gateway plugin	Yes	Synchronisation behind a flag
Ray Serve LLM	Ray 2.58	Approximate, characters	Ray actor	No	Declared alpha
Envoy AI Gateway	v1.1.0, 21-Aug-2026	None of its own	Delegates to EPP	Yes	Stable 1.x API

How much it really improves

Here two types of source have to be separated.

The projects’ own numbers. Red Hat published the llm-d case in May: Qwen3 32B, eight vLLM pods over 16 H100s with tensor parallelism 2, synthetic shared-prefix load with 150 groups of five prompts, 6,000 tokens of system prompt, 1,200 of question and 1,000 of output. Result: up to 109 % more throughput and up to 99 % less TTFT, around 200 concurrent users within SLO against around 20. The repository’s result files give more detail: peak throughput from 6,986 to 14,892 output tokens per second, p90 TTFT from 135.5 seconds to 0.26.

That same file publishes the regression, and that is why it is worth reading: p50 inter-token latency rises by 22.4 % with vLLM and by 45.4 % with SGLang. The trade-off is declared: routing by affinity concentrates more concurrent work on the pods with a warm cache.

It is the best possible case. A synthetic shared prefix and a reference that is a flat Kubernetes Service.

The independent measurements. There are three and they say different things.

CacheRoute, from Meta, August 2026, is the most complete. Llama 3.3 70B in fp8, 30 targets with tensor parallelism 2 over 60 H100s, a semi-synthetic trace from production telemetry with 128,824 business keys, and all the alternatives reimplemented in the same harness instead of comparing against a dumb balancer. It obtains 176 requests per second with a p99 SLO of 3.5 seconds against 76 for the best competitor, and a hit rate of 93.2 % against 72.0 %.

And it publishes two negative findings that nobody else publishes. There are load regimes where affinity reduces capacity to between half and two thirds of the reference, to the point that the authors make a mirrored-traffic trial mandatory before deploying. And growing the fleet can cool a prefix even though total cache capacity grows, because the time between revisits stretches beyond the eviction window. Scaling horizontally can make the hit rate worse.

GORGO, June 2026, measures on a multi-region deployment with a dataset where intra-user prefix reuse is 89.4 % and the average prompt is almost 18,000 tokens. An improvement of 6.9 to 15.5 % in p95 TTFT. With extremely high reuse, the improvement is single-digit, because network latency and queuing dominate.

LAAR, from IBM Research Tokyo, April 2026, is the uncomfortable counterpoint: in its comparison, session affinity routing was the worst of the baselines, and at 64K contexts a load-based routing gives lower absolute latency at the cost of fewer hits.

The joint reading fits in three conditions. The big improvements require high, concentrated prefix reuse, long prompts, and a weak reference. Take away any of the three and the numbers deflate.

The five failure modes

Hot spot from affinity. The busiest bucket inherits the load’s skew, and tail latency starts following the hottest target instead of the fleet average. Mitigations exist in the four main products and in three of the four they ship switched off: Dynamo’s credit decay at 0, Ray’s imbalance threshold at infinity, and production-stack’s prefixaware with none at all.

Starvation of the freshly scaled pod. A new pod has no cache, so it never wins the scoring, so it never warms up. Dynamo documents it as the reason for the decay parameter. I found no primary documentation from any of the projects about the specific interaction with KEDA.

Cost of the index. The llm-d numbers in the previous table are the best public reference that exists. Raising the cap on tokens to match from 16,384 to 400,000 can more than double the EPP’s CPU with little traffic.

Traffic with unique prefixes. Each project degrades to something different. SGLang routes to the smallest tree, which is not the same as the least loaded. Ray falls back to power of two choices. Dynamo has an explicit preset, --load-aware, to say “I want the load model without reuse”. And production-stack’s prefixaware keeps inserting into the trie, that is, keeps paying the cost while receiving nothing.

Sibling bursts. Parallel sampling, best of N, or an agent that opens five branches at once. With event-driven routing, no engine has yet emitted the block stored event, all the siblings score zero overlap, and the prefix is precomputed on every replica. Dynamo has a prediction TTL for this; the others do not.

The architecture that comes out of all of the above

Two layers, with the responsibilities separated.

On top, LiteLLM, with what it does do well: non-forgeable identity through the virtual key, budgets, spend logging, concurrency limits, fallback between providers and API unification. For the gateway, each model group has a single destination: the URL of the router below.

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://kv-router.inferencia.svc:8000/v1
 api_key: os.environ/VLLM_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 1
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

Below, the KV-aware router, inside the engine’s domain, talking to the vLLM replicas and consuming their events.

Four consequences of this separation that have to be accepted before building it.

Per-replica visibility in LiteLLM is lost. For the gateway there is a single deployment, so its cooldowns, its per-deployment counters and its strategies stop meaning anything. Observability of which replica served what goes downstairs.

Retries stack up. The client SDK retries, LiteLLM retries, and the router below may have its own failover. In the previous article 45 calls per turn came out with three layers; with four it is worse. Lowering num_retries in the gateway to 1, or to 0, is the sensible decision when there is a router below that already retries.

LiteLLM’s session affinity stops making sense for the KV, because it no longer chooses a replica. It is kept if it is used for something else, such as pinning a conversation to a specific provider.

And the choice of the router below conditions the rest of the cluster far more than the gateway’s does. llm-d requires Kubernetes 1.32 or above, Gateway API, the inference extension and a conformant Gateway. Dynamo does not require Kubernetes. production-stack has a Helm chart and works outside it. That decision is the expensive one, not the gateway’s.

When not to build this

Three cases where the correct answer is not to do it, at least not yet.

A single replica per model. With one replica, the engine’s prefix cache already does all the work and there is nothing to route. That is the case for a good part of the four H100 deployments with one large model and tensor parallelism.

Low or scattered prefix reuse. Before building anything, the measurement is rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) per replica, with a mental correction for the bias from reprefills after eviction. If it is already high with blind distribution, reuse is concentrated inside each conversation, and there LiteLLM’s session affinity costs one line of configuration and resolves the case.

Traffic where the network dominates. The GORGO result is the warning: with enormous prompts and a distributed deployment, the improvement is eaten up by queuing and network latency.

The case where it does pay off is specific: several replicas of the same model, a large system prompt shared between different users, or a RAG with a corpus of documents that repeat across requests. There the prefill saved is real and it is most of the work.

Checklist

Measure the current hit rate per replica with the two vLLM counters before touching anything, and check whether the engine is evicting with vllm:num_preemptions_total.
Check that prefix caching is switched on. It is no longer --enable-prefix-caching that switches it on: the default value is derived from the model and the log of the decision is at DEBUG level. What is actionable today is --no-enable-prefix-caching to switch it off.
Audit the prompts before routing: a timestamp or a session identifier at the start of the system prompt breaks the match from block zero and no routing layer fixes it.
If the KV event stream is enabled, size the bandwidth by tokens per second and watch discarding at the high water mark. With data parallelism, subscribe to one port per rank.
If an exact router is chosen, verify the vLLM version against the one the replay buffer requires.
If llm-d is chosen, explicitly configure the vllm tokenisation backend. The default estimate does not fail, it degrades.
If production-stack is chosen with kvaware or loadaware, check first whether the traffic goes through /v1/chat/completions, because today it does not work there.
Review the load-based affinity-breaking parameter, because in three of the four main projects it ships disabled.
Lower num_retries in LiteLLM when there is a router below that already does failover.
Trial with mirrored traffic before changing production. It is the explicit recommendation of the only independent measurement that went looking for bad regimes, and found them.

Traps and things that are not what they seem

LiteLLM has no prefix routing in any version. Neither the open one nor the paid one. I got it wrong in June.
prompt_caching does nothing if the client does not send cache_control, and with the default automatic injection it fails on every turn because it marks the last message.
That pin’s TTL is 300 hardcoded seconds, incompatible with the one-hour caches that LiteLLM itself knows how to request.
production-stack’s prefixaware indexes characters, not tokens, in non-configurable chunks of 128, and its trie is never purged and models no eviction.
production-stack’s kvaware and loadaware do not support chat completions. They tokenise an empty string.
The default tokeniser in llm-d’s EPP is a byte estimate, and with it the exact scorer degrades silently.
llm-d’s active-active mode is discouraged with approximate prefixes, and the reference guides fix one replica even with the exact one.
Dynamo’s overlap credit decay ships at zero, which can leave a freshly autoscaled replica with no traffic.
The SGLang router’s default values differ between the Rust and the Python CLI, by as much as four orders of magnitude.
vllm:gpu_prefix_cache_hit_rate no longer exists and vllm:gpu_cache_usage_perc is now called vllm:kv_cache_usage_perc. Old panels and autoscalers are measuring nothing.
The hit rate counts the reprefill after eviction as a hit. A high number can be reuse or it can be thrashing.
KV events do not publish the hits by default, only the additions, unless full mode is requested request by request.
The hashes travel truncated to 64 bits unless it is disabled with an environment variable.
The default block size is 16, and the attention backend can change it without anyone asking.
The Endpoint Picker left the Gateway API Inference Extension repository. Looking for it there today leads nowhere.

Closing

June’s error has a reading that goes beyond the erratum. It came from reading a product’s documentation and not its code, and from the fact that LiteLLM’s documentation uses the word “caching” in a pre-call check that does something else. The distance between what a name suggests and what a function does is, at this point in the track, the most repeated pattern of the whole series.

The architectural conclusion is that KV-aware routing is not a gateway function. It is not one today in LiteLLM and probably should not be: it requires keeping an index synchronised with each engine’s memory, replicating its hash function, consuming an event stream and sizing CPU per request per second. That belongs to the engine’s domain, not to that of identity and budget.

What remains is a two-part decision. The first is whether the traffic deserves it, and that is answered with two Prometheus counters and without installing anything. The second is which of the three routers fits the cluster you already have, and there the deciding variable is not the hit rate but whether you are willing to bring in Gateway API and the inference extension, or to pin vLLM 0.13.0, or to operate a Dynamo frontend.

And the honesty the material demands: the only independent measurement that specifically went looking for regimes where this makes things worse found them, and they are not marginal. Half to two thirds of reference capacity is a serious regression. That no project publishes that side of the curve does not mean it does not exist in theirs.

Sources

LiteLLM, Router - Load Balancing (estrategias, comprobaciones previas, afinidad): https://docs.litellm.ai/docs/routing.
LiteLLM, código: litellm/router.py, litellm/router_utils/prompt_caching_cache.py, litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/litellm_pre_call_utils.py, litellm/proxy/middleware/admission_control_middleware.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
vLLM, código: vllm/v1/core/kv_cache_utils.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_manager.py, vllm/distributed/kv_events.py, vllm/config/kv_events.py, vllm/v1/metrics/loggers.py, vllm/v1/metrics/stats.py: https://github.com/vllm-project/vllm.
vLLM, KV offloading usage (niveles CPU, filesystem y objeto; compartición entre procesos): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html.
vLLM, notas de la versión 0.29.0 (semilla determinista, seguimiento de propiedad en eventos, flags de admisión): https://github.com/vllm-project/vllm/releases/tag/v0.29.0.
vLLM, RFC 16669, publicación de bloques KV y métricas: https://github.com/vllm-project/vllm/issues/16669.
vLLM production-stack, código: src/vllm_router/routers/routing_logic.py, src/vllm_router/prefix/hashtrie.py, src/vllm_router/parsers/parser.py: https://github.com/vllm-project/production-stack.
LMCache Lab, resultados de production-stack frente a AIBrix (parte interesada): https://blog.lmcache.ai/en/2025/03/06/open-source-llm-inference-cluster-performing-10x-faster-than-sota-oss-solution/.
llm-d Router, código y documentación de operaciones (pesos por defecto, dimensionamiento del EPP, aviso de activo-activo): https://github.com/llm-d/llm-d-router.
llm-d, guía de enrutado preciso por prefijo y resultados publicados (parte interesada): https://github.com/llm-d/llm-d.
llm-d Router, incidencia 1290, estado de prefijo no compartido entre réplicas del EPP: https://github.com/llm-d/llm-d-router/issues/1290.
Red Hat, Same 16 GPUs, twice the users (parte interesada): https://www.redhat.com/en/blog/same-16-gpus-twice-users-inference-aware-routing-llm-clusters.
Kubernetes SIG Network, Gateway API Inference Extension (traslado del EPP y del BBR): https://github.com/kubernetes-sigs/gateway-api-inference-extension.
NVIDIA Dynamo, diseño y ajuste del router KV: https://github.com/ai-dynamo/dynamo.
SGLang, sgl-model-gateway: https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway.
AIBrix, enrutado por caché de prefijo: https://github.com/vllm-project/aibrix/blob/main/pkg/plugins/gateway/algorithms/prefix_cache_readme.md.
Ray Serve LLM, enrutado consciente de prefijo (API en alfa): https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html.
Huang Cheng (Meta), CacheRoute, agosto de 2026: https://arxiv.org/html/2608.19677.
Yuan et al., DualMap, febrero de 2026: https://arxiv.org/abs/2602.06502.
Ricci Toniolo et al., GORGO, junio de 2026: https://arxiv.org/html/2602.11688.
Yoshimura, Chiba y van de Beek, Accuracy Is Speed (EuroMLSys ‘26), abril de 2026: https://arxiv.org/html/2604.15732v1.

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

Thu, 10 Sep 2026 08:00:00 +0200

Fourth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability and virtual keys covered governance. Here comes the new tenant: agents, which speak the same protocol as humans and behave in a different way. Everything that follows is verified against LiteLLM 1.100.1 and vLLM 0.29.0, both from the first ten days of September 2026.

TL;DR

The day a team connects an agent to the corporate gateway, the platform changes regime without anyone touching a YAML. These are the seven conclusions.

LiteLLM’s priority parameter prioritises nothing under normal conditions. It is still marked as beta and its queue is only consulted when all the deployments in the group are in cooldown. With free capacity, the only thing it produces is a response header. On top of that, the success branch does not pop the item off the queue, so the list grows indefinitely; and on /chat/completions the value 0, which is the one used by all the documentation examples, is falsy in Python and disables queuing.

Separating traffic is done with two pools, not with priorities. The non-forgeable way is model_group_alias at key or team level: the client always asks for qwen-30b and the credential decides whether that resolves to qwen-30b-interactivo or to qwen-30b-agentes. The way that does not touch clients is tag_regex against the User-Agent, with a pattern like ^User-Agent: claude-code\/, which serves to classify load and not to control access, because that header is set by the client.

The useful admission control arrives in the 1.101 branch, with max_in_flight_requests_per_worker and a per-process queue that returns 503 with retry-after. It rejects before authenticating, so what it discards does not appear attributed to any key.

max_parallel_requests is only applied at virtual key level. The team, user, end customer and organisation endpoints accept the field and store it in the database, and nobody reads it. global_max_parallel_requests is a no-op unless the old limiter is reactivated, and general_settings.max_parallel_requests is read by no code path.

vLLM with --scheduling-policy priority does not push ahead anyone who is already generating. The only thing that changes is who it evicts when KV cache runs short. If the agents have filled max_num_seqs, a chat with the highest priority waits just the same. And the two knobs that in 2025 allowed short prompts to get ahead of long ones, max_num_partial_prefills and max_long_partial_prefills, disappeared from the code in 0.27.

The retry product reaches 45 calls per agent turn. Three from the client SDK times three from the router times up to five fallback groups. The proxy sets max_retries=0 on the SDK it uses itself, but it cannot touch the client’s. The timeout is per attempt, not per request, and LiteLLM’s Timeout comes out as a 408, which is retryable.

A single 429 takes a deployment out of the pool for five seconds, without going through the allowed_fails threshold. With an agentic burst against a fleet of four replicas, the whole pool enters cooldown and the gateway returns a 429 of its own to everybody, humans included.

You are here: the gateway layer with two classes of traffic

In the seven-layer stack this still lives in the gateway layer, with one difference compared to the previous three articles. There the variable was load: more requests, more teams, more budget to share out. Here the variable is the shape of the load.

A gateway sized for three hundred engineers using a chat works. The same gateway, with thirty of those engineers running coding agents, serves a fraction of the human traffic with the previous latency and the rest degrades. The GPUs stay just as busy, the utilisation metrics stay good, and the support team gets complaints from people waiting nine seconds for a paragraph.

The analogy: the registry desk and the courier with two hundred files

A registry office serves people in order of arrival. Citizens come in one at a time, present a piece of paper, and leave. The system is fair and the queue moves.

One Tuesday a courier turns up with a trolley of two hundred files. He joins the queue like anyone else, because rules are rules, and when his turn comes he occupies the desk for forty minutes. Behind him are twelve people with one piece of paper each. Nobody has broken any rule. The courier has the same right as everyone else and uses the same procedure. What fails is that the office was designed assuming a certain size of transaction and another one three orders of magnitude bigger has turned up.

The real solutions for an office are three, and all three have an exact equivalent in the platform. Open a desk for bulk deliveries, which is separating pools. Put up a capacity sign and send people back later, which is admission control with its 503. And say that files are handed in during the afternoon, which is the part that on an inference platform can almost never be applied, because the agent does not negotiate schedules.

What does not work in the office is giving the citizen a priority number when the courier is already at the desk. That number does not interrupt the transaction in progress. It is, literally, what priority does in the two layers of the platform, and it is worth keeping in mind while reading the next section.

The two kinds of traffic, measured

Before configuring anything, you have to be able to state the difference in numbers, because every later adjustment comes out of it.

Trait	Human chat	Agentic loop
Input per request	hundreds of tokens	tens of thousands, and it grows with each turn
Output per request	hundreds of tokens	from two tokens (a tool call) to several thousand
Concurrency per user	1	3 to 20 requests in flight
The metric that hurts	time to first token	total turn throughput
Tolerance for waiting	seconds	minutes
Cancelled requests	rare	routine, by the agent’s own decision
Retries	the browser’s	the SDK’s, plus the loop’s
Witness when it fails	the user	nobody

Two rows deserve comment because they contradict intuition.

The first is the metric row. An agent does not perceive time to first token, so on a shared engine it is the natural candidate to give up that metric. The operational conclusion is that the agent pool can be configured for aggressive throughput, with large batches and deep queues, while the interactive pool is configured for the opposite.

The second is the cancellations row. An agent that launches three speculative continuations and discards two generates two aborted streams per turn. At the gateway, that touches spend accounting, the release of concurrency counters and the open connection to the engine. LiteLLM 1.100 handles it reasonably well, and it did not six months ago.

What does not work: `priority` in LiteLLM

The first reaction of any operator is to look for a priority field. LiteLLM has one, documents it on its Request Prioritization page, and that page carries the beta label and the sentence saying it is for testing. The label is correct and has to be taken literally, because behind it there are four things.

The queue is only consulted when the whole group is in cooldown. The scheduler’s poll() method returns true if there is any healthy deployment, without touching the queue. With available capacity, which is the normal situation, the request goes straight through and priority orders nothing. The only thing that changes is that the response carries the x-litellm-request-prioritization-used header.

The queue does not empty. In that success branch there is no heappop and no call to the cleanup, which only happens in the expiry branch. The list under the key scheduler:queue:{model} grows without bound, is serialised whole into Redis on every insertion, and there is no maximum size and no discard policy. When the group does enter cooldown, the head of the heap is occupied by fossil entries from requests that finished hours ago, and new ones compare their identifier against that head, never match, and wait until the timeout runs out.

On /chat/completions, the highest priority disables the feature. The code reads the value with a kwargs.get("priority") or self.default_priority, and in Python 0 or None is None. Since in this scheduler the lowest number is the highest priority, priority: 0, which is the value in the documentation’s own examples, goes down the path without queuing. On /completions the check is done with is not None and it works; the two routes do not behave the same.

And default_priority in router_settings breaks the request. If the client does not send a priority but the global setting is in place, the value is truthy, the scheduling branch is entered and a function is invoked that requires the positional argument nobody has injected. The setting has no associated test and does not appear in the router_settings reference.

There is also a /queue/chat/completions, marked as experimental and hidden from the OpenAPI schema, which reads data["priority"] with no default value: a request without that field raises an exception that comes out as a 400 authentication error, with a message that has no relation to the cause.

There is a priority feature that is solid, and it is a different thing. The v3 dynamic limiter shares out token and request per minute quota between classes with weights, through litellm.priority_reservation with values like {"premium": 0.75, "standard": 0.25}. It shares out capacity, it does not order a queue: below the saturation threshold it lends free capacity and above it applies the weights. It never makes anyone wait, either it passes or it returns a 429. It requires Postgres and an enterprise licence, and it is still marked as beta.

What does work: two pools

Effective separation is done earlier, by deciding which set of replicas each class of traffic goes to. LiteLLM offers two mechanisms with very different security properties.

By credential: `model_group_alias` at key or team level

Router settings are resolved in the order key, team, global. Among those that accept that override is model_group_alias, and there lies the piece:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 model_info:
 id: chat-01
 max_input_tokens: 32768
 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 model_info:
 id: agentes-01
 max_input_tokens: 131072

And on the agent team’s key:

curl -X POST "$PROXY/key/generate" \
 -H "Authorization: Bearer $MASTER_KEY" \
 -d '{
 "team_id": "plataforma-agentes",
 "models": ["qwen-30b"],
 "router_settings": {
 "model_group_alias": {"qwen-30b": "qwen-30b-agentes"}
 }
 }'

The client asks for qwen-30b and knows nothing about any of it. The resolution is tied to the credential, so the caller cannot get around it by changing a header. The same list allows routing_strategy, fallbacks, context_window_fallbacks, retry_policy, cooldown_time and allowed_fails to be set per key or team, which gives different reliability policies for each class of traffic on the same proxy instance.

It should not be confused with access groups (model_info.access_groups), which are permissions and not routing: they decide whether a key can invoke a model name, not which replica the request goes to.

By `User-Agent`: `tag_regex`

When separate credentials cannot be issued, because the agentic client uses the same personal key as that person’s chat, LiteLLM classifies by regular expression against the request headers:

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 tag_regex: ["^User-Agent: claude-code\\/", "^User-Agent: .*codex"]
 model_info: {id: agentes-01}
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 tags: ["default"]
 model_info: {id: chat-01}

router_settings:
 enable_tag_filtering: true
 tag_filtering_match_any: true

The proxy composes strings of the form User-Agent: <value> and tests them with re.search. Exact tag matching takes precedence over the regex. The documentation itself carries a warning that has to be respected: the User-Agent is written by the client, so this is traffic classification and not a security boundary. As load sharing it is enough, because an agent that lies about its identity to sneak into the interactive pool is a governance problem, not a routing one.

The behaviour when no deployment carries the requested tag has a nuance that comes as a surprise in production. If the tag is unknown to the group, the request falls through to the pool marked with tags: ["default"]. If the tag exists in the group but no healthy deployment carries it at that moment, the request fails with the tag configuration error, even though a default pool is available. To get deterministic fallthrough to the default pool you have to set model_info.allow_fail_open: true.

A different strategy per pool with `routing_groups`

router_settings:
 routing_strategy: simple-shuffle
 routing_groups:
 - group_name: interactivo
 models: [qwen-30b-interactivo]
 routing_strategy: simple-shuffle
 - group_name: agentes
 models: [qwen-30b-agentes]
 routing_strategy: simple-shuffle

The recommendation to leave simple-shuffle on both is not laziness, and sustaining it requires looking at what the alternatives do on a homogeneous vLLM fleet.

least-busy keeps a complete dictionary under a single key and updates it with non-atomic reads and writes, without using INCR, so with several proxy replicas increments are lost. It has no TTL, so a request that never completes leaves the counter raised forever and that replica is permanently excluded. Counters can go negative. And when the minimum corresponds to an identifier that is no longer in the healthy list, it falls back to random.choice without warning. The pattern of requests that do not complete is exactly that of an agent that cancels.

latency-based-routing measures seconds per output token, not wall-clock latency. A chat turn of twenty tokens looks extremely slow by that metric and an agent generation of two thousand tokens looks extremely fast, so mixing both kinds of traffic in the same group poisons the signal. It also seeds new deployments with zero latency, and with the buffer defaulting to zero, a freshly started replica takes all the traffic until its first success callback.

usage-based-routing-v2 uses minute windows that reset all at once, and it accounts for tokens when the request completes, so long generations in flight are invisible to the router. With long outputs, it systematically underestimates the real load.

cost-based-routing sorts by the sum of the input and output unit prices and takes the first, with no random tie-break. With replicas of the same model at the same price, all the traffic always goes to the same place. And a model that is not in the price map gets a default cost of 5.0 per token, a figure designed to deprioritise it.

Admission control: the new piece in the 1.101 branch

When the agent pool saturates, what you want is to reject fast and with a code the client understands, not to accumulate requests inside the proxy until memory runs out. That arrives with per-worker admission control, added in the 1.101 branch, in release candidate as this is written.

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

It is a semaphore with a per-uvicorn-process queue, without Redis. When the queue fills or the wait expires it returns 503 with retry-after: 1 and a body with "type": "overloaded_error". It is observed through /health/backlog, which exposes requests in flight, admitted, queued and rejected, and through the Prometheus litellm_admission_* metrics, whose rejection counter distinguishes queue_full from queue_timeout.

Two properties have to go into the runbook. The middleware rejects before authenticating, so what is discarded is not attributed to any key in the spend records, and the real number of requests rejected per team cannot be reconstructed from the spend table. And the limits are read on the first request: changing them requires a restart.

Concurrency per key, and the three settings that do nothing

max_parallel_requests is the right brake for a specific agent, and its configuration surface is full of dead ends.

The only level where it is applied is the virtual key. The v3 limiter, active by default since 1.94, has descriptors for user, team, team member, end customer, organisation, model per key, tag and agent, and all of them carry only request and token counters per window. The only reference to max_parallel_requests is the one in the key descriptor. The old limiter says so in a comment: supporting it for model, user and team is pending.

At the same time, /team/new, /user/new, /customer/new and the organisation endpoints accept the field, validate it and write it to Postgres, and the documentation shows it in their examples. Nobody reads it afterwards. A team with max_parallel_requests: 20 in its row has no concurrency limit at all.

global_max_parallel_requests is injected into the request metadata and is only consulted by the old limiter, which is disabled unless LEGACY_MULTI_INSTANCE_RATE_LIMITING=true is set. The server settings documentation still describes it as a global limit coordinated through Redis. And general_settings.max_parallel_requests does not appear in any execution path: its only two appearances are the copy from the database and the type schema of the interface.

What does work, in detail:

curl -X POST "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" \
 -d '{"team_id":"plataforma-agentes","max_parallel_requests":8,"rpm_limit":600}'

Counting is done with a sorted set in Redis scored by Redis’s own clock, with Lua scripts to acquire and release. Each slot expires after 3,600 seconds, which is at once the maximum request duration the meter can track and the time a leaked slot takes to heal by itself. Release is hooked to success, to failure and to client disconnection during streaming, which was the classic leak with clients that cancel.

One live leak remains and it affects exactly the case in this article: every tool call through the MCP gateway acquires a slot and does not release it. The MCP route invokes the pre-call hook, which acquires, and none of the release functions is called from that module. With an agent that makes twenty tool calls per turn and a key with eight slots, the 429 arrives on the first turn and lasts an hour.

The order of evaluation matters for interpreting 429s. First the window counters, requests and tokens per minute, are checked, and only if they pass is the concurrency slot acquired. The code comment explains why: the other way round, every rate limit rejection would leave an orphan slot. In practice, with a low rpm you always see the requests type 429, and the max_parallel_requests one only appears when the relationship between concurrency and duration makes it binding first. The message distinguishes the two cases with the Limit type: field and the rate_limit_type header.

A minor detail with consequences on the client: the response to a concurrency rejection carries retry-after and reset_at computed from the window size, sixty seconds by default, when a concurrency meter has no window. A well-behaved client that respects that header will wait a minute when it might have had a free slot in two seconds.

Session affinity: the KV cache that gets thrown away

An agent turn resends the whole conversation context, system prompt and tools included. If that request lands on a different replica from the previous turn’s, vLLM’s prefix cache does not have the prefix and the prefill is recomputed in full. With twenty turns and four replicas balanced blindly, the long prefill is paid for fifteen times over.

LiteLLM has two mechanisms for this and they are not equally good.

The prompt_caching check computes a SHA-256 of the cacheable prefix, defined as everything up to the last block with cache_control, and pins the deployment with that hash. It has two problems. The TTL is hardcoded to 300 seconds in two places in the code, with no way to configure it, something which is reported and open; with one-hour caches, affinity is lost after five minutes of inactivity. And the key is an exact hash, so a client that moves the cut point forward on every turn, which is exactly what agents do to extend the cached prefix, generates a new key every time.

The session_affinity check is the one to use:

router_settings:
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

It pins the deployment with an atomic Lua script in Redis, with a configurable TTL that defaults to one hour and that is refreshed on every request, so it bounds the idle time between turns and not the duration of the conversation. If the pinned deployment enters cooldown, the request goes out through the normal strategy and keeps the pin so it can come back when it heals.

The session identifier comes from a precedence chain that starts with x-litellm-trace-id, continues with x-litellm-session-id and then accepts any header of the form x-<something>-session-id whose value looks like an identifier. That includes the session header Claude Code already sends, with nothing configured on the client. Whatever matches populates both the session identifier and the trace identifier, which links this to the correlation from the first article in the track.

One dangerous interaction when combining it with tags: affinity is evaluated before tag-based routing. If the pin reduces the list to a deployment that does not satisfy the request’s tag, the request fails instead of falling through to the default pool, unless allow_fail_open has been enabled. Retagging a pool with live sessions breaks them.

And a clarification that saves a pointless search: LiteLLM does not do prefix routing. There is no prefix tree and nothing that directs the request to the vLLM replica that already has that prefix warm. What there is, is affinity by session or by key. For the other thing you need a specific router underneath the gateway, and that architecture has its own article pending.

The engine: what vLLM can and cannot do

Everything above shares out requests between engines. Inside one engine, the question is what happens when a twenty-token request and a sixty-thousand-token one live side by side.

Priority exists and does less than it looks like. --scheduling-policy priority turns the waiting queue into a heap ordered by priority, arrival time and identifier, with the lowest number first. The request can carry the priority field in the body, and there is also the X-Vllm-Priority header, which takes precedence over the body and is not documented anywhere.

What priority does not do is push ahead of whoever is already generating. In the running queue loop, when KV cache runs short, the policy only changes who is chosen as the victim of eviction: with priority the one with the worst priority and arrival time is evicted, and with fcfs the last in the queue. In the waiting queue loop, if there are no blocks available the iteration is cut short, without evicting anyone. And the max_num_seqs cap is a hard cut. If the agents have filled that cap, a chat with maximum priority waits just the same. There has been an open issue since April and an unmerged pull request about this.

That nuance about the victim is not negligible, because with fcfs the victims are the most recently admitted, that is, the chats that have just come in. Switching the priority policy on already improves the situation even if it pushes nobody ahead.

Eviction is always by recomputation. The swap-to-host-memory mode and the enumeration that controlled it disappeared from the code; all that is left is freeing blocks and returning the request to the queue with the computed token counter at zero. An evicted request loses all its prefill, and the prefix cache recovers part of it, but the perceived time to first token restarts.

Two admission settings help and do not appear in the narrative documentation. watermark reserves a fraction of free KV blocks on admission, with the declared purpose of avoiding repeated eviction, and it ships disabled. And scheduler_reserve_full_isl, active by default, checks that the full input length fits in the cache before admitting, instead of looking only at the first chunk.

Chunked prefill is active by default and prioritises decode. The scheduler walks the whole running request loop before touching the waiting queue, so a long agent prefill does not freeze a chat’s generation in progress. That protects the inter-token latency of what has already been admitted, and it does not protect the time to first token of what is waiting.

Here there is a breaking change that forces a review of the manifests. max_num_partial_prefills and max_long_partial_prefills were removed in 0.27. They were precisely the mechanism that allowed short prompts to get ahead of long ones. A deployment with those flags in its args does not start. long_prefill_token_threshold still exists, and its meaning changed: today it is a per-request chunk size cap, not a classification of long requests.

The partial replacement arrives in 0.29 with max_num_queued_tokens and max_num_queued_reqs, admission valves that return 503 when the pending prefill work exceeds the target. The code itself frames it as a quality of service mechanism for time to first token, with the formula of setting it to the latency target multiplied by prefill throughput. They are blind to the class of traffic: they reject a chat just the same as an agent.

And the default value of --max-num-batched-tokens depends on the hardware and on the start-up mode. With vllm serve on H100 or H200 it is 8,192, on cards of 160 GiB or more it is 16,384, and on the rest 2,048. The 2,048 that appears in the scheduler configuration only applies to tests.

The metrics for diagnosing coexistence, with the exact names from 0.29:

Metric	What for
`vllm:num_requests_waiting`	queue depth, the autoscaling signal
`vllm:num_requests_waiting_by_reason`	separates waiting for capacity from waiting in a blocked state
`vllm:num_preemptions_total`	evictions, the signal that the two classes are treading on each other
`vllm:kv_cache_usage_perc`	cache pressure, the cause of the evictions
`vllm:request_queue_time_seconds`	what a chat waits with the agents inside
`vllm:prefix_cache_hits_total` and `_queries_total`	measured in tokens, not in requests

A warning for anyone with old alerts. The log warning about evicted sequences that appears in the optimisation documentation belongs to the previous version of the engine and no longer exists in the code. Preemptions are only visible as a counter. An alert based on searching for that string in the logs has not fired for months, and not because there are no evictions.

On isolation between classes inside one engine, the short answer is that it does not exist. There are two queues and neither is per class of traffic. The conclusion, which is the one that holds up this article’s architecture: effective isolation means separate fleets, and on top of them the gateway deciding who goes to which.

Retries: the product nobody calculates

A human who gets an error sees it and decides. An agentic loop retries, and beneath it there are two more layers that also retry.

Layer	Default
Client SDK	2 retries, 3 attempts
LiteLLM router	2 retries, 3 attempts
Fallbacks	up to 5 groups

The product is about 45 calls to the engine for one agent turn, before counting the loop’s own retry. LiteLLM does what it can: when the call comes from the router, it forces max_retries=0 on the SDK it itself uses against the engine. What it cannot touch is the client’s SDK. The only layer that breaks the product is that one, and it is broken by setting max_retries=0 on the client and delegating the retry to the gateway.

The retryable errors are 408, 409, 429 and anything 500 or above. And there lies the problem with long reasoning: litellm.Timeout is a subclass of OpenAI’s timeout exception and comes out with code 408, which is retryable, so the most expensive request of all, the one that expired after ten minutes of generating, is launched again in full. Twice by the router, and as many times again by the client.

On timeouts you have to be precise because the code comment is: the configured timeout is per attempt, not per request. A request_timeout: 600 with two retries and two fallback groups gives a worst case of almost two hours before the client receives anything. The default value is still 6,000 seconds, as was already covered in the day 2 article.

The retry storm has two specific causes and both are default settings.

The first: a single 429 takes the deployment out of the pool, without going through the allowed_fails threshold. The cooldown logic has an explicit branch which, on a 429 in a group with more than one deployment, returns immediate cooldown. With an agentic burst against four replicas, all four leave the pool at once and the gateway answers everybody with its own 429 of “no deployments available”. The default cooldown is five seconds, which in a sustained burst gets renewed.

The second: backoff is zero while the pool looks healthy. The function that computes the wait returns 0 if the healthy deployment list is not empty. Exponential backoff with jitter, capped at eight seconds, only kicks in when there is nothing healthy left. Under partial degradation, the proxy hammers away without waiting and only slows down once everything has already fallen over.

On top of that, the engine’s Retry-After header is only respected if its value is between 1 and 60 seconds. A longer limit window is ignored and replaced by the proxy’s own backoff.

The settings that cut the problem off:

router_settings:
 num_retries: 1
 retry_after: 2 # real floor; the default 0 is hammering
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3 # stops a 429 taking the deployment out
 retry_policy:
 TimeoutErrorRetries: 0 # do not relaunch the expensive request
 AuthenticationErrorRetries: 0
litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false

That last setting deserves an explanation, because it is a silent way of breaking an agent. With drop_params: true, LiteLLM removes from the body any parameter the destination provider does not support, and in that branch it writes no warning at all. The candidates for disappearing are tools, tool_choice, response_format and parallel_tool_calls. An agent that has tools taken away receives free text where it expected a function call, and interprets it as the model having decided not to use tools.

Worse still is prompt-based emulation. For a non-OpenAI-compatible provider that does not support tools, LiteLLM does not fail: it turns the tools into text inside the prompt, forces JSON format, discards tool_choice, and on the way switches on a process-global variable that makes any other tool-less provider in the same process go to emulation instead of erroring.

With that in mind, the warning about fallbacks is clear: LiteLLM does not check capabilities before falling back. The function that knows whether a model supports function calling exists and is only used in an informational endpoint. The pre-call checks filter by context window, request limit and region, and nothing else. A fallback list that goes from a model with tools to one without them degrades the agent silently. There are guards for provider-scoped resources, files and batches, so the absence of the capability guard is an omission and not a design oversight.

A month’s budget in twenty minutes

A runaway agent consumes in minutes what a team consumes in weeks, and the operational question is how long the gateway takes to find out.

The answer has changed, and the old mental model, the one where spend consolidates every minute and until then there is no brake, is no longer correct. Since 1.84 there are two new pieces. An optimistic reservation before the call, which estimates the maximum cost of the request and reserves it atomically in the counter before heading out to the engine, reconciling it with the real cost on completion. And a counter increment after the call that is explicitly awaited, with the comment that this way the counter is up to date before the next request goes through authentication. Budget checks read that counter, with Redis first and seeding from the database when cold.

Batch writing to Postgres still exists, every 10 to 15 seconds, and it affects persistence and what is shown in the interface, not the cut-off. The production documentation’s recommendation to raise that interval to 60 does not relax the control, and it does delay by a minute what /key/info shows, something which frequently generates operational panic. And the budget rescheduler’s ten minutes do not consolidate spend: they fire the job that resets expired budgets.

The overspend window then comes down to the cost of the requests in flight, provided there is Redis. Without Redis, the counters are per pod, and with N replicas you can spend up to N times the budget. That is the configuration failure that matters and it is the same Redis that the day 2 article declared non-optional for other reasons.

Two settings that do not appear in the public documentation, only in the data model descriptions, and that decide behaviour at the worst moment:

general_settings:
 fail_closed_budget_enforcement: true # false by default: if the counter fails, it lets things through
 # disable_budget_reservation: do not set it to true

With the default value, if the counter backend does not respond, the proxy fails open and carries on serving. For an agent in a loop that is exactly the opposite of what you want.

Time for the honest warning: this route is still unstable. There are open issues in both directions, one where the key and user budget limit stops being applied while spend is recorded, and another where a 429 is returned for stale spend in the reservation counters while the key query shows spend below the limit. Project budgets are not on the atomic reservation route and concurrency overshoots them. The practical conclusion is not to trust the hard cut-off to the budget alone.

On the manual cut-off, there is an asymmetry to know about before you need it. /key/block invalidates the local cache, the Redis one, and notifies all workers by publish, so with Redis the effect is almost immediate. /customer/block does the same. /team/block only writes to the database, without invalidating or notifying, so it takes however long the management object cache TTL takes, sixty seconds by default. To cut a team off dead you have to block its keys one by one, or lower user_api_key_cache_ttl. And /user/block no longer exists, even though the docstring of the function that replaced it still shows that route in its example.

On alerts, two numbers. The default thresholds warn at 85 % and 95 % of the budget. And deduplication of those alerts is 24 hours per event and identifier, so crossing 85 % generates one warning and then silence for a day. For agentic traffic, where 85 % and 100 % can be twenty minutes apart, the second warning arrives when it is no longer any use.

Attribution: who the user is when the caller is an agent

With agents, the question of who the spend is charged to has two answers and both are needed: the agent that executes and the human on whose behalf it acts.

The chain that extracts the end customer identifier tries the standard headers first, then the configurable header mapping, and only then the user field of the body, the litellm_metadata.user of the Anthropic format, the metadata.user_id and the safety_identifier. The last four are body fields and are therefore written by the client. The code itself warns about it: for untrusted callers the identifier has to be set from headers or from a middleware on the server, to avoid impersonation.

Out of that comes the correct pattern, which is the inverse of the one people put together out of inertia: the virtual key identifies the agent and is the non-forgeable part, and the header or the user field identify the human and serve for attribution, not for control. An agent that lies about the human behind it is still bounded by the budget and the limits of its key.

The spend table stores user, end_user, team_id, organization_id, request_tags, session_id and agent_id, with daily aggregation per agent, so the detail is available once it is sent. The x-litellm-agent-id header is what populates that last field.

A note about the MCP gateway, which on a platform with agents ends up being the other entry door. Tool calls do appear in the spend records, with call types of their own and a metadata block that stores tool name, arguments, result and server. What is missing is token counting: the cost is 0.0 by default and is only populated if the administrator configures a price per query, per server or per tool. It is the same pattern of silent zero cost that already came up with unpriced models in the first article in the track.

And the permission setting to review on day one: access control for MCP servers is open by default. If no level of the hierarchy defines a list, the request reaches every configured server. general_settings.require_key_mcp_access_defined: true inverts that inheritance, and it is not the default value.

A reference `config.yaml`

All of the above, together, on a fleet split into two pools:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018 # without this, silent zero cost
 output_cost_per_token: 0.00000072
 model_info:
 id: chat-01
 max_input_tokens: 32768 # without this there is no context filter

 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072
 tag_regex: ["^User-Agent: claude-code\\/"]
 model_info:
 id: agentes-01
 max_input_tokens: 131072
 allow_fail_open: true

router_settings:
 routing_strategy: simple-shuffle
 enable_pre_call_checks: true
 enable_tag_filtering: true
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600
 num_retries: 1
 retry_after: 2
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3
 retry_policy:
 TimeoutErrorRetries: 0
 AuthenticationErrorRetries: 0

general_settings:
 fail_closed_budget_enforcement: true
 require_key_mcp_access_defined: true
 user_api_key_cache_ttl: 10
 proxy_batch_write_at: 10
 # 1.101 branch onwards:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false
 cache: true # Redis, not optional

And the start-up of the two engines, with the deliberate asymmetry:

# interactive pool: protect the first token
vllm serve Qwen/Qwen3-30B \
 --scheduling-policy priority \
 --max-num-seqs 64 \
 --max-model-len 32768 \
 --max-num-batched-tokens 4096

# agent pool: throughput and deep queues
vllm serve Qwen/Qwen3-30B \
 --max-num-seqs 256 \
 --max-model-len 131072 \
 --max-num-batched-tokens 8192

Checklist

Issue separate keys for agents, with model_group_alias at key or team level. It is the only separation the client cannot evade.
Add tag_regex over the User-Agent as a safety net for agents that use personal credentials, with a default pool and allow_fail_open: true.
Leave simple-shuffle on both pools. The other four strategies have verifiable defects on homogeneous replicas.
Enable session_affinity with a one-hour TTL. Do not use prompt_caching for this while its TTL stays hardcoded.
Declare max_input_tokens on every deployment. Without that value, the context window filter is not applied and it fails silently for models that are not in the price map.
Declare per-token prices on every self-hosted model, or the agents’ spend will be zero and their budget will never run out.
max_parallel_requests on the agent’s key, never on its team. On the team it is stored and not applied.
Lower num_retries to 1, put a floor on retry_after, raise cooldown_time and disable retry on timeout. On the client, max_retries=0.
fail_closed_budget_enforcement: true and mandatory Redis. Without Redis, N replicas are N times the budget.
Start the interactive engine with --scheduling-policy priority even though it pushes nobody ahead: it changes who it evicts.
Alert on vllm:num_preemptions_total and vllm:num_requests_waiting_by_reason, not on log strings that are no longer emitted.
Prepare the cut-off: block by key, never by team, and verify the real propagation time in your own deployment.

Traps and things that are not what they seem

priority: 0 on /chat/completions disables prioritisation. It is the value in the documentation examples and it is falsy in Python.
max_parallel_requests on a team is stored and not applied. The field exists in the API, in the database and in the documentation; the limiter only reads it from the key descriptor.
global_max_parallel_requests is a no-op with the default limiter, and the server settings documentation still describes it as active.
Every MCP call leaks a concurrency slot, which only heals when its hour expires. With agents calling tools in a loop, the 429 appears early and lasts a long time.
The TTL of prompt cache affinity is 300 hardcoded seconds. For agent conversations, session affinity is the right option.
Affinity is evaluated before tag-based routing. A pin to a deployment that stops satisfying the tag makes the request fail instead of falling through to the default pool.
LiteLLM’s timeout is per attempt. The worst case is the timeout multiplied by attempts and by fallback groups.
A 429 takes the deployment out of the pool without going through allowed_fails, and backoff is zero while anything healthy remains.
The engine’s Retry-After is ignored if it asks for more than 60 seconds.
drop_params: true removes tools and response_format without leaving a warning, and the fallback does not check whether the destination model supports function calling.
/team/block does not invalidate the cache. It takes the full TTL; /key/block does propagate by publish in Redis.
Access to MCP servers is open by default if no level defines a list.
max_num_partial_prefills and max_long_partial_prefills no longer exist in vLLM. A manifest with those flags does not start from 0.27 onwards.
vLLM’s priority does not push ahead of running requests, and with max_num_seqs full it pushes nobody ahead.
The eviction log warning that appears in the vLLM documentation is no longer emitted. Only the Prometheus counter is left.

Closing

The coexistence of humans and agents on the same platform is not solved with a priority field, and that is the conclusion that takes the most effort to accept because the field exists and looks made for this. Neither the gateway nor the engine has a usable notion of class of service today: one orders a queue that is almost never consulted, the other picks who to evict. What they do have is the ability to direct each request to a different place, and that is where the win is.

The architecture that comes out of all of the above fits in one sentence. Two vLLM fleets with opposite parameters, a credential that decides which one each request goes to, session affinity so as not to throw away the prefill work, concurrency limits on the agent’s key and not on its team, retries trimmed back on the three layers where they multiply, and budgets that fail closed.

The rest is measurement. If vllm:num_preemptions_total rises in the interactive pool, the two classes are treading on each other despite the separation. If chat queue time grows while total throughput holds, the partition is badly sized. And if the agents’ spend comes out as zero, the problem is not coexistence but undeclared prices, which is where this track started.

Sources

LiteLLM, [BETA] Request Prioritization: https://docs.litellm.ai/docs/scheduler.
LiteLLM, Router - Load Balancing (estrategias, afinidad de sesión, comprobaciones previas): https://docs.litellm.ai/docs/routing.
LiteLLM, Tag Based Routing (incluye tag_regex y el ejemplo con clientes agénticos): https://docs.litellm.ai/docs/proxy/tag_routing.
LiteLLM, Router Settings for Keys and Teams (resolución clave, equipo, global): https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.
LiteLLM, Server Tuning (control de admisión por worker): https://docs.litellm.ai/docs/proxy/server_tuning.
LiteLLM, Dynamic TPM/RPM Allocation (reserva de prioridad, beta y enterprise): https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.
LiteLLM, Budgets, Rate Limits: https://docs.litellm.ai/docs/proxy/users.
LiteLLM, Production Best Practices: https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Reliability (fallbacks, ventana de contexto, enfriamientos): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, código: litellm/scheduler.py, litellm/router_strategy/tag_based_routing.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/hooks/parallel_request_limiter_v3.py, litellm/proxy/middleware/admission_control_middleware.py, litellm/router_utils/cooldown_handlers.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de la afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en cada llamada MCP: https://github.com/BerriAI/litellm/issues/34534.
LiteLLM, incidencia 26672, presupuesto de clave y usuario no aplicado: https://github.com/BerriAI/litellm/issues/26672.
LiteLLM, incidencia 872, comprobación previa de soporte de llamada a funciones, cerrada sin implementar: https://github.com/BerriAI/litellm/issues/872.
vLLM, Optimization and Tuning (prefill troceado, prioridad de decodificación): https://docs.vllm.ai/en/v0.29.0/configuration/optimization.html.
vLLM, Metrics design (nombres, convención _total, autoescalado como problema abierto): https://docs.vllm.ai/en/v0.29.0/design/metrics.html.
vLLM, Data Parallel Deployment (caché KV independiente por réplica): https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment.html.
vLLM, código: vllm/v1/core/sched/scheduler.py, vllm/v1/core/sched/request_queue.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/metrics/loggers.py: https://github.com/vllm-project/vllm.
vLLM, incidencia 40004, la prioridad no desaloja peticiones en ejecución: https://github.com/vllm-project/vllm/issues/40004.

LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

Tue, 08 Sep 2026 07:00:00 +0200

Second article in the operational track of the control layer. The operational pair with Langfuse covered the seam between the gateway and observability; here everything stays inside the gateway. The prior decisions, which gateway and why, are in choosing the OSS gateway and in the L7 inference router.

TL;DR

LiteLLM Proxy starts from a twenty-line config.yaml and serves traffic that same afternoon. The operational work consists of correcting seven defaults designed for a scenario other than an in-house inference factory.

One uvicorn worker per pod. The project’s production guide asks for --num_workers 1 on Kubernetes and horizontal scaling, with 1 vCPU and 4 GiB per worker set at the same time as requests and as limits. The 4 GiB are a floor, not a target: the Prisma query engine marks a resident memory high-water mark that grows up to the largest statement it has executed and that glibc does not return to the system.

Autoscale on CPU, never on memory. A direct consequence of the above. Memory goes up and never down, so an HPA on memory scales and never scales back. The recommended target is targetCPUUtilizationPercentage: 60.

The Postgres connection arithmetic. database_connection_pool_limit is 10, and the number of connections is instances times workers times the pool. The Helm charts ship maxReplicas: 100, on the order of a thousand connections, far above what a stock Postgres accepts.

Redis stops being optional from around a thousand requests per second or ten instances. Without it, each instance applies its limits on its own, caches are not shared, and spend updates against the same rows produce locks and exhaust the connections with FATAL: sorry, too many clients already.

Three health endpoints and only two are usable as probes. /health/liveliness and /health/readiness require no authentication and answer fast. Bare /health demands a key and fires a real request against every model in the catalogue, with its cost in tokens.

The resilience numbers are 2, 3 and 5: two retries, three allowed failures and five seconds of cooldown, the last one despite the function’s own documentation saying the default is 1.

request_timeout is 6,000 seconds. A hundred minutes holding a connection against an engine that does not answer.

And a warning about the performance figures: the project publishes 8 ms of p95 overhead on one benchmark and 257.7 ms of p99 on another. Both are theirs, and the difference is in the shape of the load.

You are here: the gateway layer, the day after

In the seven-layer stack, this article lives entirely in the gateway layer. The difference with the selection article is the one that separates buying from maintaining: there the question was which piece to put in front of the fleet, and here it is what happens when that piece has been serving for six months, the team has tripled the traffic and someone has added an agent that fires twenty calls per interaction.

The gateway has an uncomfortable property worth keeping in mind: it sits on the critical path of every request and contributes not a single token. All it does is coordination, and any coordination that fails turns into an outage of the whole inference service, even with the GPUs perfectly healthy.

The analogy: the switchboard and its notebook

A hospital switchboard does three jobs at once. It puts calls through, which is the urgent part. It writes down in a notebook who called, to which extension and for how long, which is what sustains the billing. And it consults a list of extensions to know where to put each call through.

The three jobs compete. If the operator stops to write each line in the notebook before putting the next call through, the waiting queue grows. If there are three operators and each keeps her own notebook, the totals do not add up. And if the list of extensions is consulted in a filing cabinet on another floor, every call costs the trip.

The three tensions have an exact equivalent in LiteLLM. The notebook is the SpendLogs in Postgres, which is why the transaction buffer in Redis exists. The three operators with three notebooks are the pod replicas applying limits independently. And the filing cabinet on another floor is usage-based routing, which adds a Redis lookup inside the request path.

The process model: one worker per pod

The production documentation is explicit: on Kubernetes, --num_workers 1 and scale with replicas. On a virtual machine with no orchestrator, NUM_WORKERS equal to the number of vCPUs.

The reason is not ideological. An additional worker inside the same pod shares the container’s memory limit with the others, multiplies the database connections by the same factor, and registers a copy of every periodic job. With separate replicas, each one has its own resource budget, its own connection quota and a place where the scheduler can put it.

The recommended sizing is 1 vCPU and 4 GiB per worker, and the part that gets overlooked is that these go as requests and as limits at the same time. Setting less than 4 GiB produces an OOM restart loop that shows up late, once some large request has grown the query engine’s footprint.

resources:
 requests: { cpu: "1", memory: "4Gi" }
 limits: { cpu: "1", memory: "4Gi" }

That footprint needs to be described precisely, because it determines the autoscaling policy. The Prisma query engine keeps a resident memory that behaves as a high-water mark: it grows up to the largest statement the process has executed and does not come back down, because glibc does not return that freed memory to the system. A pod that served a spike yesterday still shows the footprint of that spike today.

Two rules come out of this. The first: autoscale on CPU and never on memory. An HPA on memory sees a metric that only goes up, scales at the peak and never scales back. The recommended target is targetCPUUtilizationPercentage: 60, lower than the 80 the charts ship by default, because the startup probe allows up to 300 seconds before passing the first readiness check and there needs to be headroom for the new pod to be ready before the saturated one goes down.

The second: bound the process lifetime. --max_requests_before_restart 10000 recycles the worker before the high-water mark matters.

Postgres: the arithmetic that blows up when you scale

database_connection_pool_limit is 10 by default. The total number of connections the deployment opens is replicas times workers times pool, and it has to be compared against the Postgres max_connections.

The formula the documentation gives is the inverse, and it is the one to apply when sizing:

database_connection_pool_limit = MAX_DB_CONNECTIONS / (instances × workers)

The detail that hurts in production is in the Helm charts, which ship autoscaling.maxReplicas and keda.maxReplicas at 100. With the default pool, that is on the order of a thousand connections at full scale-out, far above what an untouched Postgres accepts. And the moment the HPA reaches those replicas is exactly the moment of highest load, so connection exhaustion arrives when it can least be afforded.

There are two more valves on the same path. Provider errors are written to the database by default, and under a sustained engine failure that inflates the spend table without contributing anything that is not already in the metrics. They are switched off like this:

general_settings:
 disable_error_logs: true # stops writing provider errors
 proxy_batch_write_at: 60 # batches the spend writes

disable_spend_logs: true is the radical version, which removes the per-request detail from the interface and leaves the cost in Prometheus and in the tracing backend. Before going that far you have to decide whether that table is the system’s audit record, which is the discussion in the previous article: if it is, it cannot be switched off, and the way out is the Redis buffer.

Redis: when it stops being optional

Without Redis, LiteLLM works. Each instance keeps its own in-memory cache and applies its own counters. The consequences are three, and all of them show up as you grow.

Requests-per-minute limits are applied per instance, so a limit of 100 with five replicas is in practice a limit of 500. Cache hits are local, so the same repeated question hits once in five. And there is no leader election, so periodic jobs run in every process at once.

The threshold the documentation gives for enabling the transaction buffer is a thousand requests per second or ten instances. Below that, each instance updates the key, user and team rows directly; above it, they all write to the same rows, locks appear and Postgres starts rejecting with FATAL: sorry, too many clients already.

general_settings:
 use_redis_transaction_buffer: true

The metrics to watch when that is enabled are litellm_in_memory_spend_update_queue_size, litellm_redis_spend_update_queue_size and litellm_pod_lock_manager_size. If the first grows without the second coming down, the flush to Postgres is not keeping up.

A note about the routing strategy, which interacts with this. simple-shuffle consults nothing external. Usage-based routing does, and it adds a round trip to Redis inside the request path. On a homogeneous vLLM fleet behind the same model, the improvement it brings rarely pays for that latency; on a heterogeneous fleet, the conversation is different and the router article covered it.

Background jobs register per worker

This detail appears in no getting-started guide and produces considerable bewilderment when it is discovered. The proxy’s periodic jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every job, all of them running with no coordination if there is no Redis for leader election.

The variable that separates them is LITELLM_JOB_ROLE: value serving on the pods that serve traffic and a separate replica with value worker for the jobs. With the one-worker-per-pod recommendation the problem shrinks, but it does not go away while there are several replicas.

Probes: three endpoints and only two are usable

The three endpoints exist and do different things:

Endpoint	Authentication	What it does	What it is for
`/health/liveliness`	No	Returns `I'm alive!`, or 503 during shutdown	`livenessProbe`
`/health/readiness`	No	State of the process and the database. 503 if the configured Postgres does not answer	`readinessProbe`
`/health`	Yes	Fires a real request against every model in the catalogue	Manual diagnosis, never a probe

The last row deserves insisting on. /health spends tokens on every call, one per configured model. Set as a probe, with the Kubernetes default interval and a fleet of eight models, that is thousands of inference requests a day that serve nobody.

There are more endpoints useful for diagnosis: /health/readiness/details, /health/services?service=langfuse to check one specific callback, /health/history and /health/latest for the history, and /health/backlog, which returns the in-flight requests and shows up in the latency section. /health/drain exists but returns 404 unless it is enabled with enable_drain_endpoint: true, and it is protected with X-Drain-Token.

The periodic model checks are configured separately and not in the probe:

general_settings:
 background_health_checks: true
 health_check_interval: 300 # seconds
 background_health_check_model_groups: ["llama-70b", "qwen-30b"]

And one setting with consequences in deployments with no Internet egress: allow_requests_on_db_unavailable: true allows serving traffic with the database down, but leaves /health/readiness always returning 200. With that set, the readiness probe stops detecting the very failure it was designed to detect, and budget and model errors keep blocking just the same.

Retries, cooldowns and fallbacks: the real numbers

The defaults, read from the project’s code and not from the documentation:

Constant	Value	What it governs
`DEFAULT_MAX_RETRIES`	2	Retries before moving on to fallbacks
`DEFAULT_ALLOWED_FAILS`	3	Failures before cooling down a deployment
`DEFAULT_COOLDOWN_TIME_SECONDS`	5	Seconds a deployment stays out
`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD`	1000	Minimum requests before applying the cooldown logic with a single deployment
`DEFAULT_REQUEST_TIMEOUT_SECONDS`	6000.0	Request timeout

Besides the failure counter, there is a proportional criterion: a deployment enters cooldown if 50% of its requests fail in any given minute.

On the cooldown there is a discrepancy worth knowing about before debugging blind. The docstring of the router function says cooldown_time defaults to 1, and the code uses the constant of 5. The code wins.

The fallbacks are three separate lists and are applied after the retries are exhausted:

router_settings:
 num_retries: 2
 timeout: 120
 fallbacks: [{"llama-70b": ["qwen-30b"]}]
 context_window_fallbacks: [{"llama-70b": ["llama-70b-128k"]}]
 content_policy_fallbacks: [{"llama-70b": ["llama-70b-sin-guardrail"]}]

A note on the behaviour when everything is in cooldown: if no deployment is left available in the group, LiteLLM falls back onto a specific model_info.id, skipping the cooldown check. It is a deliberate degradation, serve something rather than return 503, and it is worth knowing it exists because it masks the real state of the fleet.

And a capability lost along the way: since version 1.85.0, the mock_testing_fallbacks parameter and its two siblings are stripped from requests arriving through the proxy and have no effect. Fallbacks can no longer be tested against the proxy with a doctored request; it has to be done against litellm.Router directly, in a test. Anyone who had a smoke test built on that has it broken without warning.

The six thousand second timeout

It deserves its own section because it is the setting with the worst damage-to-effort ratio in the whole system. request_timeout is 6,000 seconds. A hundred minutes. An inference engine that stops answering without closing the connection holds a worker for that long, and with one worker per pod that is an entire pod out of service per trapped request.

general_settings:
 request_timeout: 600

Ten minutes is still generous for a long generation, and it bounds the damage.

Overhead: the project’s two figures

LiteLLM publishes two measurements of its own overhead that differ by a factor of thirty, and both are legitimate because they measure different loads.

The benchmarks page, with Locust, a thousand users with think time and a fake endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms at 1,170 requests per second. With two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures correspond to around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, from July 2026, measures the Python proxy at 257.7 ms of added p99 and 329.5 MB of peak memory, with no callbacks, no spend tracking and no persistence. The Rust variant, in beta, gives 0.7 ms.

The reading for an in-house platform: the first figure is the best case with human clients, and the second is the order of magnitude to plan for agentic load, which arrives in a closed loop against fast responses. There is one reported case that resembles this scenario closely: an in-house OpenAI-compatible engine served around 16 requests per second directly and around 9 through LiteLLM, with the degradation growing with concurrency, on a 4 vCPU, 8 GB pod.

The latency gap the metrics cannot see

LiteLLM’s timers start when its handler starts. The time the request spends queued in the uvicorn event loop, before getting there, appears in none of its metrics. The example in its own troubleshooting guide is a case in which LiteLLM records 10 seconds and the user experiences 20.

It is detected by comparing two sources: GET /health/backlog or the litellm_in_flight_requests gauge against the response time measured by the load balancer in front. If they diverge, the gap is in the input queue and the answer is more replicas, not more tuning.

Two headers are always on and serve to watch it without instrumenting anything: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms. For the second, the documentation sets the suspicion threshold at 100 ms, above which the diagnosis is that the payloads are too large.

And the first cause of latency that guide mentions is none of the above: LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and with payloads over 2 MB that alone can cost between 2 and 5 seconds per request. A log level set to debug a problem and forgotten there produces exactly the problem it was meant to debug.

The failure mode to put in the runbook

There is an open issue describing a cascade with a characteristic shape. Under sustained 429s from the engine upstream, the pods stopped answering readiness probes during startup, Kubernetes killed them on failed probes, and the restart put more pressure on an upstream that was already saturated. Peaks of 500 requests per second, average of 60, between two and five replicas with 1.3 CPU and 4 GB.

The shape of the failure is what matters more than the specific case: the engine’s saturation turns into a restart loop of the gateway, and from the outside it looks like the problem is the gateway. The defences are the ones already in this article, applied together: startupProbe with a wide margin so readiness does not compete with startup, request_timeout bounded so trapped requests get released, cooldowns that take the deployment returning 429 out of rotation, and CPU autoscaling at 60% so the capacity exists before it is needed. It fits with what was covered in incident response runbooks.

Upgrading without cutting traffic

The deployment is a normal Deployment with RollingUpdate, maxUnavailable: 0 and maxSurge: 1, and there is no further mystery to it except on one point: schema migrations. LiteLLM uses Prisma and applies migrations at startup. With several replicas coming up at once after a version change, the migration must be run by a single process, which is another reason for the dedicated replica with LITELLM_JOB_ROLE: worker.

Before a version bump, two checks that pay for themselves: read the release notes looking for changes in the metadata emitted to the callbacks, because a change like that breaks saved dashboards without touching the service, and verify that the new configuration starts with --detailed_debug on a separate pod before applying it to the fleet.

Go-live checklist

--num_workers 1 and scaling by replicas.
requests equal to limits, 1 vCPU and 4 GiB per worker.
HPA on CPU at 60%. Never on memory.
database_connection_pool_limit computed from max_connections, and the chart’s maxReplicas lowered to something the database can take.
Redis as soon as there is more than one replica, and use_redis_transaction_buffer from ten instances or a thousand requests per second.
livenessProbe on /health/liveliness, readinessProbe on /health/readiness, and /health out of the probes.
startupProbe with a margin of up to 300 seconds.
request_timeout: 600.
LITELLM_LOG out of DEBUG, and disable_error_logs: true.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, litellm_in_flight_requests and the response time of the load balancer in front.

Traps and things that are not what they look like

The HPA scales up and never back down. It is measuring memory, which in this process only goes up.

The pods die in a loop right after deploying. Less than 4 GiB of limit. The footprint appears with the first large request, not at startup.

Postgres starts rejecting connections at the peak. The chart has maxReplicas: 100 and the pool defaults to 10.

The per-minute limits are enforced badly. There is no Redis, and each replica keeps its own count.

The health check consumes tokens. Someone set /health as the readiness probe.

The readiness probe returns 200 with the database down. allow_requests_on_db_unavailable is set.

A deployment keeps receiving traffic after failing. The cooldown is 5 seconds, not minutes, and with a single deployment in the group you have to pass a thousand requests for the logic to apply.

The fallback test no longer tests anything. The simulation parameters are stripped from requests arriving through the proxy since 1.85.0.

The p99 is ten times worse than the published benchmark. The benchmark has think time and the real load is agentic.

Closing

None of these settings is complicated and all of them are one line. What unites them is that their defaults describe a small deployment, with one instance, a human client on the other side and a commercial provider taking care of capacity. An in-house inference factory is the opposite on all four axes: several replicas, automated clients, and an engine whose capacity is finite and known.

Of the ten points in the checklist, three decide almost everything in an on-premise deployment: the process model with its sizing, the database connection arithmetic, and the request timeout. The other seven are the ones that avoid the Saturday phone call.

Sources

LiteLLM, Production best practices (topología, workers, memoria, pool de conexiones, LITELLM_JOB_ROLE, buffer de transacciones en Redis, request_timeout): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Health checks (los tres endpoints, comprobaciones de fondo, enable_drain_endpoint): https://docs.litellm.ai/docs/proxy/health.
LiteLLM, Reliability (fallbacks, reintentos, deprecación de los parámetros de simulación en la 1.85.0): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, Benchmarks y Load test advanced (las cifras de sobrecarga por número de instancias y la advertencia de la ley de Little): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/load_test_advanced.
LiteLLM, Rust AI Gateway benchmarks (los 257,7 ms de p99 del proxy en Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, Latency overhead (el hueco de la cola de uvicorn, las cabeceras de diagnóstico, el coste de LITELLM_LOG=DEBUG): https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, DB info (disable_spend_logs, disable_error_logs y qué se pierde con cada uno): https://docs.litellm.ai/docs/proxy/db_info.
LiteLLM, litellm/constants.py (los valores por defecto de reintentos, fallos permitidos, enfriamiento y timeout): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
LiteLLM, litellm/router.py (la discrepancia entre el docstring de cooldown_time y la constante que usa el código): https://github.com/BerriAI/litellm/blob/main/litellm/router.py.
LiteLLM, issue #15526 — cascada de 429 sostenidos, sondas de readiness fallidas y bucle de reinicios: https://github.com/BerriAI/litellm/issues/15526.
LiteLLM, issue #21046 — degradación de throughput contra un motor propio compatible con OpenAI: https://github.com/BerriAI/litellm/issues/21046.

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

Mon, 07 Sep 2026 18:00:00 +0200

This post opens the operational track of the control layer. The previous articles left the pieces chosen and assembled: choosing the OSS gateway and the L7 inference router explain why LiteLLM ends up in front of the fleet, and Langfuse from the inside takes apart the backend that receives the traces. Here we deal with the seam between the two, which is where the work of the next three years lives.

TL;DR

The LiteLLM plus Langfuse pair is configured with a list of callbacks and two API keys, and from there four matters decide whether the dashboard is good for operating or only for showing in a demo.

There are three integration routes and they are not equivalent. The native langfuse callback is tied to the Python SDK v2 (langfuse>=2.59.7,<3.0 in LiteLLM’s pyproject.toml), which writes against the legacy ingestion that Langfuse Cloud retires on 16 November 2026. The langfuse_otel preset exports over OTLP and is the path recommended in the current documentation of both projects. The OTel v2 route, behind the LITELLM_OTEL_V2=true flag, changes span names, leaves the success status at UNSET and, above all, its Langfuse mapper does not read the request metadata: session_id, trace_user_id, tags and trace_id stop arriving.

The cost of a self-hosted model comes out as zero unless a price is registered. get_model_info() raises an exception for a model that is not in the pricing map, completion_cost() propagates it, the logger catches it, stores it in a debugging field and returns None, and the payload turns it into 0.0. There is no warning at the default log level. A team’s budget is never exhausted because its spend is always zero.

Client-gateway-engine correlation does not come out of the box. LiteLLM does continue the client’s trace if a traceparent arrives, and it also extracts from it the 32-hex trace-id to use as litellm_trace_id, which makes the Langfuse identifier equal to the W3C one. Upstream the story changes: forward_traceparent_to_llm_provider is False by default, the function that applies it returns early if no otel logger is loaded, and when it does act it forwards the client’s original header, so the vLLM span ends up a sibling of LiteLLM’s rather than a child. Add to that the fact that vLLM exports OTLP over gRPC by default and Langfuse does not accept gRPC.

Between the request and the dashboard there are four bounded queues that drop events. LiteLLM’s logging thread pool (100 threads, 10,000 pending tasks, dropping once full), the asynchronous LoggingWorker (50,000, and once full it aggressively flushes half), OTel’s BatchSpanProcessor (2,048 spans by default) and Langfuse’s ClickhouseWriter, which drops rows after exhausting the retries with no dead-letter queue. None of the four applies back pressure on the request, which is exactly what you want, and none of them guarantees delivery, which is exactly what an auditor asks about.

The rest of the article develops the four points with configuration applicable to a generic 4×H100 cluster with vLLM, the overhead figures the project itself publishes, and the honest limit of what this data sustains before ENS or ISO/IEC 42001.

You are here: the seam between DEPLOY and OBSERVE

In the seven-layer stack, LiteLLM lives in the gateway layer and Langfuse in the observability one. The separation is clean on the diagram and false in operation: the gateway is the only point in the system that sees the complete request before chopping it up and the complete response after reassembling it, so it is also the only reasonable place from which to emit the trace. The inference engine sees tokens and latencies; it does not see who is asking or how much that team has spent this month.

That turns LiteLLM into a telemetry producer as well as a traffic distributor, and moves onto the gateway a responsibility that was not in the original design: if the callback fails, the request is served all the same and nobody notices that the dashboard has been incomplete for two days.

The analogy: the toll booth that issues the ticket

A motorway toll booth does two independent jobs. It raises the barrier, which is its visible function, and it issues a ticket with number plate, time, section and amount, which is the function the concession lives off. The two share a booth and do not share a destination: the barrier responds in a hundred milliseconds because there is a car waiting, and the ticket travels to a processing centre over a slower channel that batches thousands of records before sending them.

The design is well thought out as long as its consequence is understood. If the processing centre goes down, cars keep going through. If the ticket queue fills up, tickets are thrown away, traffic is not stopped. The concession accepts losing records rather than causing a jam, and charges on a statistical sample it knows to be imperfect.

LiteLLM and Langfuse work the same way. The four queues in the corresponding section are the slow channel, and all of them prefer to drop rather than block. Whoever builds the pair inheriting the complete analogy gets the operation right. Whoever thinks the ticket is an accounting entry ends up explaining to an auditor why traces from a Tuesday afternoon are missing.

Three routes, and one with an expiry date

As of September 2026 three paths coexist for getting a request from LiteLLM to Langfuse. LiteLLM’s documentation already labels the first as legacy.

Route	How it is enabled	Transport	Status
Native `langfuse` callback	`success_callback: ["langfuse"]`	Langfuse Python SDK v2, legacy ingestion	Maintained for compatibility. Tied to `langfuse<3.0`
`langfuse_otel` preset (v1)	`callbacks: ["langfuse_otel"]`	OTLP/HTTP against `/api/public/otel`	Recommended by both projects
OTel v2 route	`callbacks: ["langfuse_otel"]` + `LITELLM_OTEL_V2=true`	OTLP/HTTP, new mapper	Opt-in. Changes semantics and loses metadata

The date that orders the decision is 16 November 2026. Langfuse’s compatibility matrix marks the legacy batch ingestion of the Python SDK v2 as retired on that date for Cloud, at which point trace ingestion through that route stops working. On self-hosted the endpoint survives, but a v4 server in events_only mode returns 400 for every event type except score-create and sdk-log.

Since LiteLLM’s native callback is pinned to langfuse>=2.59.7,<3.0 in its own pyproject.toml, and its code calls APIs that only exist in SDK v2 (self.Langfuse.trace(...), trace.generation(...)), there is no way to move up an SDK without changing route. Anyone who today has success_callback: ["langfuse"] against Langfuse Cloud has a migration with a date on it. Anyone who has it against a self-hosted Langfuse has one with some slack, because v3 receives security patches until the end of January 2027, but they have one all the same.

One detail of the OTLP route worth checking in older deployments: the x-langfuse-ingestion-version: 4 header is the one that enables real-time ingestion. Without it the traces arrive, but with a delay that Langfuse’s documentation puts at around ten minutes (the compatibility page says fifteen, and that discrepancy remains unresolved in their own documents). LiteLLM started sending it in version 1.95.0, of August 2026. A proxy older than that version against a Langfuse v4 produces the classic symptom: the dashboard works, but it is always behind and nobody knows why.

The configuration that works today

For a generic 4×H100 cluster with vLLM behind it and Langfuse self-hosted in the same Kubernetes, the OTLP route in its v1 variant is the one that gives the most functionality with the fewest surprises.

model_list:
 - model_name: llama-70b
 litellm_params:
 model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
 api_base: http://vllm-llama70b.inference.svc.cluster.local:8000/v1
 api_key: os.environ/VLLM_API_KEY
 # Price derived from the cluster's €/GPU-hour. Without this, spend = 0.0
 input_cost_per_token: 0.00000031
 output_cost_per_token: 0.00000089

litellm_settings:
 callbacks: ["langfuse_otel"]
 turn_off_message_logging: false # a conscious decision, see the PII section
 langfuse_default_tags: ["model_group", "user_api_key_team_alias", "cache_hit"]

general_settings:
 disable_error_logs: true # provider errors inflate SpendLogs
 proxy_batch_write_at: 60

The credentials travel through the environment. LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST are enough: the preset builds the Authorization: Basic base64(public:secret) header and adds the ingestion version one, so there is no need to touch OTEL_EXPORTER_OTLP_HEADERS by hand. The preset’s default host points to the US region of Langfuse Cloud, a relevant detail for anyone building a platform with data sovereignty requirements: if the variable is not defined, the telemetry leaves Spain without anyone having decided it.

For real multi-tenancy, LiteLLM supports per-team Langfuse credentials (POST /team/<id>/callback with callback_vars) and per-key ones (metadata.logging[] when generating the key), and the key’s wins over the team’s without mixing. There is a limitation that breaks the architecture of anyone who planned to separate tenants across different Langfuse instances: under OTel v2, a per-key or per-team langfuse_host does not move that tenant to another host, because the exporter fixes the destination at startup and the only thing that varies per tenant are the headers.

And one operational warning with years of history behind it: MAX_LANGFUSE_INITIALIZED_CLIENTS is 50 by default because each Langfuse client starts a thread. The comment in LiteLLM’s code is explicit about the incident that motivated it, with the CPU at 100% from repeated initialisations. A deployment with per-team credentials and many teams hits that ceiling.

What is lost when moving to OTel v2

The LITELLM_OTEL_V2=true flag brings a step up in quality in the proxy’s general instrumentation, and at the same time a concrete regression on the path to Langfuse that the documentation acknowledges without underlining it.

The Langfuse mapper of the v2 route emits langfuse.observation.type, langfuse.observation.model.name, the observation identifier, the model parameters, input and output, usage_details and cost_details, plus langfuse.trace.metadata.team_id and team_alias. It does not emit session.id, nor user.id, nor langfuse.trace.tags, nor langfuse.trace.id, nor the generation name, nor parent_observation_id. LiteLLM’s documents put it this way: those attributes are set by the preset from the request and the response, not from metadata supplied by the client.

The practical consequence is that an application that today groups conversations by sending metadata: {"session_id": "...", "trace_user_id": "..."} in the request body stops grouping them when the flag is enabled, with no error and no warning. The traces keep arriving, loose.

The other v2 changes are semantic and affect alerts already written. The root span goes from being called Received Proxy Server Request to being named after the route (POST /v1/chat/completions), the inference span goes from litellm_request to {operation} {model}, the guardrail spans stop hanging off the inference span and hang off the root instead, and the success status stays at UNSET instead of OK. Any rule that fires on status == OK stops firing. The way back is to remove the flag and restart.

Cost: why the trace says zero

This is the part that takes the most time to diagnose and the least to fix.

LiteLLM computes the cost of a request from a per-model pricing map. For commercial provider models the map comes filled in. For hosted_vllm/whatever, which is all there is in an in-house inference factory, the map is empty, and the failure path is as follows:

litellm.get_model_info() raises an exception with the message This model isn't mapped yet.
completion_cost() and response_cost_calculator() propagate the exception.
Logging._response_cost_calculator() catches it, writes a StandardLoggingModelCostFailureDebugInformation into response_cost_failure_debug_information, logs it at debug level and returns None.
The standard logging payload does llm_response_cost = raw_response_cost or 0.0.

The system delivers a spend of zero, with no visible error at the default log level, and writes it into the spend column of LiteLLM_SpendLogs. The per-key, per-team and per-organisation budgets are evaluated against that zero and are never exhausted. The Langfuse dashboard shows a total cost of zero for everything self-hosted, which is precisely the part you wanted to measure.

Registering the price

The pricing keys are accepted both in litellm_params and in model_info, and those in litellm_params win when they are in both places. The complete set has 91 fields, but for a deployment with vLLM two are enough, and there are three things to know about them.

The first is that the price is per individual token, in USD. There are no per-thousand variants: searching for per_1k or per_1000 in the pricing model returns nothing. A three-orders-of-magnitude error here is easy to make and hard to see, because it produces plausible figures.

The second is that input_cost_per_request exists in the schema and is not applied in chat calls: a model registered with that field returns a cost of 0.0. To charge per request it has to be converted into a cost per token.

The third is that input_cost_per_second needs the response time and is computed as price times response_time_ms / 1000, so outside the proxy’s request path it silently returns zero.

The number that goes into those two keys comes from the article on FinOps and multi-tenancy with LiteLLM and the one on cost per token and per request: amortisation plus energy divided by tokens served. The example arithmetic, to set the order of magnitude: a cluster whose fully loaded cost is 9.60 USD per hour sustaining 3,000 output tokens per second produces 10.8 million tokens an hour, that is 0.00000089 USD per output token. That is the value that goes into output_cost_per_token. If the cluster’s average utilisation is 30 % and not 100 %, the divisor changes and the price triples, which is the same lever that article develops.

Before sending traffic, the /spend/calculate endpoint accepts the same parameters as completion_cost and returns the computed cost. It is the cheap way to check that the price resolves.

How that number reaches Langfuse, and who wins

The two integration routes write the cost into different fields. The native callback sends it as usage.total_cost of the v2 schema, and duplicates it in the trace metadata as litellm_response_cost. The OTLP preset emits langfuse.observation.cost_details as a JSON string of the form {"total": cost}, and langfuse.observation.usage_details with input, output and total.

The rule on the Langfuse side is documented and is the right one for this case: when there is ingested cost and inferred cost, the ingested one takes priority. Langfuse only computes from its own pricing table when the emitter sends no cost, and changes to a model definition apply to new generations, with no retroactive recalculation. A self-hosted model therefore needs no pricing definition in Langfuse: it is enough for LiteLLM to send cost_details.

One detail remains that explains a puzzling symptom. Langfuse stores the ingested cost and the computed one in separate columns, totalCost and calculatedTotalCost, and the session header totals add up only the first. A mixed deployment where part of the traffic brings cost and part lets Langfuse compute it shows sessions at zero while the individual generations show cost. There is an open issue about it.

The tokens of a streaming response

Almost all of an assistant’s traffic is streamed, and there the OpenAI usage object only appears if the client asks for stream_options.include_usage. LiteLLM solves this by injecting the option on its own: on the asynchronous completion routes with stream: true, if the caller did not set it and all the candidate deployments support it, it adds it and marks _litellm_strip_stream_usage so the usage block is used in the cost calculation and not returned to the client. The general_settings.always_include_stream_usage flag forces or disables that behaviour, and is worth setting to True when capability detection does not recognise the vLLM deployment.

When the usage is missing anyway, LiteLLM reassembles the response with stream_chunk_builder() and counts tokens with the tokeniser. It is an approximation, and it produces the second path towards a cost of zero. The two paths to zero are told apart by looking at total_tokens in LiteLLM_SpendLogs: zero tokens with a correct price is a usage problem, correct tokens with zero cost is a price problem.

Correlation: from the client to the engine

The operational question that justifies the whole setup is being able to take a user complaint, find their request and see where the time went. That requires the trace identifier to survive three hops.

What does work

If the client sends traceparent, LiteLLM’s spans hang off the existing trace. In addition, and independently of the OTel subsystem, litellm_pre_call_utils.py extracts by regular expression the 32-character hexadecimal trace-id and assigns it to data["litellm_trace_id"] as a last resort, and takes the session_id from the W3C baggage header. That makes the Langfuse trace identifier equal to the W3C trace identifier, which is the most robust correlation key available: with it, an identifier copied from the application log opens the trace in Langfuse with no translation.

For the routes that are not OTel v2, moreover, headers prefixed with langfuse_ inject per-request metadata. langfuse_trace_id, langfuse_trace_user_id and langfuse_trace_metadata are documented, but the code strips the prefix and writes the rest into the metadata, so any body key works as a header. Two warnings about this: header values arrive as strings and are not interpreted as JSON, and Nginx drops headers with underscores by default (underscores_in_headers off), which produces the symptom that the metadata works locally and disappears behind the ingress.

What does not work without effort

Towards the engine, propagation is off. The flag is litellm.forward_traceparent_to_llm_provider, it is False, and its implementation has three behaviours to know about before enabling it:

The function returns immediately if open_telemetry_logger is None. A deployment with callbacks: ["langfuse_otel"] and without the generic otel callback propagates nothing, whatever the flag is set to.
When it does propagate, it copies the client’s traceparent header as is, instead of injecting the context of LiteLLM’s active span. The vLLM span ends up hanging off the client’s span-id, a sibling of the gateway call’s span. The trace is one, the hierarchy is wrong.
If the client sends no traceparent, none is created.

Add to that the fact that LiteLLM’s general policy is not to forward client headers to the provider, and that the forward_client_headers_to_llm_api allowlist covers headers prefixed with x-, among which traceparent is not. This is the only route.

The vLLM side, and why the Collector stops being optional

vLLM enables traces with --otlp-traces-endpoint, and --collect-detailed-traces with model, worker or all adds forward and execute times, with a performance impact its own documentation warns about. The attributes it emits are in vllm/tracing/utils.py and are the most useful part of the setup: gen_ai.latency.time_in_queue, time_to_first_token, time_in_scheduler, time_in_model_prefill, time_in_model_decode, time_in_model_forward, time_in_model_execute, time_in_model_inference and e2e, plus the token counters. That breakdown is what turns a “it’s slow” into “the time goes in the queue, not in decode”, which is the conversation you want to have during an incident. It fits with what was already covered in instrumenting vLLM with OTel and with the metrics in DCGM and vLLM.

There are three constraints that decide the architecture:

vLLM exports over gRPC unless OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf is set, and its gRPC exporter is built with insecure=True hard-coded. Langfuse, for its part, does not support gRPC. And vLLM emits the old convention names (gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens), renamed in the specification to input_tokens and output_tokens, plus a gen_ai.latency.* family that was never standard and that the file itself documents as its own attributes until they are normalised.

An OpenTelemetry Collector in between solves all three at once: it receives gRPC, exports HTTP with the Basic header towards /api/public/otel/v1/traces, and renames attributes with a transform. It is also the only place where tail sampling can be done, which is the next point. And add the warning from the llm-d community, which holds for any deployment with Envoy or sidecars in between: OTEL_SERVICE_NAME has to be set per engine, because vLLM sends no service name and without it every span arrives as unknown_service.

The four serial queues

A request’s telemetry crosses four bounded buffers before it is queryable. All four drop once full and none blocks the request.

Queue	Capacity	What it does once full
LiteLLM logging thread pool	100 threads, 10,000 pending tasks	Drops the new ones with a rate-limited warning
Asynchronous `LoggingWorker`	50,000 events, concurrency 100	Aggressively flushes 50 % of the queue; `enqueue()` documents that it never blocks
OTel’s `BatchSpanProcessor`	2,048 spans (`OTEL_BSP_MAX_QUEUE_SIZE`), batch 512, sent every 5 s	Drops spans silently
Langfuse’s `ClickhouseWriter`	Bounded retries	Drops rows after exhausting retries, with no dead-letter queue

The first is documented in LiteLLM’s own code with a sentence that saves half an hour of discussion: logging is best-effort and, once the cap is reached, new entries are dropped with a rate-limited warning instead of being queued forever. The third is a barely visible consequence: LiteLLM builds the BatchSpanProcessor without adjusting max_queue_size, so the OTel SDK defaults govern, and no LiteLLM documentation mentions the OTEL_BSP_* variables that change them. The Langfuse SDK does the same on its own account: it sets batch size and interval, and leaves the queue size at the default.

On top of this comes a fifth loss point that is not a queue: LiteLLM does not call flush() per request on the native route, so a SIGKILL on the pod takes the last batch with it. On Kubernetes, this is an argument for a generous terminationGracePeriodSeconds and a preStop that gives some slack.

None of this is a defect. It is the right design decision for a piece that sits on the critical path of inference. What changes is what can be asserted with the resulting data.

What cannot be promised to the auditor

A dashboard built on this pipeline answers statistical questions well: what is the p95 of time to first token for the model group, which team consumes the most, what proportion of requests fails, how cost per request evolves. That is what it is designed for, and that is what it is good for.

It does not answer completeness questions well, which are the ones an ENS auditor or an ISO/IEC 42001 management system auditor asks. Faced with “show me that I have a record of every interaction with the AI system during March”, the honest answer is that the traces are a best-effort record with four documented points of silent dropping, with no dead-letter queue on the last hop and no durability guarantee on any of them.

The clean separation is the one already proposed in technical controls ENS, 42001 and the AI Act: the regulatory audit record is a different artefact, with a different write path, durability guarantees and retention of its own. In the pair this article deals with, the LiteLLM_SpendLogs table in Postgres is a much better candidate than Langfuse for that function, because it is a transactional write against a relational database, and because it already contains request_id, api_key in hashed form, team_id, organization_id, end_user, model, tokens, startTime, endTime, status and requester_ip_address. Writing the audit event from the same place and in the same transaction as the spend is the sensible route; delegating it to the tracing pipeline is not.

Here an operational tension appears that has to be resolved deliberately. LiteLLM’s performance recommendation for production includes disable_error_logs: true and, in very loaded deployments, disable_spend_logs: true, because provider errors inflate the table and concurrent writes against the same rows produce locks and exhaust the Postgres connections. Disabling spend_logs to gain performance and at the same time leaning on that table for the audit record are incompatible decisions. Above a thousand requests per second or more than ten instances, the way out is use_redis_transaction_buffer: true, which batches the spend updates in Redis before pushing them down to Postgres, watching the litellm_in_memory_spend_update_queue_size and litellm_redis_spend_update_queue_size metrics.

PII: switching off the content before it leaves

An LLM trace contains, by definition, the user’s prompt. In an on-premise deployment for a regulated customer, that content may be exactly what cannot leave the application’s domain, not even towards the observability backend in the same cluster.

The controls on the proxy side, from most global to most fine-grained:

litellm_settings.turn_off_message_logging: true cuts messages and responses in every callback, and keeps the metadata and the spend. The redaction replaces the content with the REDACTED_BY_LITELLM sentinel in messages, choices, reasoning_content, streaming deltas, tool call arguments and in the standard logging object.
The x-litellm-enable-message-redaction: true header enables redaction per request, and LiteLLM-Disable-Message-Redaction: true disables it. The second is checked first and wins over everything else, including the global configuration, which makes it a control to block at the ingress if the policy is that content is never logged.
redact_user_api_key_info: true removes the hashed token, the user_id and the team_id from the log.
Per request and per callback, metadata: {"mask_input": true, "mask_output": true}.
The body parameter "no-log": true disables logging of that call. It is a body parameter, not a header, and it can be disabled globally with global_disable_no_log_param: true so a client cannot make itself invisible.

On the OTel v2 route the policy is inverted towards the safe side: content is off by default and is opened with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, with values no_content, span_only, event_only and span_and_event. The documentation underlines that the gate is applied centrally and that a user’s request cannot force their prompt towards the backend while capture is disabled. It is also the stance the OpenTelemetry specification recommends, whose preferred pattern for production is to store the content in external storage and leave only the reference in the span.

One specific version to check in the inventory: below LiteLLM 1.98.0, with per-team Langfuse credentials, the team’s own credentials ended up inside its traces. That version changed the origin of the metadata emitted to the StandardLoggingPayload to cut it off. The same change removed a couple of dozen metadata fields (model_group, deployment, queue_time_seconds, attempted_retries and others), so it breaks saved dashboards and alerts that filter on them. It is an upgrade with work attached, and it has to be done anyway. The same applies to the general hardening of the stack, covered in hardening and secrets of the sovereign stack.

The price in latency

LiteLLM publishes two overhead figures that differ by a factor of thirty, and both are theirs.

The benchmarks page, with Locust, a thousand users with think time and a fake OpenAI endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms, at 1,170 requests per second. For two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures hold with around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, of July 2026, measures the Python proxy at 257.7 ms of added p99 with 329.5 MB of peak memory, against 0.7 ms for the Rust variant in beta. That measurement has no callbacks, no spend tracking and no persistence.

The operational reading is that the first figure is the best case and the second the order of magnitude to plan for agentic load, which is the one that arrives in a closed loop against fast responses. To watch it there are two always-on headers: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms, and the documentation sets the useful threshold of the second at 100 ms, above which the diagnosis is that the payloads are too large.

Three production settings that appear in LiteLLM’s documentation and that are wrong by default:

request_timeout is 6,000 seconds. A hundred minutes holding a connection against a hung upstream. The recommendation is to lower it to 600.

LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and the documentation puts the cost of that serialisation at 2 to 5 seconds with payloads over 2 MB. It is the first cause of latency their troubleshooting guide mentions.

Background jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every periodic job. The LITELLM_JOB_ROLE variable separates the pods that serve traffic from the replica that runs jobs.

And one failure mode with an open issue that deserves to be in the runbook: under sustained 429s from the upstream, pods have been reported unable to answer readiness probes during startup, with the ensuing restart loop. It fits with what was covered in incident response runbooks.

Start-up checklist

Ten checks before signing off the installation of the pair:

callbacks: ["langfuse_otel"] instead of success_callback: ["langfuse"], unless you depend on prompt management, which the OTLP preset does not support.
LANGFUSE_HOST defined explicitly. The preset’s default value points to the US region.
LiteLLM at 1.98.0 or higher, because of the team credential leak into the traces, and at 1.95.0 or higher because of the v4 ingestion header.
A price registered for every self-hosted model, per token and in USD, verified with /spend/calculate before opening traffic.
A test request with a known traceparent, checking that the same identifier appears in Langfuse.
An OpenTelemetry Collector between vLLM and Langfuse, with OTEL_SERVICE_NAME per engine and renaming of the old token attributes.
request_timeout: 600 and LITELLM_LOG out of DEBUG.
Tail sampling in the Collector, with a latency and status code policy, to keep 100 % of what is slow and what failed.
A written decision about prompt content, and blocking of the LiteLLM-Disable-Message-Redaction header at the ingress if the policy is not to log.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms and Langfuse’s ingestion queue depth (langfuse.queue.ingestion.depth), which is the autoscaling signal for its workers.

Traps and things that are not what they look like

The trace arrives, but ten minutes late. The x-langfuse-ingestion-version: 4 header is missing, which LiteLLM sends from 1.95.0. It is not a ClickHouse capacity problem.

The metadata works in development and disappears in production. Nginx drops headers with underscores by default, and all of this integration’s have one.

Session cost comes out as zero and generation cost does not. Langfuse adds up only the ingested cost in the session totals, not the one it computes itself.

Enabling OTel v2 breaks nothing visible and ungroups the conversations. The v2 mapper does not read session_id or trace_user_id from the request.

A team’s budget is never exhausted. Its spend is zero because the model has no price, and the calculation failure is only logged at debug level.

The alerts stop firing after the migration to v2. The success status becomes UNSET; no rule on OK is satisfied.

Forwarding the traceparent is enabled and it does not reach the engine. The generic otel callback is missing, without which the function that applies it returns before looking at the flag.

All the engines’ spans appear under unknown_service. vLLM emits no service name; OTEL_SERVICE_NAME has to be set per deployment.

Closing

The gateway and the tracing backend are the two pieces of the inference stack that take the least time to install and the longest to operate well. The installation fits in a ConfigMap. The operation consists of four decisions no getting-started guide raises: which integration route to take knowing that one expires in November, where the price per token of a model nobody invoices comes from, what to do with the traceparent at each hop, and what to tell an auditor about data the system drops by design when things get tight.

The answer to the fourth is the one that orders the other three. This pipeline is operational instrumentation, with statistical precision and best-effort guarantees, and separating it from day one from the regulatory audit record avoids the awkward conversation two years from now. With that separation made, the LiteLLM and Langfuse pair is the pair of pieces that turns a fleet of GPUs into a platform with an owner, a known cost and explainable latency.

Sources

LiteLLM, Langfuse integration (rutas de integración, metadata aceptada, cabeceras langfuse_*): https://docs.litellm.ai/docs/observability/langfuse_integration.
LiteLLM, OpenTelemetry v2 y Migración a OTel v2 (preset, captura de contenido, cambios de semántica): https://docs.litellm.ai/docs/observability/opentelemetry_v2 · https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.
LiteLLM, Logging y Config settings (redacción, turn_off_message_logging, variables de entorno): https://docs.litellm.ai/docs/proxy/logging · https://docs.litellm.ai/docs/proxy/config_settings.
LiteLLM, Custom pricing y Spend tracking (claves de precio, SpendLogs, endpoints de gasto): https://docs.litellm.ai/docs/proxy/custom_pricing · https://docs.litellm.ai/docs/proxy/cost_tracking.
LiteLLM, Production best practices (topología, request_timeout, buffer de transacciones en Redis, LITELLM_JOB_ROLE): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Benchmarks y Latency overhead (cifras de sobrecarga y cabeceras de diagnóstico): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, Rust AI Gateway benchmarks (la cifra de 257,7 ms de p99 del proxy Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, notas de versión v1.95.0, v1.98.0 y v1.100.0: https://docs.litellm.ai/release_notes/v1.95.0/v1-95-0 · https://docs.litellm.ai/release_notes/v1.98.0/v1-98-0 · https://docs.litellm.ai/release_notes/v1.100.0/v1-100-0.
Langfuse, OpenTelemetry (endpoint, autenticación, tabla de mapeo de atributos, ausencia de gRPC): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, Compatibility y Migrate v3 to v4 (la fecha del 16 de noviembre de 2026 y los modos de escritura): https://langfuse.com/docs/compatibility · https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4.
Langfuse, Token and cost tracking (prioridad del coste ingerido sobre el inferido): https://langfuse.com/docs/observability/features/token-and-cost-tracking.
Langfuse, Scaling y Sampling (dimensionado, señal de autoescalado, muestreo de cabecera): https://langfuse.com/self-hosting/configuration/scaling · https://langfuse.com/docs/observability/features/sampling.
Langfuse, issue #13468 — descarte de filas en ClickhouseWriter sin cola de mensajes muertos: https://github.com/langfuse/langfuse/issues/13468. Issue #15109 — coste de sesión a cero: https://github.com/langfuse/langfuse/issues/15109. Issue #5173 — Nginx y cabeceras con guión bajo: https://github.com/langfuse/langfuse/issues/5173.
vLLM, vllm/tracing/utils.py y vllm/tracing/otel.py en la etiqueta v0.28.0 (atributos emitidos, protocolo por defecto, insecure=True): https://github.com/vllm-project/vllm/blob/v0.28.0/vllm/tracing/utils.py.
Langfuse, vLLM integration (la advertencia de que vLLM solo exporta contadores y latencias): https://langfuse.com/integrations/model-providers/vllm.
LiteLLM, litellm/constants.py y litellm/litellm_core_utils/logging_worker.py (capacidades de las colas y política de descarte): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
OpenTelemetry, Semantic Conventions for Generative AI y el repositorio semantic-conventions-genai (estado de desarrollo, renombrados, captura de contenido): https://github.com/open-telemetry/semantic-conventions-genai.
OpenTelemetry Collector, tailsamplingprocessor (políticas de muestreo por cola y la restricción de instancia única por traza): https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md.
llm-d, End-to-end and fine-grained tracing in llm-d (OTEL_SERVICE_NAME y los spans huérfanos tras Envoy): https://llm-d.ai/blog/end-to-end-and-fine-grained-tracing-in-llm-d.

Choosing the switchboard: which OSS gateway to put in front, and why the licence decides before the features

Sat, 06 Jun 2026 08:30:00 +0200

This post is the decision companion to The LLM inference router. That one explained what an inference router is and why it exists (catalogue, traffic splitting, cross-cutting policy, prefix-aware routing). This one answers the question that actually blocks a deployment: which one to choose, with verified licences and an order of criteria that is not the one most people use.

TL;DR

The gateway is the piece 100 % of the traffic passes through and the most expensive to replace later: the client SDK, the auth and rate-limit policies, the tracing and the model catalogue all couple to it. That is why the order of the criteria matters, and it is not the usual one. First the licence: not “is it open source?” but “are the features I need under a permissive licence or behind an Enterprise wall?”. Second the architectural fit: is it a citizen of your Kubernetes (Gateway API) or a separate process you have to operate on the side? Third, maturity and documentation. Fourth, the features. With data verified as of June 2026: LiteLLM is MIT in the core, but SSO, audit logs, fine-grained RBAC and several guardrails are Enterprise; Kong has an Apache 2.0 core but the AI plugins that matter (semantic cache, prompt guard, AI proxy advanced) are gated; Envoy AI Gateway, Gateway API Inference Extension (GIE), Higress, APISIX and Bifrost are Apache 2.0 end to end. For an RKE2 + vLLM stack with a K8s-native priority (the case we assume here), the recommendation is to adopt the Gateway API Inference Extension model, the Endpoint Picker that routes with awareness of prefix cache, KV and LoRA, which is exactly what multiplies the hit rate. Implement it with Envoy AI Gateway if the AI-native trajectory weighs more, or with Higress if today’s maturity weighs more; and put LiteLLM (MIT) behind it as an optional multi-provider control plane. With explicit scepticism about v0.x releases, about “OpenAI-compatible ≠ inference-aware” and about phone-home telemetry.

The principle that reorders the criteria: the gateway is a marriage, not a date

Some pieces of the stack can be swapped on a Sunday afternoon: the embedding model, the reranker, even the inference engine behind an OpenAI-compatible interface. The gateway cannot. It is the piece everything else couples to: every client SDK points at its URL, the security policies live in it, the tracing is born in it, it defines the model catalogue. Ripping it out two years later means touching every consumer at once. It is the infrastructure decision with the highest cost of reversal in the whole serving layer.

When a decision is expensive to reverse, the dominant criterion is not “what does it do today?” but “can I own it and operate it for years without surprises?”. And that puts the licence ahead of the features. A brilliant tool whose SSO, audit log or RBAC live behind an Enterprise contract is a tool that, the day your ENS deployment needs those controls, forces you to pay or to migrate, which is exactly the scenario the choice was supposed to avoid. That is why this post’s filter order is licence → fit → maturity/docs → features, and not the other way round.

The field of candidates (June 2026)

The real OSS pieces anyone would put in front of an on-premise vLLM fleet:

Candidate	What it is	Language
LiteLLM Proxy	OpenAI-compatible gateway, 100+ providers, virtual keys, spend	Python (FastAPI)
Envoy AI Gateway	AI layer on top of Envoy Gateway; integrates GIE/EPP, InferencePool	Go (Envoy)
Gateway API Inference Extension (GIE)	K8s-SIG extension: inference-aware Endpoint Picker	Go
Higress	API gateway + AI plugins, Envoy/Istio, CNCF	Go/C++
Apache APISIX	Mature API gateway with AI plugins	Lua/Nginx
Kong (+ AI plugins)	API gateway; Apache 2.0 core, Enterprise AI plugins	Lua/Nginx
Bifrost	High-performance AI-first gateway	Go
vLLM Production Stack / Semantic Router	vLLM-specific router, KV/prefix/intent-aware	Go/Python

Filter 1 — Licence: where the small print is

This is where half the options die, and where “open source” misleads if you do not look at the detail. What has been verified:

Candidate	Core licence	What is gated (paid)
LiteLLM	MIT	SSO (Okta/Azure AD), audit logs, JWT auth, fine-grained RBAC, several guardrails (llmguard, llamaguard, prompt-injection) → Enterprise (~250 USD/month and rising)
Envoy AI Gateway	Apache 2.0	— (full AI layer is OSS)
GIE (Inference Extension)	Apache 2.0	— (K8s-SIG project)
Higress	Apache 2.0	— (AI plugins included)
APISIX	Apache 2.0	— (more built-in AI than Kong OSS)
Kong	Apache 2.0 (core)	AI Semantic Cache, AI Prompt Guard, AI Proxy Advanced, AI RAG Injector → Enterprise
Bifrost	Apache 2.0	—

Two conclusions that settle the field:

Kong falls out for an OSS AI deployment. Its core is Apache 2.0, but precisely the AI plugins that would justify choosing it (semantic cache, prompt guard, proxy advanced) are Enterprise, with contracts that the sector’s own comparisons put above five figures a year. For AI on an OSS budget, APISIX offers more built-in at no cost. Kong remains excellent as a classic API gateway; as an OSS AI gateway, no.
LiteLLM really is MIT in the core, and that is genuine: you can fork it, modify it and use it commercially. But SSO, audit logs, fine-grained RBAC and several guardrails are Enterprise. Under the criterion we set (“a permissive core is enough, governance is solved with OIDC and external auditing in the stack”) LiteLLM stays in play; under an “all OSS or nothing” criterion it would be wounded. It pays to know exactly what falls on which side of the line before committing.

LiteLLM: what is gated and its OSS equivalent

The nuance that decides whether LiteLLM’s gating is a real problem or not: “paid built-in feature” is not the same as “capability impossible in OSS”. In almost every case the capability is achieved by changing the how, wiring OSS through LiteLLM’s open hook, or solving it in the layer next door (the gateway, the observability). The breakdown, verified against the docs:

Capability	In LiteLLM Enterprise	OSS equivalent
Guardrails	prebuilt integrations (`llmguard`, `llamaguard`, `lakera`, `aporia`, `hide_secrets`)	the framework + custom guardrail hook + Presidio are OSS (MIT core); you invoke LLM Guard, Llama Guard or Presidio from the hook yourself
Audit log	turnkey UI with retention policies	request/response logging + custom callbacks + OTel/Langfuse exporters (official OSS integration) → you build the trail and you own it
RBAC	fine-grained (`enforce_rbac`, org/team/user roles)	coarse (virtual keys per team/budget/model) is OSS; fine-grained is done at the edge in the gateway (`ext_authz` + OPA)
SSO	Admin UI SSO (Okta/Azure/Google/OIDC)	user/API SSO is solved by fronting with OSS OIDC (Keycloak + oauth2-proxy) or in the gateway itself; JWT auth + JWT→virtual-key mapping are already in the core

The conclusion that closes the decision: in a K8s-native design with a gateway in front, OIDC and authorisation live in the gateway (Envoy/Higress + OPA) and the audit trail in Langfuse/OTel, both Apache 2.0 and already in the stack, so LiteLLM placed behind barely needs its Enterprise tier: the governance pieces sit where they architecturally belong and happen to be OSS. The only real cost is the DIY integration and maintenance, and, for ENS, designing the guarantees of the audit trail (retention, immutability/WORM) is on you, it does not come out of the box.

The rest, Envoy AI Gateway, GIE, Higress, APISIX and Bifrost, pass the licence filter clean: Apache 2.0 end to end, with no critical feature behind a wall.

Filter 2 — Fit: Kubernetes citizen or tenant?

With a K8s-native priority (RKE2), the question is whether the gateway is modelled as cluster resources, versioned with GitOps, observed with the same tools and integrated with the scheduler, or whether it is a separate process you have to operate on the side. This is where the structural novelty of 2025-2026 appears: the Gateway API Inference Extension (GIE).

GIE is an extension of the standard Kubernetes Gateway API, from SIG-Network, which adds two pieces: the InferencePool (a pool of replicas of a model as a native resource) and the Endpoint Picker (EPP), a scheduler that decides which replica each request goes to according to the inference state: queue length, available LoRA adapters, and, the key piece, the prefix cache state of each replica. It is exactly the prefix-aware routing that the router post explained as the reason hit rate goes from 5-15 % to 60-85 %, now as a community standard instead of a proprietary feature of each product.

Why this settles the decision when the priority is K8s-native: GIE turns inference-aware routing into a standard interface that several implementations satisfy (Envoy AI Gateway, Higress, kgateway, Istio, GKE Inference Gateway). You pick an implementation today and, if you change it tomorrow, the resource model (InferencePool, HTTPRoute) is preserved. That is precisely the property you wanted from a decision that is expensive to reverse: the coupling is to a standard, not to a product. A standalone proxy like LiteLLM, however good it is, lives outside this model. It is one more Deployment, with its own config, its own catalogue format and its own management plane.

Filter 3 — Maturity and documentation: the uncomfortable honesty

Here comes the tension no marketing comparison admits: what is K8s-native-correct and what is battle-tested-today do not quite coincide in June 2026.

Envoy AI Gateway: the AI layer is at v0.5, pre-1.0 and young. But it runs on Envoy and Envoy Gateway, which are among the most mature things that exist in proxies. It has the most complete GIE/EPP integration, model virtualisation and OpenInference tracing. Risk: the AI layer still moves fast (breaking changes are possible). Docs are growing, and good.
GIE: the project is reaching GA (one of the first standardised inference-aware extensions). Standard maturity is high and rising; the maturity of each implementation varies.
Higress: the most mature today among the K8s-native Apache-2.0 options with AI. Production at Alibaba scale, CNCF, Envoy/Istio base, AI plugins included, Gateway API support. If “mature” weighs more than “cutting-edge AI-native”, it is the safe bet.
APISIX: a very mature classic gateway, with growing AI plugins; less specialised in inference-aware routing (prefix/KV) than the GIE model. Battle-tested at massive scale.
LiteLLM: the one with the best documentation and the widest adoption in the field, by a distance. Mature as a proxy. Its ceiling is the language (Python/FastAPI: ~250-300 RPS per instance, scaling out by replicas) and the governance gating already covered.
Bifrost: Apache 2.0, Go, the highest performer (~11 µs overhead at 5,000 RPS), with semantic caching and governance built in. Younger and less proven over years, but technically strong.

Decision matrix

Weighting the four filters for the case of RKE2 + vLLM, K8s-native priority, permissive core acceptable:

Candidate	Licence	K8s-native	Maturity	Docs	Inference-aware	Verdict
Envoy AI GW + GIE/EPP	✅ Apache 2.0	✅✅ standard	⚠️ v0.5 (mature Envoy)	✅ good	✅✅ prefix/KV/LoRA	Primary (trajectory)
Higress (+ GIE)	✅ Apache 2.0	✅✅	✅✅ production	✅ good	✅ via GIE/plugins	Primary (maturity today)
APISIX	✅ Apache 2.0	✅	✅✅	✅✅	⚠️ less specialised	Solid alternative
LiteLLM	⚠️ MIT core, gov. gated	❌ separate proxy	✅✅	✅✅✅	⚠️ basic	Control plane behind
Bifrost	✅ Apache 2.0	⚠️	⚠️ young	✅	✅	One to watch (perf)
Kong	⚠️ AI plugins Enterprise	✅	✅✅	✅✅	❌ gated	Ruled out (OSS AI)

The recommendation

For the case we fixed, RKE2 + vLLM, K8s-native priority, permissive licence with a sufficient core, the recommendation has two layers, not one:

Data layer (what sits in front of the replicas): adopt the Gateway API Inference Extension model. It is the decision that ages well because it couples you to an Apache 2.0 standard, not to a product, and because it brings the prefix/KV/LoRA-aware routing that really moves the hit rate. For the implementation:

If the AI-native trajectory weighs more and you accept operating a v0.x layer on top of a mature Envoy: Envoy AI Gateway + GIE/EPP. That is where the cutting edge and the most complete integration are.
If today’s maturity outweighs everything else (ENS, critical production, zero appetite for v0.x in the path of 100 % of the traffic): Higress with GIE. CNCF, proven at scale, Apache 2.0, and migratable to the standard EPP.

Control layer (optional, behind): LiteLLM (MIT) if you need a single multi-provider point with virtual keys, spend tracking and fallbacks to external models. It goes behind the data gateway, not in its place, and you accept that SSO, audit and RBAC are solved with the cluster’s own OIDC and auditing (as we set out in the criterion). If your front end is only on-premise vLLM with no external providers, this layer is probably unnecessary.

What I would not recommend here: Kong (its AI features are gated, which contradicts the licence criterion); and starting with LiteLLM as the main data gateway in a stack that will be multi-replica and K8s-native, since its Python ceiling and its position outside the Gateway API model turn it into debt the day you scale.

Applied to our infrastructure: RKE2 + vLLM + 4×H100

The concrete deployment on the generic reference cluster:

# The InferencePool groups the replicas of a model (GIE resource)
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata: { name: llama-70b-pool }
spec:
 selector: { app: vllm-llama70b }
 extensionRef: { name: llama-70b-epp }  # the Endpoint Picker
---
# The standard Gateway API route points at the pool
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: llama-70b }
spec:
 parentRefs: [ { name: ai-gateway } ]
 rules:
 - backendRefs:
 - group: inference.networking.x-k8s.io
 kind: InferencePool
 name: llama-70b-pool

Three notes on how this fits the blog series:

It lives in layer 1 of the seven-layer stack, in front of the replicas pinned by the RKE2 resource managers. The gateway routes; the kubelet pins; both are resources of the same GitOps cluster.
The EPP reads the prefix cache state of each replica, which connects directly with the engineering of prefix cache hit rate: the gateway is what turns that affinity into actual routing.
The gateway’s gen_ai.* tracing feeds the OTel observability pipeline and lands in Langfuse, closing the loop of the Observe layer.

Traps and things that are not what they seem

“It is open source” without reading what falls under Enterprise. The mistake this post tries to avoid: Kong and LiteLLM are OSS, but the features that justify choosing them for AI (AI plugins in Kong; SSO/audit/RBAC/guardrails in LiteLLM) are partly or wholly gated. “Open source” is the wrong question; “is what I need permissive?” is the right one.

“OpenAI-compatible” ≠ “inference-aware”. Almost all of them expose the OpenAI API. That does not mean they understand the KV cache state, the queue of each replica or the LoRA adapters. API compatibility is table stakes; intelligent routing (EPP) is a different league. Do not confuse “speaks OpenAI” with “routes well”.

Choosing by the RPS benchmark. Bifrost wins on raw overhead, but the bottleneck of an on-premise vLLM fleet is rarely the proxy: it is the GPUs. Optimising the gateway for 5,000 RPS when your replicas serve 300 req/s is solving the problem you do not have. The latency the gateway adds matters; its peak throughput almost never does, in this context.

Phone-home telemetry in a sovereign deployment. Several gateways send telemetry home by default. In an ENS or sovereignty context, that is an audit finding, not a detail. Verify and disable the phone-home before putting the gateway in the path of 100 % of the traffic; document it for the ENS/42001/EU AI Act controls file.

Betting on a v0.x in the critical path with no rollback plan. Envoy AI Gateway v0.5 is promising, but it sits in the path of 100 % of the traffic. If you choose it, have the rollback rehearsed and follow the changelog: v0.x releases break things. The GIE standard mitigates this (you can change implementation), but the specific implementation has to be operated with a safety net.

Confusing the data gateway with the control plane. LiteLLM in front of everything seems to simplify, but it mixes two roles: routing data traffic (where you want K8s-native + inference-aware) and managing multi-provider keys and spend (where LiteLLM shines). Separate them: a GIE-native data gateway, with LiteLLM as a control plane behind it if needed.

Conclusion

The gateway is the piece with the highest cost of reversal in the serving layer, and that forces you to invert the intuitive order of criteria. First the licence, not “is it OSS?” but “is what I need permissive or is it gated?”, which already rules Kong out for OSS AI and puts an asterisk on LiteLLM for its Enterprise governance. Then the fit, where the Gateway API Inference Extension turns inference-aware routing into an Apache 2.0 standard you can couple to without marrying a product. And only after that maturity, docs and features, where honesty forces you to admit that the cutting-edge AI-native option (Envoy AI Gateway v0.5) and the battle-tested-today one (Higress) are still not the same thing. For an RKE2 + vLLM stack with a K8s-native priority, the answer is not a product but a model, GIE/EPP, implemented with Envoy AI Gateway if you look at the trajectory or with Higress if you look at maturity, and LiteLLM as an optional control plane behind. The decision that ages well is the one that couples you to a standard, not to a vendor; in gateways, in June 2026, that standard finally has a name.

References

Kubernetes, Introducing Gateway API Inference Extension (blog oficial, jun 2025): https://kubernetes.io/blog/2025/06/05/introducing-gateway-api-inference-extension/.
Gateway API Inference Extension, doc del proyecto (Endpoint Picker, InferencePool): https://gateway-api-inference-extension.sigs.k8s.io/.
Envoy AI Gateway, sitio y release notes (v0.x, GIE integration): https://aigateway.envoyproxy.io/.
LiteLLM, Enterprise Features (qué está gated): https://docs.litellm.ai/docs/proxy/enterprise · licencia MIT: https://github.com/BerriAI/litellm/blob/main/LICENSE.
Kong, Announcing Kong’s Open Source AI Gateway y matriz de plugins Enterprise: https://konghq.com/blog/product-releases/announcing-kong-ai-gateway.
Higress (Apache 2.0, CNCF, AI gateway): https://higress.cn/en/.
Apache APISIX, APISIX vs Kong (cobertura de IA OSS): https://apisix.apache.org/learning-center/apisix-vs-kong/.
vLLM, Semantic Router v0.1 Iris y Production Stack (routing específico de vLLM): https://blog.vllm.ai/2026/01/05/vllm-sr-iris.html.

Incident response runbooks for LLM inference: every alert to a concrete action with Kafka and Keep

Tue, 02 Jun 2026 04:30:00 +0200

This post closes the observability trilogy opened by GPU observability for LLM inference (which metrics) and Anatomy of the twelve DCGM and five vLLM metrics (which documented anomaly per metric). Here each anomaly gets its concrete action and is fitted into the incident management machinery that compliance demands.

TL;DR

The GPU observability alerts are useless without a codified procedure for each one; the operator who interprets them by hand every time is operating on intuition. The right combination has three indispensable pieces. (1) Runbook catalogue: for each of the six critical alerts (GpuHbmNearOom, GpuThermalOrPowerThrottle, GpuXidErrorDetected, GpuEccDoubleBit, VllmKvCachePoolNearFull, VllmTtftP95OutOfSlo), a severity, an immediate mitigation, the evidence to capture before remediating, a resolution action, a closure criterion and a postmortem trigger. (2) Reproducible pipeline: Prometheus + DCGM → Alertmanager → Kafka as event bus (topics gpu.alerts.enriched, incidents.lifecycle, audit.actions with WORM retention) → Keep as workflow engine (declarative YAML workflows versioned in git) → Kubernetes job / script / ChatOps executors. (3) Formal fit into incident management under the regulatory corpus: ISO/IEC 27035 phases identify → report → assess → respond → learn; ENS controls op.exp.7 (incident management), op.exp.8 (activity logging), op.exp.10 (notification to users); NIS2 art. 23 with an early warning at 24 h, a formal notification at 72 h and a final report at 1 month; EU AI Act art. 73 for a serious incident on a high-risk system, deadlines of 2 to 15 days depending on severity; ISO/IEC 42001 clause 10 (continual improvement of the AIMS). The action taxonomy is immediate mitigation (drain, throttle, scale-down: contains the damage in seconds) → diagnosis (evidence capture with nvidia-smi -q, dmesg, a vLLM /metrics snapshot, the related OTel trace; without this the postmortem is not defensible) → resolution (restart, reset, RMA, rollback) → postmortem (5-whys RCA, prevention plan, runbook update). Kafka contributes the immutable audit trail that ENS and the EU AI Act demand: every action executed by Keep or by a human is published as an event on audit.actions with timestamp, actor, decision and evidence, retained WORM for at least 6 months. Keep contributes workflows as code: this post includes three complete workflows (XID with drain plus Jira ticket, ECC DBE with immediate paging and node blocking in the scheduler, automatic canary rollback on TTFT P95 out of SLO). Four anti-patterns close the material: alerts without a runbook (most of them), a runbook without prior evidence capture (it perpetuates the incident because the root cause is lost), escalation by seniority rota instead of severity (a junior operator handling an ECC DBE), and the absence of a human gate for destructive actions (Keep running nvidia-smi --gpu-reset without confirmation). Applicable to a generic 4×H100 SXM cluster with Kafka and Keep already deployed.

You are here: OBSERVE → DEPLOY (incident response closes the loop)

The analogy: a nuclear reactor control room

In a nuclear plant control room, the operator on shift never decides what to do on seeing an alarm. The decision is pre-made and codified in a written procedure (SOP) covering every alarm on the panel: if X sounds, open book X, read steps 1-N, execute exactly, call the supervisor at step M, escalate to the plant director at step N+3. The reason is strict: critical alarms are rare but catastrophic if handled badly; an operator improvising in an emergency makes worse decisions than one applying a procedure reviewed by experts and validated by simulation.

The reactor does not expect the operator to be a genius. It expects them to know the procedures to the letter and the operations management system to hand them the right procedure at the right moment. If the procedures are not written, not versioned, or not integrated with the alarms that fire them, the control room is operating on intuition. The difference between the two operations, procedural versus intuitive, is the difference between a plant that runs 30 years without incidents and one that ends up on a blacklist.

Incident response on an LLM inference cluster works identically. The DCGM and vLLM alerts listed in the previous posts are the panel alarms. Each one needs its written SOP, versioned, integrated with the alert that fires it and reviewed after every incident. Without that codification, the operator on shift improvises in the middle of an ECC DBE failure at 4 in the morning; with it, they execute the nine steps of runbook 12 and the incident closes in 20 minutes.

The incident pipeline architecture

Prometheus + DCGM. Collects the metrics described in the two previous posts. PrometheusRules define the six critical alerts with for: <duration> to avoid noise.

Alertmanager. Receives raw alerts; deduplicates, groups by labels ({cluster, node, gpu, model}), routes. Instead of sending straight to PagerDuty or Slack, it sends to Kafka via a webhook receiver. That turns the alert into a bus event that multiple consumers process (Keep for action, the audit topic for compliance, dashboards for visualisation).

Kafka as event bus. Three canonical topics:

gpu.alerts.enriched — alerts with added context (tenant, model, version, namespace owner, effective severity). Retention: 7 days, replication factor 3.
incidents.lifecycle — incident cycle events: incident.opened, incident.acknowledged, action.proposed, action.executed, incident.escalated, incident.resolved, postmortem.attached. Retention: 90 days.
audit.actions — an immutable record of every action executed (by Keep automatically or by a human confirming). Retention: 6 months minimum with compaction off plus tiered storage, WORM storage. This is the topic ENS op.exp.8, EU AI Act art. 12 and NIS2 require you to keep.

Keep as workflow engine. Consumes from gpu.alerts.enriched, fires YAML workflows versioned in git, executes actions (HTTP calls, kubectl jobs, Slack messages, Jira tickets) and publishes the result to incidents.lifecycle plus audit.actions. Choosing Keep over Alertmanager alone (or over PagerDuty alone) is deliberate: Keep separates runbook declaration (readable, reviewable YAML) from notification delivery (PagerDuty). The runbook is versioned code; notifications are operational details.

Executors. What actually moves the cluster:

Kubernetes jobs: kubectl drain, kubectl cordon, kubectl rollout undo.
NVIDIA API: nvidia-smi --gpu-reset, dcgmi diag -r <level>.
ChatOps: human confirmations through Slack interactive messages before a destructive action runs.
External tooling: Jira ticket, PagerDuty notification, CMDB call.

The six critical alerts and their runbooks

For each alert: severity, immediate mitigation (seconds), evidence to capture before remediating, resolution action, closure criteria, postmortem trigger.

RB-01 · `GpuHbmNearOom` — HBM > 92 % sustained

Severity: WARNING. Risk of OOM on the next PagedAttention allocation.

Immediate mitigation. Reduce admission temporarily by lowering max_num_seqs on the affected engine via hot reload (if the engine supports it) or a staggered restart of replicas. Trigger an additional scale-out via KEDA if there are free GPU nodes. There is no need to drain the node.

Evidence to capture.

nvidia-smi --query-gpu=index,memory.used,memory.free,memory.total --format=csv
nvidia-smi -q -d ROW_REMAPPER | grep -i pending
curl http://vllm-pod:8000/metrics | grep -E "gpu_cache_usage|num_requests"
kubectl logs <pod> --tail=200 | grep -i "preempt\|swap"

Save the snapshot to audit.actions with a timestamp and an incident_id.

Resolution. If the cause is a traffic spike: let the autoscaler scale to a stable regime, monitor for 30 min. If the cause is a model regression (canary v2 consumes more KV cache than v1): roll the canary back (see RB-06). If it is a leak (the metric grows without traffic growing): restart the pod with a heap dump capture.

Closure. gpu_cache_usage_perc < 80 % sustained for 15 min AND num_requests_waiting == 0.

Postmortem. Not mandatory unless the incident lasted > 30 min or had SLO impact.

RB-02 · `GpuThermalOrPowerThrottle` — bit ≠ 0 and not Idle in CLOCK_THROTTLE_REASONS

Severity: WARNING (thermal) or CRITICAL (sustained HW Power Brake, PDU risk).

Immediate mitigation. Identify the bit (decode the bitmap). If it is 0x40 HW_THERMAL or 0x20 SW_THERMAL: drain the workload from the node to other replicas if the temperature does not drop within 2 min, and stop new pods landing on that node (kubectl cordon). If it is 0x80 HW_POWER_BRAKE: alert DC infrastructure immediately (likely an over-committed PDU, the Dell KB 000220508 / Lenovo HT514380 case) and lower the TDP of the rack’s GPUs via nvidia-smi -pl to a lower value to relieve load on the breaker.

Evidence.

nvidia-smi --query-gpu=index,temperature.gpu,temperature.memory,power.draw,clocks_throttle_reasons.active --format=csv
ipmitool sdr | grep -i "fan\|temp\|inlet"
# PDU data if they are instrumented (modbus / SNMP)

Resolution.

Thermal: review rack airflow, check the rear-door HX, T_inlet, DGX fans. An infra issue, not an engine one.
Power Brake: review the sizing of the PDU branch, the breaker, the 415 VAC distribution. Likely a redistribution of load to another branch or a temporary TDP cap.

Closure. CLOCK_THROTTLE_REASONS == 0x1 (Idle only) or 0x0 for 30 min under normal load.

Postmortem. Mandatory if it was a HW Power Brake: that implicates the DC electrical infrastructure.

RB-03 · `GpuXidErrorDetected` — `increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0`

Severity: CRITICAL.

Immediate mitigation. kubectl cordon the node (no more new pods). If the XID is 31/48/79/94/95 (hardware or cascade): drain the node’s existing pods. If the XID is 13/43 (possibly software): keep the pods but block new ones, capture the trace and the active workload.

Evidence.

# The specific XID from dmesg
dmesg | grep -i xid | tail -30
nvidia-smi -q -d ERROR
nvidia-smi -q -d PCIE
# State of the retired pages
nvidia-smi -q -d ROW_REMAPPER
# Workload that was running
kubectl get pods -o wide | grep <node>
kubectl logs <pod> --previous --tail=500

Resolution.

XID 13/43 (software exception / channel verif): if it recurs only with one specific model, it is a workload bug; raise an issue with the model team. If it is transient, restarting the pod is enough.
XID 31 (MMU fault): usually a cascade from a previous XID 48. Reset the GPU (nvidia-smi --gpu-reset -i <index>) or reboot the node if the reset does not resolve it.
XID 48 / 95 (DBE / uncontained ECC): see RB-04. The node goes into quarantine.
XID 79 (fallen off the bus): reboot the node. If it recurs after the reboot, open an RMA for the GPU. ByteDance reports 43 % co-occurrence with PCIe errors, so check the slot and the cable too.
XID 94 / 145 / 149: catalogued in NVIDIA’s Xid Catalog with a specific procedure.

Closure. Node smoke test passed (dcgmi diag -r 3), 24 h with no new XIDs, back into the pool.

Postmortem. Mandatory. Include the specific XID, the distribution of XIDs across the cluster, an updated MTBE.

RB-04 · `GpuEccDoubleBit` — `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0`

Severity: CRITICAL, data corruption in progress.

Immediate mitigation. Drain the node immediately without waiting for further evidence. Page (PagerDuty / OpsGenie) the primary ON-CALL. Mark the node unschedulable and failed. XID 48 has a 100 % probability of killing the running job according to the Story of Two GPUs dataset; any inference already in flight is compromised.

Evidence (in parallel with the mitigation).

nvidia-smi -q -d ECC
nvidia-smi -q -d ROW_REMAPPER # Pending: Yes expected
dmesg | grep -E "Xid.*48|DBE|double-bit" | tail -50
# Full capture of the GPU state
dcgmi diag -r 4 -i <gpu_index>

Resolution. Full GPU reset (nvidia-smi --gpu-reset) or a node reboot if the reset does not complete. The reset activates the row remap. After the reboot:

nvidia-smi -q -d ROW_REMAPPER # Pending: No expected
nvidia-smi -q -d ECC # volatile counters at 0

If RETIRED_DBE > 8 pages after the remap: schedule a GPU replacement in the next window, because silicon degradation is progressive. The published real case documents ~19 hours of downtime as typical.

Closure. Node back in the pool after 48 h with no new DBEs.

Postmortem. Mandatory. If the incident affected a request carrying personal or classified data, evaluate notification to the DPO under GDPR art. 33 (it is not necessarily a breach, but it has to be assessed).

RB-05 · `VllmKvCachePoolNearFull` — `gpu_cache_usage_perc > 95 %` sustained 3 min

Severity: WARNING (risk of preempt-on-OOM, not of a real OOM).

Immediate mitigation. Activate autoscaler scale-out by lowering the KEDA threshold temporarily (from 0.85 to 0.75) for 30 min. In recompute mode, preempts raise TTFT but do not break requests; acceptable in the short term. In swap mode latency goes through the roof, so it is better to cut new traffic (return 503 from the router) for 5 min.

Evidence.

curl http://vllm-pod:8000/metrics | grep -E "gpu_cache|num_requests|num_preemptions"
kubectl get hpa vllm-llama70b
kubectl logs <pod> --tail=200 | grep -i preempt

Resolution. If it recurs regularly: revisit capacity planning, possibly lowering max_num_seqs or raising the stable replica count. See Capacity planning.

Closure. Pool < 85 % sustained 30 min, no preempts in the last 15 min.

Postmortem. Not mandatory unless it recurs > 3 times / week.

RB-06 · `VllmTtftP95OutOfSlo` — TTFT P95 > 1.5 s for 5 min

Severity: CRITICAL (contractual SLO violation).

Immediate mitigation. Quick diagnosis of the regime (in order of likelihood):

If a v2 canary is active and the ratio ttft_p95(v2)/ttft_p95(v1) > 1.30: automatic rollback of the canary via Argo Rollouts (argo rollouts abort vllm-llama70b).
If num_requests_waiting > 5: scale out via KEDA.
If DRAM_ACTIVE > 90 % plus gpu_cache_usage_perc > 90 %: HBM bottleneck, reach for the quantisation lever or a context reduction.
If CLOCK_THROTTLE_REASONS != 0: see RB-02.

Evidence.

# Histogram snapshot
curl http://vllm-pod:8000/metrics | grep time_to_first_token
# Breakdown by version if a canary is running
# DCGM state at that moment
curl http://dcgm-exporter:9400/metrics | grep -E "PIPE_TENSOR|DRAM_ACTIVE|THROTTLE"
# Active traffic
kubectl top pods -n inference

Resolution. Depends on the diagnosis. Typical cases:

Canary regression → full rollback (see Canary).
Capacity saturation → scale replicas or accept a temporary 503 with Retry-After.
Prefill bound → enable or calibrate chunked prefill or disaggregated serving (see Disaggregated serving).

Closure. TTFT P95 inside SLO sustained 30 min.

Postmortem. Mandatory. Document the root cause and the lever applied; update the runbook.

Keep YAML workflows — three complete examples

Runbooks are only useful if they are codified in the workflow engine. Keep lets you declare them in YAML versioned in git.

Workflow 1 — `xid-detected.yaml`

workflow:
 id: xid-detected-drain
 name: "XID error detected — cordon node and capture evidence"
 description: "RB-03 implementation"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: GpuXidErrorDetected
 steps:
 - name: capture-evidence
 provider:
 type: bash
 with:
 command: |
 set -e
 NODE="{{ alert.labels.node }}"
 GPU="{{ alert.labels.gpu }}"
 INC_ID="{{ alert.fingerprint }}"
 mkdir -p /var/evidence/$INC_ID
 kubectl debug node/$NODE -it --image=nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04 -- \
 bash -c "nvidia-smi -q -d ERROR,PCIE,ROW_REMAPPER > /host/var/evidence/$INC_ID/smi.txt"
 kubectl describe node $NODE > /var/evidence/$INC_ID/node.txt
 - name: cordon-node
 provider:
 type: kubernetes
 with:
 action: cordon
 name: "{{ alert.labels.node }}"
 if: "{{ alert.labels.severity == 'critical' }}"
 actions:
 - name: open-jira-ticket
 provider:
 type: jira
 config: "{{ providers.jira-prod }}"
 with:
 project: GPUOPS
 issuetype: Incident
 summary: "RB-03: XID {{ alert.annotations.xid_code }} on {{ alert.labels.node }}/{{ alert.labels.gpu }}"
 description: |
 Severity: {{ alert.labels.severity }}
 XID: {{ alert.annotations.xid_code }}
 Evidence: /var/evidence/{{ alert.fingerprint }}
 Runbook: https://runbooks.example.local/RB-03
 - name: notify-slack
 provider:
 type: slack
 config: "{{ providers.slack-gpu-incidents }}"
 with:
 message: |
 :warning: *RB-03 triggered*
 Node: `{{ alert.labels.node }}` GPU: `{{ alert.labels.gpu }}`
 XID: `{{ alert.annotations.xid_code }}`
 <{{ jira.url }}|Jira ticket>
 - name: emit-audit
 provider:
 type: kafka
 config: "{{ providers.kafka-audit }}"
 with:
 topic: audit.actions
 message:
 incident_id: "{{ alert.fingerprint }}"
 action: "cordon_node"
 actor: "keep-workflow"
 workflow_id: "xid-detected-drain"
 target: "{{ alert.labels.node }}"
 timestamp: "{{ now }}"

Workflow 2 — `ecc-dbe.yaml` — immediate paging

workflow:
 id: ecc-dbe-critical
 name: "ECC double-bit — page on-call and quarantine node"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: GpuEccDoubleBit
 steps:
 - name: cordon-immediately
 provider:
 type: kubernetes
 with:
 action: cordon
 name: "{{ alert.labels.node }}"
 - name: drain-workload
 provider:
 type: kubernetes
 with:
 action: drain
 name: "{{ alert.labels.node }}"
 options:
 ignore-daemonsets: true
 delete-emptydir-data: true
 grace-period: 120
 - name: page-oncall
 provider:
 type: pagerduty
 config: "{{ providers.pagerduty-critical }}"
 with:
 service_key: "{{ env.PD_SERVICE_KEY }}"
 severity: critical
 summary: "RB-04 ECC DBE on {{ alert.labels.node }}/{{ alert.labels.gpu }} — node drained"
 - name: emit-lifecycle
 provider:
 type: kafka
 config: "{{ providers.kafka-incidents }}"
 with:
 topic: incidents.lifecycle
 message:
 incident_id: "{{ alert.fingerprint }}"
 event: incident.opened
 severity: critical
 runbook: RB-04
 requires_postmortem: true
 - name: notify-dpo
 provider:
 type: email
 with:
 to: dpo@example.local
 subject: "ECC DBE on a production GPU — assessment required"
 body: |
 RB-04 ECC DBE incident detected on {{ alert.labels.node }}.
 Affected model: {{ alert.labels.model }}.
 Please assess whether personal/classified data was processed
 during the error window and whether GDPR art. 33 notification is needed.

Workflow 3 — `canary-rollback.yaml` — TTFT P95 out of SLO

workflow:
 id: canary-rollback-ttft
 name: "Rollback canary when TTFT P95 ratio v2/v1 > 1.30"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: VllmTtftP95OutOfSlo
 - key: canary_active
 value: "true"
 steps:
 - name: check-ratio
 provider:
 type: prometheus
 config: "{{ providers.prom-prod }}"
 with:
 query: |
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
 /
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))
 condition: result > 1.30
 actions:
 - name: argo-rollback
 provider:
 type: kubernetes
 with:
 action: exec
 command:
 - kubectl
 - argo
 - rollouts
 - abort
 - "{{ alert.labels.rollout }}"
 - -n
 - "{{ alert.labels.namespace }}"
 - name: notify-and-audit
 provider:
 type: kafka
 config: "{{ providers.kafka-audit }}"
 with:
 topic: audit.actions
 message:
 incident_id: "{{ alert.fingerprint }}"
 action: canary_rollback
 ratio: "{{ steps.check-ratio.result }}"
 actor: keep-workflow
 timestamp: "{{ now }}"

Each workflow lives in repos/keep-workflows/ versioned in git, reviewed by pull request, validated by CI (keep workflow validate). The written runbook lives as docs/runbooks/RB-XX.md linked from the workflow YAML; the two always evolve together.

The canonical Kafka event schema

For the topics to be consumable by compliance, postmortem tooling and dashboards without each consumer having to guess the shape, the schema is fixed with Avro / Protobuf.

{
 "name": "IncidentLifecycleEvent",
 "type": "record",
 "fields": [
 { "name": "incident_id", "type": "string" },
 { "name": "event", "type": { "type": "enum", "symbols": [
 "incident.opened", "incident.acknowledged", "action.proposed",
 "action.executed", "action.failed", "incident.escalated",
 "incident.resolved", "postmortem.attached"
 ]}},
 { "name": "timestamp", "type": "string", "logicalType": "timestamp-millis" },
 { "name": "actor", "type": "string" },
 { "name": "severity", "type": { "type": "enum", "symbols": ["low","warning","critical"] } },
 { "name": "runbook", "type": ["null","string"], "default": null },
 { "name": "alert_name", "type": "string" },
 { "name": "labels", "type": { "type": "map", "values": "string" } },
 { "name": "annotations", "type": { "type": "map", "values": "string" } },
 { "name": "evidence_uri", "type": ["null","string"], "default": null },
 { "name": "requires_postmortem", "type": "boolean", "default": false }
 ]
}

For audit.actions (WORM), a separate and stricter schema with non-modifiable fields:

{
 "name": "AuditAction",
 "type": "record",
 "fields": [
 { "name": "incident_id", "type": "string" },
 { "name": "action", "type": "string" },
 { "name": "actor", "type": "string" },
 { "name": "actor_type", "type": { "type": "enum", "symbols": ["human","workflow","scheduler"] } },
 { "name": "workflow_id", "type": ["null","string"], "default": null },
 { "name": "target", "type": "string" },
 { "name": "command", "type": ["null","string"], "default": null },
 { "name": "result", "type": { "type": "enum", "symbols": ["success","failure","partial"] } },
 { "name": "timestamp", "type": "string", "logicalType": "timestamp-millis" },
 { "name": "evidence_uri", "type": ["null","string"], "default": null },
 { "name": "approver", "type": ["null","string"], "default": null }
 ]
}

The topic is configured with cleanup.policy=delete, retention.ms=15552000000 (6 months) and min.insync.replicas=2 with acks=all to guarantee durability. For longer retention without the Kafka cost, use tiered storage to Ceph RGW or an S3-compatible backend: new log in the hot tier, old log in the cold tier, transparently to the consumer.

Formal fit into incident management

Runbooks are not an isolated SRE practice. They fit into four regulatory frameworks that production LLM platforms touch daily.

ISO/IEC 27035 — information security incident management

It defines the formal cycle in five phases: plan & prepare → detect & report → assess & decide → respond → lessons learned. Each phase has outputs that must be documented. Translated to the stack:

Plan & prepare: runbooks RB-01 to RB-06 plus the Keep workflows are part of the Information Security Incident Management Plan. Versioned in git, reviewed annually.
Detect & report: the Prometheus alerts entering Kafka are the materialisation.
Assess & decide: the severity in gpu.alerts.enriched plus the Keep workflow logic.
Respond: execution of the workflow’s steps plus actions.
Lessons learned: a mandatory postmortem for the runbooks that flag it; the output documented in the postmortem repo plus the runbook update.

ENS (Esquema Nacional de Seguridad) — op.exp controls

op.exp.7 Incident management: the runbook catalogue plus the Keep / Kafka pipeline materialise the “organised and procedural response”.
op.exp.8 Activity logging: the audit.actions topic with WORM retention of 6 months (minimum for level ALTO).
op.exp.9 Incident management logging: the incidents.lifecycle topic with the full cycle of each incident.
op.exp.10 Protection of activity logs: WORM plus encryption at rest plus access control (compliance consumers read-only).

NIS2 — notification to the competent authority

For essential and important entities, art. 23 sets three deadlines from the detection of significant impact:

24 hours: an early warning to the national CSIRT (INCIBE-CERT in Spain).
72 hours: a formal notification with an initial assessment.
1 month: a final report with root cause, impact and corrective measures.

The data for those reports comes straight out of incidents.lifecycle plus audit.actions with a consumer that generates the dossier in the required format. Without the auditable pipeline, the NIS2 deadlines are unreachable.

EU AI Act — art. 73 (serious incident reporting)

Applicable to high-risk systems. Deadlines:

2 days: for incidents causing death or irreversible harm to people or critical infrastructure.
10 days: for incidents producing serious disruption of critical infrastructure.
15 days: for all other “serious incidents”.

The definition of “serious incident” includes systematic model failures, a fundamental rights breach, material or environmental harm. Runbooks must flag which alerts can lead to a serious incident (typically anything affecting the model’s output in a high-risk context) and fire a specific legal assessment sub-workflow.

ISO/IEC 42001 — AIMS clause 10 continual improvement

The mandatory post-incident postmortem feeds clause 10. Updating the runbook after every incident that reveals a new pattern is the “corrective action with verification of effectiveness” the standard demands. See ISO 42001 AIMS.

Four anti-patterns

Anti-pattern 1 — alerts without a runbook. The alert fires, the junior operator on call looks at the dashboard, searches Confluence, finds nothing up to date, calls the senior on Slack, waits 20 minutes. In that time the incident has grown. Rule: no alert goes to production without a published runbook and an approved Keep workflow. CI validates that every PrometheusRule with severity ≥ warning has its corresponding keep workflow.

Anti-pattern 2 — a runbook without prior evidence capture. The workflow runs nvidia-smi --gpu-reset as soon as the XID arrives, losing the state that would have diagnosed the root cause. The next identical XID forces the diagnosis to be redone from scratch. Rule: steps before actions; all evidence is captured first, destructive actions come after.

Anti-pattern 3 — escalation by seniority rota instead of severity. The junior operator on call handles an ECC DBE because “it is their turn”. They lack the context to understand row remap, retired pages or the data corruption risk. Rule: page by severity, not by rota: RB-04 and RB-03 page the senior primary ON-CALL with automatic escalation to infra/hardware if there is no acknowledgement within 10 min.

Anti-pattern 4 — no human gate for destructive actions. The workflow runs kubectl drain automatically on any alert marked CRITICAL. On the first false alarm (a transient that resolved itself in 30 s), Keep drained a production node during peak hour. Rule: destructive actions (drain, reset, RMA, full rollback) require human confirmation via a Slack interactive message, with a configurable timeout. A justified exception: ECC DBE confirmed by > 1 measurement, where the corruption risk outweighs the false alarm risk.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with Kafka and Keep already deployed:

Kafka: a 3-broker cluster on non-GPU nodes of the K8s cluster; topics gpu.alerts.enriched, incidents.lifecycle, audit.actions configured with replication factor 3, min.insync.replicas 2. Audit with tiered storage to Ceph RGW for retention > 6 months without a brutal cost.
Keep: 2 operator replicas plus 1 worker replica in a keep namespace; connected to Prometheus (read provider), Kafka (read plus write provider), Slack, PagerDuty, Jira, Kubernetes (provider with a dedicated SA holding get/list/patch nodes and create jobs permissions).
Workflows: ~25-40 YAML files in the infra/keep-workflows/ repo, synchronised with the cluster via Flux or Argo CD. Validated by CI (keep workflow validate) on every PR.
Event volume: for 16 GPUs in normal operation with debounced alerts, ~50-200 events/day on gpu.alerts.enriched. In a typical incident, peaks of 500-2,000 events/day.
Compliance consumers: a python consumer in the compliance namespace that generates NIS2 / ENS / EU AI Act reports weekly, reading audit.actions and incidents.lifecycle.

What we have not covered (upcoming posts)

Postmortem playbooks — the mechanics of RCA with 5-whys, Ishikawa adapted to LLMs, integration with MLflow tracking for re-training if the postmortem produces an enriched dataset.
Chaos engineering for LLMs — controlled injection of XID errors, simulated ECC, artificial HBM latency to validate runbooks before the real incident.
Multi-cluster incident coordination — how to coordinate Keep across geographic clusters when an incident affects multiple regions.
CMDB and procurement integration — the RMA → ticket → ServiceNow → hardware replacement cycle automated via workflow.
LLM forensics — extracting the full OTel trace of a request affected by an incident, with redacted PII, kept in an evidence vault.

References

ISO/IEC 27035-1:2023 — Information security incident management — Principles and process.
ISO/IEC 27035-2:2023 — Information security incident management — Guidelines to plan and prepare for incident response.
ENS — Real Decreto 311/2022, Anexo II controles op.exp.7 a op.exp.10.
Directiva NIS2 (UE 2022/2555) — art. 23 (notificación de incidentes significativos).
Reglamento EU AI Act (UE 2024/1689) — art. 73 (reporting of serious incidents).
ISO/IEC 42001:2023 — AI management system — cláusula 10 (mejora continua).
Keep project — keephq.dev y github.com/keephq/keep (documentación de workflows YAML, providers).
Apache Kafka — Tiered Storage y cleanup.policy (docs.confluent.io / kafka.apache.org).
Confluent — Schema Registry y best practices para eventos lifecycle.
NVIDIA — Xid Errors Documentation y procedimientos de remediación.
Google SRE Book — Effective Troubleshooting y Postmortem Culture.
Atlassian — Incident Management Handbook (referencia para severity matrices).

Mixed NVIDIA + Intel environments for LLM inference: from the central H100 cluster to the NUC in the branch office

Tue, 02 Jun 2026 04:30:00 +0200

This post complements those on Capacity planning for on-premise LLM inference (which assumed a pure NVIDIA cluster), Seven layers of the stack (which also did not get into hardware heterogeneity) and The LLM inference router (where capability routing makes full sense once there is mixed hardware). It is the missing piece for talking about “hardware sovereignty” without reducing it to “which vendor do I pick”.

TL;DR

A production LLM inference cluster in 2026 can stop being monolithically NVIDIA if it accepts heterogeneity as an architectural decision. The motivation is not theory but three measurable operational advantages. (1) Cost: an Intel Xeon 6 with AMX (Advanced Matrix Extensions) delivers 7B INT4 at ~80 tok/s while serving embeddings and reranking at a fraction of the cost of dedicating an H100 to that task; the capacity planning adds up better with an Intel CPU handling the cheap work and an NVIDIA H100 the big LLM. (2) Sovereignty and supply chain diversification: NVIDIA holds roughly 94 % of the AI accelerator market (November 2025), single-vendor dependency with all its risks; Intel manufactures in Europe (Leixlip operational, Magdeburg planned) against a design-only NVIDIA with TSMC as foundry, which for a Spanish or European organisation subject to ENS / NIS2 / EU AI Act is a real hedging argument. (3) Edge: an Intel NUC with a Lunar Lake CPU (48 TOPS NPU) or Panther Lake (50 TOPS NPU + 120 TOPS Xe3 = 180 TOPS platform) runs 7B INT4 models at usable speed, which opens up the “branch office with local inference + central DC for complex cases” pattern. Relevant Intel hardware in June 2026: Intel Gaudi 3 (128 GB HBM2e, 1,835 TFLOPS BF16/FP8, 3.67 TB/s; a direct competitor to the H100, Intel claims +20 % on Llama 2 70B but Signal65 published H200 9× over Gaudi 3 on Llama 3.1 405B, you have to cite both; Falcon Shores cancelled in January 2025, Jaguar Shores 2026 as the restart bet, Gaudi 4 confirmed as never happening); Intel Xeon 6 with AMX (up to 288 E-cores on Sierra Forest or 86 P-cores on Granite Rapids, 1,024 FLOPS BF16/cycle/core with AMX, Intel claims 2.7× tok/s vs EPYC 9965 on the vLLM CPU backend); Intel Arc Pro B60 (Battlemage, 24 GB GDDR6, 456 GB/s, 197 TOPS INT8, launched September 2025, with a dual-GPU 48 GB variant and a “Battlematrix” rack with 8× = 192 GB VRAM); Intel NUC with NPU (Lunar Lake 48 TOPS, Arrow Lake similar, Panther Lake 50 TOPS at CES 2026; realistic for 7-13B INT4, not for the 30-70B Intel claims in its marketing). Software: OpenVINO 2025.3 with the GenAI API and vLLM-OpenVINO; IPEX-LLM with integrations into llama.cpp, vLLM, HF, LangChain; vLLM CPU backend with AMX; llama.cpp SYCL (better than Vulkan on Arc). Four canonical patterns: embeddings + reranker on Intel alongside the LLM on NVIDIA; guardrails + PII redaction on a near-edge NUC; speculative drafter on a NUC near the user and target on the H100; NUC dev workstations. Unified observability via DCGM + habana-metric-exporter + intel-gpu-exporter + Intel PCM federated in Prometheus. Pitfalls: tokenizer mismatch between engines, edge↔central round-trip latency, FP8 Hopper ≠ INT8 AMX in quality, version synchronisation. Applied to a generic cluster: central DC 4×H100 SXM + Xeon 6 AMX sidecar + 6-12 Intel NUCs in branch offices. Critical disclaimer: as of June 2026 there are no publicly verifiable cases of a mixed NVIDIA + Intel deployment in European banking or government; the pattern is emergent architecture and worth recommending, not established practice with an industrial track record.

You are here: DEPLOY (with heterogeneity as a decision)

The analogy: the factory with several different machines

A serious factory has several machines with different purposes, not one universal machine. A 200-tonne hydraulic press for heavy stamping; a bench lathe for turned parts; a 3D printer for quick prototypes; a pick-and-place robot for SMD. Each machine does what it does better than the others in its niche, and the plant manager sizes the mix according to the real product portfolio, not according to fashion. Buying three hydraulic presses because “they are the most impressive” when 60 % of the work is turned parts wastes capital: the lathe is cheaper, faster in its niche and frees the press for what genuinely needs it.

An LLM inference cluster with an NVIDIA H100 dedicated to computing embeddings for a RAG corpus is using a hydraulic press to drill bolts. The H100 is magnificent for a 70B LLM in BF16 with concurrency of 40+; for embeddings of an 800-token document with bge-m3, what you need is an Intel Xeon 6 with AMX at a fraction of the cost and power draw. A cluster that wants to serve light guardrails (Llama Guard 4 8B) on every request, with a 50 ms budget, does not need that guardrail on an H100 either: an Intel NUC with a 48 TOPS NPU covers the case with room to spare.

The heterogeneous factory is not theoretical elegance: it is maximising the useful utilisation of fixed capital. Nor is the heterogeneous LLM inference cluster.

Three operational reasons for heterogeneity

Reason 1 — cost

An H100 SXM 80 GB running 24/7 draws ~700 W (real measurement at the wall ~697 W with vLLM Llama 3.1 405B batch=4) and represents between 25,000 € and 35,000 € of amortised hardware. An Intel Xeon 6 with AMX (Granite Rapids 86 cores or Sierra Forest 288 E-cores) draws 350-500 W for the socket and costs a fraction. The operational split: the H100 is reserved for the big LLM (Llama 70B BF16 or FP8, where its HBM3 and FP8 tensor cores are worth their weight); the Xeon AMX absorbs embeddings (bge-m3, e5-large), reranking (bge-reranker-v2-m3), small models (Llama 3.2 1B / 3B INT4) and offline batch processing. It is the same logic as the capacity planning taken one step further: instead of budgeting KV cache VRAM only on the H100, budget each workload on the silicon where its arithmetic intensity fits best.

Reason 2 — sovereignty and supply chain diversification

As of November 2025, NVIDIA holds approximately 94 % of the AI accelerator market. That concentration is risk. For an organisation subject to ENS / NIS2 / EU AI Act, depending on a single supplier whose foundry is concentrated in Taiwan (TSMC) introduces supply chain vulnerabilities that recent regulations (NIS2, supply chain provisions) are starting to require you to document and mitigate. Intel diversifies: it has its own fabs in Europe (Leixlip operational in Ireland; Magdeburg planned in Germany, with EU Chips Act funding), which for a Spanish or European public sector client is a real contractual argument, not marketing.

Mandatory disclaimer: the Intel roadmap after Falcon Shores is unstable. Intel cancelled Falcon Shores in January 2025 and relegated Gaudi 4 to “will never exist”; the restart bet is Jaguar Shores in 2026 as a rack-scale platform, still with no confirmed public specs. Diversification is strategically correct, but assuming continuity of the Intel roadmap at NVIDIA’s level in 2026 would be naive. The operational strategy: Intel for workloads where lock-in is lower (CPU for embeddings, NUC for light edge, replaceable by AMD/Apple/SiFive if Intel pivots again), NVIDIA for the big LLM where the maturity of the software stack still has no rival.

Reason 3 — edge

The “everything travels to the central DC” pattern breaks in three cases: latency (branch office 100+ ms away from the DC, unacceptable for chat), data sovereignty (prompts with personal or classified data that must not leave the local perimeter), and offline operation (branch office with intermittent connectivity). The Intel NUC with a modern CPU (Lunar Lake / Arrow Lake / Panther Lake) brings 48-50 TOPS NPU + 100-180 TOPS Xe2/Xe3 iGPU in a box of 0.5-1.5 L volume drawing 30-65 W. 7B INT4 models run at usable speed; with more aggressive quantisation (Q3_K) a Llama 13B fits. For branch offices with RAG over a local corpus + a 7B LLM + guardrails, the NUC is perfect.

Relevant Intel hardware (June 2026)

Part	Memory	Key performance	Launch	Status
Intel Gaudi 3	128 GB HBM2e, 3.67 TB/s	1,835 TFLOPS BF16/FP8; 1,200 GB/s networking	Apr 2024	Active; successor Jaguar Shores 2026 (not Gaudi 4)
Intel Xeon 6 (Granite Rapids)	DDR5 + MRDIMM	86 P-cores, AMX 1,024 FLOPS BF16/cycle/core	2024-2025	Active
Intel Xeon 6 (Sierra Forest)	DDR5	288 E-cores	2024	Active
Intel Arc Pro B60 (Battlemage)	24 GB GDDR6, 456 GB/s	197 TOPS INT8; 12.28 TFLOPS FP32	Sep 2025	Active; dual 48 GB variant, rack 8× = 192 GB
Intel Data Center GPU Max	128 GB HBM	successor to Ponte Vecchio	discontinued	Discontinued Jan 2026
Intel NUC (Lunar Lake)	DDR5x	NPU 48 TOPS + Xe2 67 TOPS = 120 TOPS platform	2024	Active
Intel NUC (Arrow Lake)	DDR5	NPU 13 TOPS + Xe iGPU	2024	Active (less NPU than Lunar)
Intel NUC (Panther Lake)	DDR5x	NPU 50 TOPS + Xe3 120 TOPS = 180 TOPS platform	CES Jan 2026	Rolling out

Intel Gaudi 3 — the critical note on the marketing

Intel publishes that Gaudi 3 delivers +20 % throughput vs H100 on Llama 2 70B inference and 2× price/performance. The figure appears in the official whitepaper and in launch presentations. However, Signal65 (an independent firm) published in 2025 that the H200 beats Gaudi 3 by a factor of 9× on Llama 3.1 405B. The discrepancy matters: both numbers can be true for their specific benchmarks (Llama 2 70B FP16 vs Llama 3.1 405B FP8) but the operational conclusion changes radically depending on which one you keep.

This post’s recommendation: treat Gaudi 3 as a valid option for Llama-class 70B in BF16/FP8 where Intel claims parity or advantage, not for frontier 200B+ models where NVIDIA keeps a clear margin. And consider the roadmap risk: Gaudi 4 will never exist; the line’s successor is Jaguar Shores 2026 with a completely new rack-scale architecture, a discontinuity rather than an evolution.

Intel Xeon 6 with AMX — the CPU workhorse

The Advanced Matrix Extensions (AMX) are the non-obvious piece. Each Granite Rapids P-core executes up to 1,024 BF16 FLOPS per cycle via AMX, which turns a Xeon 6 with 64-86 cores into a respectable matrix accelerator for small and mid-sized models. Real reported figures: Llama 3.2 INT4 at ~57 tok/s with AMX vs 28 tok/s without AMX (a clean 2× factor). For serving 7B INT4 with the vLLM CPU backend + AMX, Intel claims 2.7× tok/s vs EPYC 9965, a figure with Intel bias but qualitatively corroborated by LMSYS in their DeepSeek R1 671B deployment on Xeon 6 + SGLang.

Operational use case: embeddings and reranking on a Xeon 6 sidecar next to the H100 cluster. Models such as bge-m3 (multilingual embedding) or bge-reranker-v2-m3 run at acceptable throughput on CPU AMX; they do not deserve a dedicated H100. Freeing the H100 for the 70B LLM raises the cluster’s effective RPS without buying more GPUs.

Intel Arc Pro B60 and Battlematrix

Launched in September 2025, the Arc Pro B60 (Battlemage) brings 24 GB GDDR6 with 456 GB/s of bandwidth and 197 TOPS INT8 at 200 W. Maxsun has a dual-GPU 48 GB variant. The “Battlematrix” rack configuration combines 8 units = 192 GB of aggregate VRAM. The interesting point: at a cost far below an H100 SXM 80 GB, which makes it a candidate for 30-70B INT4-INT8 LLMs served via OpenVINO or llama.cpp SYCL.

Phoronix verified that on SYCL the Arc Pro B70 reaches parity with the Radeon PRO W7900 (previous AMD generation) on DeepSeek R1 Llama 8B pp512. The Vulkan backend loses heavily (~1/4 of SYCL performance); for Arc Pro, always SYCL.

Intel NUC with NPU — the edge node

Intel NUCs with a Lunar Lake CPU (Core Ultra Series 2) bring NPU 4 with 48 TOPS and a platform total of 120 TOPS adding the Xe2 iGPU and CPU AVX. Panther Lake (CES January 2026) goes up to NPU 5 = 50 TOPS + Xe3 120 TOPS = 180 TOPS platform.

Intel states that Panther Lake “runs 30-70B models locally”. Reality check: it is marketing. A 30-70B INT4 fits in RAM (DDR5x 32-64 GB) but sustained speed with Q4_K_M quantisation on a NUC hovers around 2-8 tok/s; comfortable for occasional use, not for serving traffic. The NUC’s real sweet spot is 7B INT4 at 20-40 tok/s on the iGPU/NPU, perfect for a client branch office with casual queries.

Intel software — the relevant stack

OpenVINO 2025.3 (June 2026) is the central piece. It supports single-command deployment via the OVMS CLI with automatic download from the HF Hub; it integrates OpenVINO GenAI with a C++/Python API for generative pipelines; it exposes a vLLM v1-compatible API (vLLM-OpenVINO). GGUF model support: DeepSeek Distill, Qwen 2/2.5, Llama 3. Optimisations: Sage Attention (first token with long prompts), per-channel KV-cache compression.

Intel Extension for PyTorch (IPEX), XPU version 2.8.10+xpu, adds Intel backends to PyTorch. IPEX-LLM is the subproject that integrates with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM and DeepSpeed. May 2025: it ran DeepSeek V3/R1 671B and Qwen3MoE 235B on 1-2 Arc A770/B580 cards with FlashMoE.

vLLM CPU backend, the CPU branch of vLLM with AMX optimisations. For 7B INT4 on a 4th-gen Xeon with AMX: 12-50 tok/s; with a Xeon Gold 6530 + INT4: ~80 tok/s. Academic figures (arXiv 2410.04466).

llama.cpp SYCL, the recommended backend for Arc; Vulkan works but hovers around 1/4 of SYCL performance on an Arc B580. SYCL reaches parity with the previous AMD generation.

Habana SynapseAI, the Gaudi 3 stack. The PyTorch bridge habana_frameworks.torch registers the hpu device; it integrates with torch.compile. It is not a complete port to oneAPI but a partial integration via oneMKL. That implies the Gaudi ecosystem keeps a certain separation from Intel’s general oneAPI, relevant in view of the hypothetical Jaguar Shores and future unification.

The four canonical patterns

Pattern 1 — embeddings + reranker on Intel

The most mature and the easiest to adopt. In a typical RAG system, each user request invokes:

Query embedding (50 ms on an H100, 80 ms on a Xeon AMX, 30 ms on an Arc Pro B60).
Vector search (Qdrant / Milvus / Chroma; latency ~10-30 ms).
Reranker over the top-k candidates (60 ms on an H100, 100-150 ms on a Xeon AMX).
LLM over the augmented prompt (200-500 ms TTFT, 30-50 ms/token).

Steps 1 and 3 are memory-bound and relatively small (100M-1B models): a Xeon 6 with AMX (the Arc Pro B60 is faster but is already a dedicated GPU) does the job at a hardware cost that is a fraction of a dedicated H100. Step 4 stays on NVIDIA because that is where its tensor architecture + HBM3 + FP8 is worth what it costs.

Operational implication: a Xeon 6 sidecar (~40 cores, ~10-15 k€) serving embeddings + reranking frees the equivalent of 1-2 H100s of “cheap” load, recovering that capacity for the big LLM. Clear ROI in sizing.

Pattern 2 — guardrails + PII redaction on a near-edge NUC

A bank branch, a medical practice or a law office generates prompts with personal or classified data. Sending those prompts to the central DC (even if it is corporate on-premise) can clash with local retention policies or with specific compliance (GDPR, professional privilege).

The pattern: the NUC in the branch office runs two critical steps before the round-trip:

PII redaction with Presidio (CPU-only, fast) or Llama Guard 4 8B on the NUC’s NPU + iGPU. It replaces names, tax IDs and account numbers with placeholders.
Light guardrails (PromptGuard 2 86M, Llama Guard 4 8B) on the NPU + iGPU. It filters prompt injection, jailbreaks and prohibited content.

Only afterwards does the redacted prompt travel to the central DC so the big LLM can answer. The response comes back to the NUC, which rehydrates the placeholders with the real values before showing it to the user. The sensitive data never leaves the branch office.

Costs: a Panther Lake NUC is ~1,500-2,500 € per unit, scalable to dozens of branch offices with no additional central GPU cost. Latency: 50-150 ms for the edge step before the DC’s 200-500 ms round-trip.

Pattern 3 — speculative decoding drafter on a NUC

Speculative decoding uses a small drafter that proposes γ tokens and a large target that accepts or rejects them in a single forward pass. If the drafter is geographically close to the user (a NUC in the branch office) and the target is in the central DC, the client’s perceived latency falls further still.

Setup: drafter Llama 3.2 1B INT4 on a NUC + target Llama 3.1 70B FP8 on the central H100. The NUC generates γ=4 tokens in ~50 ms locally; the target verifies them in one pass (40-80 ms including round-trip); if the acceptance rate is > 60 %, effective TTFT falls ~50 % compared with Llama 70B without speculative decoding.

An important constraint: drafter and target must share a tokenizer. Llama 3.2 1B and Llama 3.1 70B have compatible tokenizers. Mixing a Llama drafter with a Qwen target breaks the pattern.

Pattern 4 — NUC dev workstations

Dev / CI iterating on prompts, evals and retrieval logic does not need a production GPU to validate correctness. A NUC with Llama 3.2 3B INT4 runs the end-to-end functional tests (including embeddings + retrieval + LLM + guardrails) at a tenth of the cost of iterating on an H100. Only the last pre-prod smoke test uses the production cluster.

A mature pattern in organisations with many developers and scarce production GPU. Iteration that is 10× faster and cheaper translates into feature delivery speed.

Unified observability in a heterogeneous cluster

The GPU observability post covered DCGM Exporter for NVIDIA. In a mixed cluster you need more:

Hardware part	Exporter	Key metrics
NVIDIA H100/A100	`nvidia/dcgm-exporter`	DCGM_FI_DEV_* + DCGM_FI_PROF_*
Intel Gaudi 3	`HabanaAI/habana-metric-exporter`	habana_hpu_utilization, habana_hbm_used
Intel Arc Pro	`intel/intel-gpu-exporter` (unofficial; alternatives exist)	xe_engine_utilization, xe_memory_used
Intel Xeon CPU + AMX	`prometheus/node-exporter` + Intel PCM	cpu_amx_utilization (via PCM)
Intel NUC (NPU+iGPU)	`intel/intel-gpu-exporter` + custom NPU exporter	npu_utilization, xe_iGPU

All federated into a single Prometheus + Grafana. The dashboards are organised by hardware family (NVIDIA, Intel server, Intel edge) plus an aggregate “heterogeneous cluster” view with per-tenant SLOs combining all four.

Cardinality: ~1.5-2× that of a pure NVIDIA cluster. Manageable with Thanos / Mimir for long retention.

Capability routing — from the L7 router to the heterogeneous one

The LLM inference router stops being a version selector for the same model and becomes a capability dispatcher:

models:
 - name: "llama-70b-chat"
 endpoint: "vllm-llama70b.inference.svc:8000"
 backend: nvidia-h100
 capabilities: [chat, tool_use, json_mode]
 - name: "embedding-multilingual"
 endpoint: "ipex-bge-m3.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [embeddings]
 - name: "reranker-multilingual"
 endpoint: "ipex-bge-reranker.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [reranking]
 - name: "guardrail-prompt-injection"
 endpoint: "openvino-llama-guard.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [guardrails, redact-pii]
 region: sucursal-01
 - name: "llama-3b-draft"
 endpoint: "openvino-llama-3b.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [speculative-drafter]
 region: sucursal-01
 target_model: "llama-70b-chat"

The router resolves model=embedding-multilingual → Intel Xeon; model=llama-70b-chat → H100; model=guardrail-prompt-injection with region=sucursal-01 → local NUC. If the branch office NUC goes down, failover to an equivalent replica in the central DC, accepting the latency cost.

LiteLLM Proxy, NVIDIA Dynamo and Envoy AI Gateway support this capability routing. The non-obvious piece: the router has to know the compatible tokenizer between drafter and target for pattern 3, which is modelled in extra catalogue metadata.

Specific pitfalls

Tokenizer mismatch between engines. OpenVINO with a Llama 3.2 GGUF and vLLM with the same nominal Llama 3.2 can use slightly different tokenizers (chat template, special tokens). Validate token identity with tokenizer.encode("hola") on both sides before assuming interchangeability. For speculative decoding, a single different token breaks the pattern.

Edge ↔ central round-trip latency. Patterns 2 and 3 assume the NUC and the DC are on the same corporate WAN with controlled latency. If the branch office is on 4G/5G with 100-200 ms of jitter, the speculative drafter compensates for nothing; on the contrary, it adds latency. Measure before promising.

FP8 Hopper ≠ INT8 AMX in output quality. The operator assumes a request that runs FP8 on an H100 and INT8 on a Xeon AMX will produce the same output. That is not true: the two quantisations have different degradation profiles. If the system expects idempotence (for example, evals with golden output), validate offline that the Intel version reproduces the expected behaviour within tolerance.

Model version synchronisation between sites. The model in the central DC gets updated, but the branch office NUCs keep the old version of the drafter or the guardrail for weeks. Result: divergent behaviour between branches with no easy diagnosis. Policy: the central model and the edge model advance together or within a documented window; the canary extends to the NUC fleet.

Unstable Intel roadmap. Falcon Shores cancelled, Gaudi 4 will never exist, Jaguar Shores 2026 still with no confirmed public specs. Buying Gaudi 3 today is reasonable if the use case justifies the 18-24 months of amortisation; committing an architecture for 5+ years on an Intel accelerator is a riskier bet than the NVIDIA equivalent, at least until Jaguar Shores materialises with a mature software stack.

Absence of public production deployments. As of June 2026, the confirmed Gaudi 3 deployments are IBM Cloud, Dell AI Factory and a handful of early adopters (Bharti Airtel, Bosch, Naver). There is no publicly verifiable case of a mixed NVIDIA + Intel cluster in European banking or government. This pattern is recommended emergent architecture, not practice with an industrial track record. The first adopter takes on a validation cost that a second adopter avoids.

Applied to a generic on-premise cluster

For an organisation with a generic heterogeneous LLM inference cluster:

Central DC: 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink = 16 H100s. Serves the big LLMs (Llama 70B, Mixtral 8×22B, Qwen 72B) in BF16 or FP8.
Xeon 6 sidecar: 2-4 Xeon 6 servers (Granite Rapids 64-86 cores) with AMX, 512 GB DDR5, in the same rack as the H100 cluster. Serves embeddings (bge-m3), reranking (bge-reranker-v2-m3) and small models (Llama 3.2 1B/3B) on the vLLM CPU backend with AMX.
Arc Pro sidecar (optional): 1-2 servers with 4-8× Arc Pro B60 24 GB each (Battlematrix), for 13-30B INT8 models via OpenVINO. Useful if the cost per mid-sized LLM has to come down from the H100.
Edge NUCs in branch offices: 1-2 Panther Lake NUCs per branch, with a 50 TOPS NPU + Xe3 120 TOPS, serving Llama Guard 4 + Presidio + a Llama 3.2 1B INT4 drafter via OpenVINO. Corporate WAN connectivity with latency < 80 ms to the DC.

Estimated envelope: central cluster ~120 kW of GPU peak + ~10-15 kW of Intel sidecars. Edge: ~50 W per NUC, negligible compared with office costs.

Observability: federated Prometheus in the DC + pull scrape from the NUCs (via corporate VPN). Dashboards “NVIDIA GPU fleet”, “Intel server fleet”, “Intel edge fleet” plus a “consolidated SLO” view.

Router: LiteLLM Proxy or NVIDIA Dynamo in the DC, with an extended model catalogue to include Intel backends and regions (sucursal-01, sucursal-02, …). Documented edge→central failover.

What we have not covered (upcoming posts)

Reproducible benchmarks of Llama 70B on Gaudi 3 vs H100 SXM on equivalent hardware, the missing material for making decisions with your own data rather than Intel’s or Signal65’s.
AMD ROCm in the mix: how MI300X / MI355X fit into this heterogeneous pattern and what changes in the router catalogue.
Apple Silicon as edge: M3/M4 Max with a Neural Engine of ~38 TOPS + a 40-core GPU, hardware equivalent to the Panther Lake NUC but with a different software stack (MLX).
Energy cost optimisation: how nvidia-smi -pl 500W + an Intel TDP cap on Xeon 6 cuts the bill by 25-30 % with a 15-20 % throughput loss.
Model CI/CD for an edge fleet: how a rolling update of a Llama Guard reaches 50 branch office NUCs without any of them losing service.

References

Intel Gaudi 3

Intel — Gaudi 3 AI Accelerator White Paper. https://cdrdv2-public.intel.com/817486/gaudi-3-ai-accelerator-white-paper.pdf
Intel — Hot Chips 2024 Gaudi 3 deep dive. https://hc2024.hotchips.org/assets/program/conference/day1/60_HC2024.Intel.RomanKaplan.Gaudi3-0826.pdf
Signal65 / DataCenterDynamics — NVIDIA H200 outperforms Intel Gaudi 3 by factor of 9× across first Llama 3.1 405B benchmark test. https://www.datacenterdynamics.com/en/news/nvidia-h200-outperforms-intel-gaudi-3-by-factor-of-nine-across-first-llama-31-405b-benchmark-test-exclusive/
IEEE Spectrum — Intel Gaudi 3 review. https://spectrum.ieee.org/intel-gaudi-3
Tom’s Hardware — Intel cancels Falcon Shores GPU; Jaguar Shores to be successor. https://www.tomshardware.com/tech-industry/artificial-intelligence/intel-cancels-falcon-shores-gpu-for-ai-workloads-jaguar-shores-to-be-successor

Intel Xeon 6 + AMX

Intel — Xeon 6 (Granite Rapids) Product Brief. https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-02/xeon-6-granite-rapids-product-brief.pdf
OpenMetal — Intel AMX AI Inference Performance. https://openmetal.io/resources/blog/intel-amx-ai-inference-performance/
LMSYS — Intel Xeon 6 + SGLang for DeepSeek R1 671B. https://www.lmsys.org/blog/2025-07-14-intel-xeon-optimization/
arXiv 2410.04466 — CPU-LLM benchmarks with AMX.
Intel community blog — Accelerating vLLM Inference on Intel Xeon 6 Processor.

Intel Arc Pro Battlemage

Intel — Arc Pro B60 Graphics Specifications. https://www.intel.com/content/www/us/en/products/sku/243916/intel-arc-pro-b60-graphics/specifications.html
StorageReview — Intel Arc Pro B60 Battlematrix Preview: 192GB VRAM for On-Premise AI. https://www.storagereview.com/review/intel-arc-pro-b60-battlematrix-preview-192gb-of-vram-for-on-premise-ai
Phoronix — Intel Arc Pro B-series review. https://www.phoronix.com/review/intel-arc-pro-b-series

Intel NUC / NPU

HotHardware — Intel CES 2026 Panther Lake is a Go. https://hothardware.com/news/intel-ces-2026-panther-lake-is-a-go
TechPowerUp — Intel Panther Lake Technical Deep Dive.
arXiv 2412.11053 — NITRO: LLM inference on laptop NPU.
Intel — AI PC brings larger LLM development to your desk.

Software

OpenVINO — Release Notes 2025.3. https://www.intel.com/content/www/us/en/developer/articles/release-notes/openvino/2025-3.html
HuggingFace — Deploy with OpenVINO. https://huggingface.co/blog/deploy-with-openvino
Intel — Intel Extension for PyTorch XPU 2.8.10. https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/releases.html
IPEX-LLM — github.com/intel/ipex-llm.
Habana — SynapseAI PyTorch Theory of Operations. https://docs.habana.ai/en/latest/PyTorch/PyTorch_Gaudi_Theory_of_Operations.html

Market context

MLCommons — MLPerf Inference v6.0 benchmark results. https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
Intel newsroom — Gaudi 3 Expanded Availability. https://newsroom.intel.com/artificial-intelligence/intel-gaudi-3-expands-availability-drive-ai-innovation-scale

Sources: the full URLs are linked inline on each reference.

Anatomy of the twelve DCGM and five vLLM metrics: analogies, documented anomalies and real cases 2024-2026

Tue, 02 Jun 2026 04:00:00 +0200

This post goes deeper into the list of metrics presented in GPU observability for LLM inference. There each metric got its G/A/R threshold and PromQL query; here each one gets its explanatory analogy and the anomaly documented in production with a referenced public case. It is the post worth having open when an alert fires and you still do not know what to do with it; the next post on runbooks translates each anomaly into concrete action.

TL;DR

The twelve DCGM metrics (compute, memory, thermal-power, health) and the five vLLM engine metrics (concurrency, KV pool, SLO latencies) covered in the previous post paint the cluster’s cockpit, but a list without context does not teach you to diagnose. Each metric has a recurring anomalous pattern documented in the public literature (academic papers, GitHub issues, OEM KBs, operator blogs) that the veteran operator recognises instantly and the junior one does not. This post develops each metric with an analogy of its own that pins down which question it answers, and with the statistically relevant anomaly backed by figures from documented incidents. Three examples of the calibre involved: Meta published that during the training of Llama 3 405B on 16,384 H100s there were 419 unplanned failures in 54 days, one every 3 hours, with GPU + HBM3 accounting for 47 % of the total; the paper Story of Two GPUs (arXiv 2503.11901) quantifies that the H100 has 3.2× worse MTBE from uncorrectable ECC than the A100, attributable to the higher density of HBM3; issue vllm#16300 documents that on a cluster of 8×A100 80 GB TP=8 delivers worse throughput than TP=4 because NVLink saturation kills the partition speedup. The KBs Dell 000220508 and Lenovo HT514380 formalise the recurring HW Power Brake case in H100 racks overcommitted at PDU level. Issue vllm#25677 showed chunked prefill 10× slower than without it on Qwen3-30B-A3B (bad calibration of max_num_batched_tokens). Issue vllm#11912 documents a TPOT regression from 15.7 ms to 25.7 ms crossing version 0.6.4. Each case includes a verifiable URL. The operational rule: when an alert arrives, look first at the anomalous pattern associated with the metric that fired, before opening the request’s trace; 80 % of degradations match one of the documented patterns.

You are here: OBSERVE — the diagnosis layer

Family 1 — Compute

`DCGM_FI_PROF_SM_OCCUPANCY` — is there parallel work in the engines?

The analogy. An industrial kitchen with 32 hobs and a single chef. The metric answers “how many hobs have a pan on them right now?”. If half are empty, the kitchen is under-used: the orders come out one after another because the chef does not open up parallelism. If all of them are occupied but the chef is standing still watching a stopwatch, the hobs are on but nothing is being cooked (a pathological kernel saturating SMs without doing useful work).

The documented anomaly. The best-known trap: high SM occupancy does not imply real throughput. The article “GPU Utilization Is a Counter, Not a Cause” (Ingero, May 2026) put it in one exact sentence: “a kernel running at 5 % of peak FLOPS for 100 ms still reads 100 % on SM_ACTIVE”. In MoE workloads the effect turns pathological: overloaded experts produce the Straggler Effect (paper arXiv 2503.05066). The SMs look busy while waiting for the saturated expert, and the utilisation dashboard paints green while latency goes through the roof.

Operational implication. Do not trust sizing or autoscaling to SM occupancy alone. Always combine it with PIPE_TENSOR_ACTIVE (is there useful compute?) and DRAM_ACTIVE (is memory the bottleneck?). The normal LLM decode regime is 30–55 %, not 99 %; seeing a sustained 99 % with high TPOT is a symptom of a kernel bug or an MoE straggler.

`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` — are the tensor cores producing?

The analogy. A factory with two lines: the manual one (CUDA cores) and the automated one (tensor cores). The metric answers “what percentage of the time is the automated line active?”. If you buy an H100 for its tensor cores and the automated line is at 5 %, you have paid for a Ferrari to do bicycle courier work.

The documented anomaly. Issue vllm#20783 (July 2025) was titled literally “Performance Anomaly: compressed-tensors shows no speedup over BF16 on H100”. The operator expected 1.5–2× with FP8 quantisation and got parity with BF16; the PIPE_TENSOR_ACTIVE metric revealed that the FP8 path was not running on the HMMA units (the tensor unit for FP16/BF16/FP8) and was falling back to CUDA cores. Issue vllm#31475 documented the parallel case on MI300X: FP8 slower than BF16 because of a regression in the ROCm path. DCGM exposes separate counters per unit (HMMA for FP16/BF16/FP8, IMMA for INT8, DMMA for TF32/FP32); if HMMA is low even though the model is BF16, the engine is not using tensor cores.

Operational implication. Check PIPE_TENSOR_ACTIVE after every change of quantisation or engine version; a supposedly neutral change may have disabled the optimised path. For prefill expect 50–80 %; for decode 15–30 % is normal (decode is memory-bound, not compute-bound). A figure below 5 % in prefill means the engine is not using tensor cores.

`DCGM_FI_PROF_DRAM_ACTIVE` — is the HBM saturated?

The analogy. A motorway with N lanes. The metric answers “what percentage of the time are all the lanes occupied moving cars?”. When the tensor cores ask for data faster than the HBM delivers it, the motorway is at 95 % and the engines wait. In decode, this is the normal regime: you walk the model weights and the KV cache through for every token.

The documented anomaly. The paper “Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference” (arXiv 2503.08311) quantifies that at contexts ≥ 128k, reading the KV cache dominates the total decode time and saturates the HBM3 (3.35 TB/s on the H100). The distinctive pattern: DRAM_ACTIVE > 80 % with PIPE_TENSOR_ACTIVE around 10–20 %. Raising the batch no longer helps: the bottleneck is not FLOPS, it is bandwidth. The useful lever is compressing the KV, see Quantisation for --kv-cache-dtype=fp8, which cuts the KV footprint by roughly 50 %.

Operational implication. If DRAM_ACTIVE > 95 % is sustained and gpu_cache_usage_perc < 70 %, something other than your engine is asking for HBM (a leak in a library, another process sharing the GPU without MIG). Investigate immediately with nvidia-smi and fuser /dev/nvidia*.

Family 2 — Memory

`DCGM_FI_DEV_FB_USED` — how much VRAM has been consumed?

The analogy. The fuel level in an aircraft’s tank in flight: the pilot needs to know how much is left and at what rate it is being consumed, not just the instantaneous figure. An H100 at 88 % FB used and stable can operate calmly; the same figure rising 2 %/min announces OOM in 7 minutes.

The documented anomaly. Issue dcgm-exporter#512 documents a surprise relevant to MIG clusters: DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE are absent on H100 GPU instances with MIG enabled, present on A100 and B200, but an exporter bug hides them on H100-MIG. Operators who assume the dashboard covers everything discover the blind spot on the day of the first OOM. Issue dcgm-exporter#271 documents another detail: FB_USED + FB_FREE does not always add up to a constant because there is driver-reserved overhead that shows up in the delta. The original PagedAttention/vLLM paper estimated that pre-PagedAttention serving frameworks wasted 60–80 % of the KV cache to fragmentation; PagedAttention brought it down to under 4 %.

Operational implication. On H100 MIG clusters, check that DCGM_FI_DEV_FB_USED appears per instance before trusting alerts; if it is absent, monitor through nvidia-smi --query-gpu=memory.used directly. Operational rule: alert on the delta (a sustained rise), not just an absolute threshold.

`DCGM_FI_DEV_FB_FREE` — the absolute complement

The analogy. The “range remaining” indicator in a modern car: it complements the percentage with an absolute figure that is directly actionable.

The documented anomaly. When an aggressive PagedAttention pool leaves FB_FREE at small absolute values (< 2 GiB), any normal allocation of transient buffers (the activations of a large prefill) can push you into OOM. The classic pattern: a “green” percentage (87 %) but a “red” absolute (< 4 GiB free on an 80 GB H100).

Operational implication. A complementary alert with an absolute threshold: DCGM_FI_DEV_FB_FREE < 4096 (MiB). It is the safety net for the cases where the percentage misleads because the engine is configured with a very high gpu_memory_utilization.

`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL` — does the internal bus hold up?

The analogy. An interstate motorway between four cities. Every car that crosses to do a tensor parallel all-reduce pays a toll and consumes width. When there are more cars than the motorway supports, the latency to reach the destination shoots up, even if each individual car is fast.

The documented anomaly. Issue vllm#16300 (April 2025) was titled “Performance degradation with tp=8 compared to tp=4 on 8×A100(80G)” and documented TP=8 delivering worse throughput than TP=4 on the same cluster, same model, same quantisation. Root cause: tensor parallelism requires an all-reduce after each attention and MLP block; at TP=8, the communication cost between 8 GPUs (even over NVSwitch) grows faster than the speedup of partitioning the compute. The practical rule that emerges: TP=4 + 2 replicas usually delivers better latency/throughput than TP=8 + 1 replica, except for extremely long contexts (≥128k) where you need the aggregate VRAM. Theoretical NVLink 4.0 capacity on the H100 SXM: around 450 GB/s per GPU; typical sustained TP=4 regime: 50–150 GB/s.

Operational implication. If NVLINK_BANDWIDTH_TOTAL > 90 % of capacity is sustained, this is not a problem solvable by raising parallelism. The opposite: lower TP. The metric is orthogonal to the sizing from capacity planning: the ceiling is not only VRAM/time, it is also the bus.

Family 3 — Thermal and power

`DCGM_FI_DEV_GPU_TEMP` — is the GPU breathing?

The analogy. The body temperature of an elite athlete under full effort. 36–37 °C is normal; 38 °C is sustainable stress; above 39 °C the body activates protective mechanisms (sweating, slowing down) that degrade performance. The GPU does the same: above a thermal threshold, it reduces its clock automatically. If it did not, it would break.

The documented anomaly. The H100 SXM5 with its 700 W TDP has thermal thresholds that are not entirely public (NVIDIA does not publish them exhaustively in the datasheet), but the behaviour is well known: above roughly 85 °C edge or roughly 95 °C HBM, the bit 0x40 HW_THERMAL appears in the clock throttle reasons. Operators on the NVIDIA developer forum report that with a rack inlet temperature above 27 °C, throttling is routine. The NVIDIA HGX Platform paper states that the minimum recommended air flow is > 1000 CFM/kW; densities above 30 kW/rack at 700 W TDP demand mandatory liquid cooling because forced air does not get there.

Operational implication. If GPU_TEMP > 83 °C is sustained, look first at CLOCK_THROTTLE_REASONS (bit 0x40) and at the rack inlet temperature: this is not an engine problem, it is an air flow one. For legacy air-cooled racks, consider redistributing thermal load or installing a rear-door HX.

`DCGM_FI_DEV_POWER_USAGE` — how much is it asking of the socket?

The analogy. The instantaneous draw of an industrial appliance plugged into a three-phase outlet with a sized breaker. If the washing machine starts at 9 kW and the breaker is 10 kW, you live on the edge; if the washing machine “gets along” with the breaker it is because somebody sized it consciously.

The documented anomaly. A published real measurement: an H100 SXM5 with vLLM running Llama 3.1 405B at batch=4 draws around 697 W at-wall sustained (NVIDIA TDP 700 W). Now the interesting operational lever: lowering nvidia-smi -pl from 700 W to 500 W delivers around 30 % energy savings with only around 20 % loss of throughput. A cluster of 4 nodes × 8 H100 at 700 W equals around 22 kW of GPU alone; at 500 W, around 16 kW. The difference pays an entire quarter’s electricity bill in clusters operated around the clock. A three-phase 415 VAC PDU branch at 60–80 A supports around 32 kW, roughly 4 DGX H100. Legacy 208 V does not support H100 density; reference: NVIDIA DGX SuperPOD Electrical Specifications.

Operational implication. A useful metric for three things: (1) detecting anomalously low workloads (unexpected idle), (2) computing per-tenant showback of real energy cost (not estimates), (3) alerting if the draw approaches the PDU branch limit. Keep GPU → PDU branch → breaker mapped in the CMDB.

`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS` — who is stepping on the brake?

The analogy. The “limited mode” warning light on the dashboard of a modern car. When it lights up, the car reduces its performance automatically, but it does not tell you why unless you know how to read the combination of letters. The bits of the bitmap are those letters.

The documented anomaly. A public case formally acknowledged by two different OEMs: Dell KB 000220508 and Lenovo HT514380 address the same phenomenon: HW Power Brake Slowdown active (bit 0x80) on the H100 SXM. The cause is not the GPU: it is the chassis PDU sending an electrical power-brake signal because the rack branch is close to the breaker limit. The operator sees throughput down 30–50 % with no XID and no ECC, and the inference engine “is healthy”; the problem is in the electrics. The NVIDIA developer forum thread on “HW Power Brake Slowdown” corroborates the pattern. Bit 0x40 HW_THERMAL appears in badly ventilated racks; bit 0x04 SW_POWER_CAP appears if somebody left nvidia-smi -pl 500 set and nobody reverts it.

Operational implication. Any bit other than 0 or Idle (bit 0x01) sustained is an immediate alert. The recommended decoding: record the complete bitmap value in the log plus the attribute throttle.reasons.decoded=["HW_THERMAL", "HW_POWER_BRAKE"] on the OTel span. Without that, incident response does not know what to do.

Family 4 — Health (the catastrophic reports)

`DCGM_FI_DEV_XID_ERRORS` — the driver’s red codes

The analogy. The critical alarm lights in a nuclear control room. They do not climb gradually; they either appear or they do not. Each XID is a predefined code (XID 13 = graphics engine exception; XID 31 = MMU fault; XID 43 = stopped channel; XID 79 = GPU fallen off the bus; XID 95 = uncontained ECC), and each one has its documented procedure.

The documented anomaly. The most studied public case: Meta published that during the training of Llama 3 405B on 16,384 H100s over 54 days there were 419 unplanned failures, one every 3 hours at cluster scale. GPUs accounted for 148 (35 %) plus HBM3 for 72 (17 %), almost half of all failures. The paper “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100” (arXiv 2503.11901) quantifies with a different dataset (2.1M GPU-hours) that the H100 has 3.2× worse MTBE for uncorrectable ECC than the A100. The ByteDance MegaScale paper reports that XID 79 (“GPU fallen off the bus”) co-occurs with PCIe errors in 43 % of cases. The NVIDIA developer forum documents persistent cases of XID 31 (MMU fault) that follow the GPU when it is moved to a different PCIe slot, a hardware bug in the module, not in the backplane.

Operational implication. Any increment of the counter is an immediate alert: many XIDs demand a node reset or an RMA of the GPU. The XID-by-XID distinction matters: XID 13/43 is usually a software bug if it coincides with a recent change; XID 31/48/79/94/95 is usually hardware. Keep a canonical xid → procedure table. See the runbooks for the translation into concrete action.

`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL` — the errors that corrupt data

The analogy. An accounting ledger where sometimes someone erases an entry and rewrites it (corrected single-bit ECC: note a change in the margin and carry on) and sometimes someone burns two pages at once (double-bit: the information is lost, the audit has to stop).

The documented anomaly. The paper “Characterizing GPU Resilience” quantifies for the H100 that when XID 48 (DBE) appears, the job in flight dies with 100 % probability (5 out of 5 in the dataset studied). The documented recovery: drain the node, reset, complete the row remap, equalling around 19 hours of node downtime. HBM3 density explains the worse MTBE against HBM2e: there are more cells per unit area, and therefore a higher statistical probability of degradation. In Llama 3, HBM3 caused 72 of the 419 interruptions (17 %).

Operational implication. Any value > 0 is a critical alert. The GPU must be drained immediately, removed from the scheduler, fully reset, validated for row remap with nvidia-smi -q -d ROW_REMAPPER (Pending: No), and given an extensive smoke test before returning to the pool. If the row remap uses more than 4–8 spare pages on one GPU, plan a replacement in the next window: the degradation is progressive.

`DCGM_FI_DEV_RETIRED_DBE` — the pages marked for retirement

The analogy. The tiles a museum restorer marks with yellow tape because they are damaged. They pose no immediate danger (the room stays open), but the accumulation says the floor is degrading structurally and the full replacement has to be planned.

The documented anomaly. NVIDIA documents up to 512 spare pages per HBM bank on the H100; the RETIRED_DBE counter indicates how many have been used. Operators on NVIDIA forums report that above 4–8 retired pages on a specific GPU, the frequency of XID 48 rises. The pattern: a GPU with 6 retired pages today becomes 12 in a month, a first XID 48 two months later, and a forced drain.

Operational implication. A trend metric, not an immediate alert. Record the value per GPU and review it monthly; GPUs with rising values go into the proactive replacement plan before the catastrophic failure.

The five vLLM engine metrics

`vllm:num_requests_running` — how many requests fit in the batch?

The analogy. The number of cars a toll booth lets through at once. If the barrier opens for N at a time, car N+1 waits in the queue. Saturation shows up because the line does not get shorter.

The documented anomaly. Reaching the configured --max-num-seqs and staying there is the typical symptom of a cluster below its sizing; the engine admits up to the ceiling and no further. The query vllm:num_requests_running == max_num_seqs for more than 5 minutes indicates firm saturation.

Operational implication. Combine it with num_requests_waiting: if running is at the ceiling AND waiting > 0, you have to scale. If running is at the ceiling and waiting is 0, you are in the optimal regime (cluster used to the maximum with no queue).

`vllm:num_requests_waiting` — the primary saturation indicator

The analogy. The visible queue in front of the toll booth. As long as it is empty, the system flows; the moment a sustained queue forms, drivers start arriving late at their destination, and TTFT goes through the roof.

The documented anomaly. A public case in “11-Second Time to First Token on a Healthy vLLM Server” (Medium, Ingero, 2026): a server with no XIDs, no preemption, DCGM metrics all green, but num_requests_waiting sustained above 0 and a TTFT of 11 seconds. Issue vllm#16985 documents progressive degradation in long sessions: the queue grows slowly over hours without any other indicator moving. The root cause is not hardware, it is admission control: the arrival rate exceeds the completion rate and the system does not enqueue any more, it leaves requests in WAITING. Red Hat designates it as the primary saturation metric in its tutorial “5 steps to triage vLLM performance”.

Operational implication. The primary HPA metric in KEDA, see Autoscaling LLM on Kubernetes. Typical threshold: alert if avg_over_time(vllm:num_requests_waiting[5m]) > 5. For canaries: if the queue only forms in the v2 pool, it is a regression of the new model, not cluster load.

`vllm:gpu_cache_usage_perc` — the KV cache pool

The analogy. The capacity of an events hall where each guest takes up a variable amount of space. The maître d’ admits guests up to capacity; when a new guest arrives and there is no room, he throws out the guest who has been there longest to make space for the newcomer. That is vLLM’s preempt-on-OOM.

The documented anomaly. Issue vllm#5051 “Add num_requests_preempted metric” was born exactly out of operators observing degradation with no direct metric telling them how many requests were being thrown out. The official vLLM documentation confirms: “sustained gpu_cache_usage_perc above 90 % indicates the server is approaching its KV cache limit and will begin preempting sequences” (oldest-first). The distinctive visual pattern: a sawtooth near 100 % with preemption spikes. In swap mode, the latency of the preempted request explodes because there is a host↔device PCIe copy; in recompute mode (the default in V1), the preempted request redoes its prefill from scratch, which pushes its TTFT to double or triple.

Operational implication. If gpu_cache_usage_perc > 92 % is sustained, there are two levers: lower max_num_seqs (admit less concurrency, but none of it gets preempted) or raise gpu_memory_utilization (more pool, less VRAM for transient activations, a different risk). The choice depends on the workload. The metric that is directly missing, a preempted counter, is exported from vLLM v1.0 onwards as vllm:num_preemptions_total (see issue #5051).

`vllm:time_to_first_token_seconds` — the latency the client sees

The analogy. The time from a customer walking into a restaurant to receiving the first piece of bread at the table. Too long and the customer thinks they have been forgotten, even if the main course is going to arrive perfect.

The documented anomaly. Three documented patterns of recurring TTFT spikes:

Badly calibrated chunked prefill. Issue vllm#25677 (Qwen3-30B-A3B) reported prefill 10–11× slower with chunked prefill enabled than without it. Cause: a very low max_num_batched_tokens forces small chunks that do not fill the kernels. Issue vllm#7604 documents an equivalent regression on Llama-3-70B v0.5.4. The lever: raise max_num_batched_tokens to 4096–8192 for typical prompts above 2k.
Regression between engine versions. Issue vllm#8819 documents a regression of vllm:time_to_first_token_seconds_sum between minor versions. Issue vllm#11912 reports that with a prompt of around 8000 tokens, TPOT rose from 15.7 ms → 25.7 ms from v0.6.4.post1 onwards with no config change, a regression confirmed and trackable only with the metric.
Long-context prefill blocking decodes. The “11s TTFT on healthy server” case cited above: a 30k-token prefill monopolises the GPU for several seconds and the active decodes freeze. Fix: well-calibrated chunked prefill, or disaggregated serving (see Disaggregated serving).

Operational implication. Do not alert only on the absolute P95; alert also on the v2/v1 ratio when there is a canary (histogram_quantile(0.95, ..., version="v2") / histogram_quantile(0.95, ..., version="v1") > 1.10). If TTFT grows and the queue is stable, the bottleneck is prefill: not solvable by adding replicas, but there are levers in quantisation or chunked prefill.

`vllm:time_per_output_token_seconds` — streaming smoothness

The analogy. The speed at which the waiter brings the courses one after another after the first. If the next one takes a while, the diner notices something is wrong even though the first course arrived on time.

The documented anomaly. The distinctive pattern is the abrupt step when gpu_cache_usage_perc crosses roughly 85 %: TPOT jumps from 35 ms to 80 ms in a few seconds because the engine starts competing for the HBM with its own evictions. Issue vllm#35387 documents another anomalous case: MTP (speculative decoding) causing a 76 % latency regression on Qwen3-Next-80B-A3B-Instruct-FP8. The TPOT metric caught it before any client complaints were filed.

Operational implication. The difference from TTFT: if TTFT grows and Queue Time is stable, you are prefill bound; if TPOT grows at a steady rate, there is pressure on the HBM (KV cache pool or swap enabled). A secondary alert on the TPOT SLO, but also watch the derivative: TPOT rising 1 ms every 10 minutes is a latent regression that does not break the SLO yet but will.

The operational rule: read metrics by family, not in isolation

Three anti-patterns of the novice operator

Anti-pattern 1 — alerting only on absolute thresholds. An H100 at 87 % FB is not necessarily an alarm; an H100 at 87 % rising 2 %/min is. Alerts that fire on a fixed threshold without looking at the derivative produce twice the noise and half the value. Rule: for metrics with known dynamics (KV cache, FB, queue), alert on a sustained delta, not just on the level.

Anti-pattern 2 — confusing SBE with DBE. The DCGM_FI_DEV_ECC_SBE_VOL_TOTAL counter (single-bit, correctable) grows continuously on any HBM under load; it is not an alarm, it is physics. The one that matters is DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (double-bit, uncorrectable). Confusing them yields false negatives (not alerting on a real DBE) or false positives (alerting on a harmless SBE).

Anti-pattern 3 — treating SM_OCCUPANCY at 99 % as “saturated”. The LLM decode regime is memory-bound, not compute-bound; high SM occupancy with low TENSOR_ACTIVE and high DRAM_ACTIVE is normal. Sizing for “GPU at 60 %” and asking for more hardware when the cluster is saturated in HBM (not in SMs) means buying twice the GPUs without gaining throughput. Rule: always read SM_OCCUPANCY together with TENSOR_ACTIVE and DRAM_ACTIVE; on its own it means nothing.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter per node (a GPU Operator DaemonSet) emitting every 15 s; cardinality per GPU is around 80 series. A 16-GPU cluster gives roughly 1.3k base series, around 85k samples/min with a 15 s scrape.
vLLM /metrics per inference pod; each replica emits around 50 base series. For 16 replicas, around 800 additional series, around 3k samples/min.
Prometheus retention: 30 days at high resolution plus 1 year downsampled through a Thanos sidecar or Mimir. Estimated volume: 25–35 GB/day.
Alertmanager: the 6 critical alerts from the previous post plus derived alerts (delta, v2/v1 ratio, decoded throttle bitmap).

Each metric is worth exposing also as an OTel attribute on the spans of the GenAI tracing: gpu.fb_used_pct, gpu.dram_active, gpu.throttle_reasons.decoded. That lets you correlate a slow request with the state of the GPU at that instant, without jumping between dashboards.

What we have not covered (upcoming posts)

Per-alert runbooks — the translation of each anomalous metric into concrete action (drain, reset, RMA, scaling, rollback) in the next post: Incident response runbooks.
Tail sampling for metric ↔ trace correlation — what is preserved when an alert fires, for post-mortem investigation.
Per-tenant showback combining vllm:request_success_total × gen_ai.usage.* × DCGM_FI_DEV_POWER_USAGE to bill real energy cost.
Multi-tenant fairness metrics — when a tenant hogs the KV cache pool and how to detect it.

References

Meta — Faulty Nvidia H100 GPUs and HBM3 memory caused half of failures during Llama 3 training (Tom’s Hardware, 2024). tomshardware.com
Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100. arXiv 2503.11901. https://arxiv.org/html/2503.11901v3
ByteDance — Robust LLM Training Infrastructure at ByteDance. arXiv 2509.16293. https://arxiv.org/pdf/2509.16293
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv 2503.08311.
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts. arXiv 2503.05066.
NVIDIA — Analyzing Xid Errors with the Xid Catalog and Memory Error Management (docs.nvidia.com/deploy).
Dell — PowerEdge XE8640 with H100 - GPU Performance Issue HW Power Brake Slowdown - Active (KB 000220508).
Lenovo — Power brake reporting on H100 GPU (HT514380).
vLLM project — issues #5051 (preempted metric), #7604 and #25677 (chunked prefill regression), #11912 (long-prompt regression), #16300 (TP=8 worse than TP=4), #16985 (long-running degradation), #20783 (compressed-tensors no speedup), #35387 (MTP regression).
Red Hat — 5 steps to triage vLLM performance. https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
AI21 — Go big or go OOM: the art of scaling vLLM. https://www.ai21.com/blog/scaling-vllm-without-oom/
11-Second Time to First Token on a Healthy vLLM Server (Medium, Ingero, 2026).
NVIDIA — DGX SuperPOD Electrical Specifications (docs.nvidia.com/dgx-superpod).

Sources: the full URLs are linked inline on each reference.

The LLM inference router: the L7 switchboard we called a LoadBalancer in the canary post

Tue, 02 Jun 2026 03:00:00 +0200

This post is the natural continuation of Canary, blue-green and shadow for LLM models. There the promotion mechanics dumped all the traffic-splitting complexity into a box we called “LoadBalancer”. The description was operational, good enough to understand the choreography, but structurally vague: what actually does that splitting is an L7 inference router with LLM awareness, a piece of the stack in its own right (layer 1 of the seven layers) that deserves its own post.

TL;DR

In the previous post on canary we called LoadBalancer the piece that splits traffic between the stable v1 and candidate v2 pools. The description was enough to understand the flow, but technically it was blurry: neither an L4 LoadBalancer (kube-proxy, MetalLB, IPVS) nor a generic L7 HTTP LoadBalancer (NGINX or HAProxy with no extension) knows what a model is, what a version is, how many tokens a request costs, what prefix the prompt has or which replica has which KV cache hot. The right piece is an LLM inference router: an L7 proxy with explicit knowledge of the domain. It combines four functions: model catalogue (resolving model=llama-70b@v2 → service.namespace:port), traffic splitting (applying the canary weight with a deterministic hash or deliberate stickiness for A/B), cross-cutting policy (OIDC auth, per-tenant rate limit and quota, pre-prompt PII redaction, lightweight inline guardrails, gen_ai.* tracing propagation) and failover/degradation (if v2 falls over, redirect to v1; if the whole cluster is saturated, return 503 with Retry-After instead of queueing forever). The non-obvious piece that justifies its technical existence beyond the operational one is prefix-aware routing: the router decides which fleet replica each request goes to based on the prompt prefix, so that a RAG system with the same system prompt plus the same block of retrieved documents systematically hits the prefix cache (RadixAttention in SGLang, PrefixCaching in vLLM, KV reuse in TensorRT-LLM) of the same replica, multiplying the hit rate from 5–15 % (blind round-robin) to 60–85 % (prefix affinity). The concrete pieces in May 2026 are LiteLLM Proxy (the simplest option, OpenAI-compatible, declarative YAML catalogue), vLLM Production Stack router (specific to vLLM fleets, aware of the KV cache and the prefix), Envoy AI Gateway (LLM-aware Envoy filters, integrable with Istio), Kong AI Gateway (enterprise alternative with a plugin ecosystem), KGateway (CNCF, still incubating) and the NVIDIA Dynamo router (production-grade, aware of disaggregated prefill/decode serving). In the seven-layer stack it lives in layer 1 (gateway); in the five maturity levels it appears from level 3 onwards; in the seven-phase deployment cycle it is the last piece that F6 closes. This post includes a minimal manifest applicable to a generic 4×H100 SXM cluster.

You are here: DEPLOY (layer 1 of the stack)

The background: what the canary post called a “LoadBalancer”

In Canary, blue-green and shadow for LLM models we described the flow like this: “the LoadBalancer progressively splits traffic following a schedule: 1 % → 5 % → 25 % → 100 %”. It was a correct operational description, and the reader understood the choreography without needing more. But technically it left unnamed a piece that deserves explicit treatment, because neither of the two usual senses of “LoadBalancer” does what that paragraph assumed:

An L4 LoadBalancer (kube-proxy with iptables/IPVS, MetalLB, F5 BIG-IP in TCP mode) splits IP packets without looking inside the payload. It does not know which model is being requested, nor which version, nor how many tokens it carries, nor whether the client has quota. It cannot apply the canary weight “for model X version 2”: to it, every packet heading for the vllm-llama70b VIP is indistinguishable.
A generic L7 HTTP LoadBalancer (NGINX or HAProxy in HTTP mode with no extension, a ClusterIP Service with multiple backends) does split by URL and can route by header, but it does not understand the OpenAI-compatible body of the request. It does not know that {"model": "llama-70b", "messages": [...]} carries the routing key in the model field; it does not count tokens; it does not apply policies over LLM structures; it does not do prefix-aware routing because that requires parsing messages and hashing the common prefix.

The piece the canary post assumed was doing this work is an L7 inference router with LLM awareness. A layer in its own right, with its own configuration, its own CI/CD, its own metrics and its own pitfalls. This post names it and takes it apart.

The analogy: the switchboard and triage of a hospital with multiple specialities

A large hospital receives patients who arrive at A&E through different doors and who need different specialities: trauma, cardiology, paediatrics, oncology. There are three possible models of “front door”.

A single door with no triage. All patients wait in the same room and are sent in arrival order to the first free doctor, whatever their speciality. It works in a village surgery with a single general practitioner. When there are 200 patients a day and 12 specialities, it falls into dysfunction fast: the cardiologist treats sprains, the paediatrician treats heart attacks, specialised resources are wasted. This is the equivalent of the L4 LoadBalancer, which distributes bodies without understanding what they bring.

A door with a receptionist who asks about the symptom. Someone at the front desk asks “what’s wrong?” and directs the patient to the right corridor. The cardiologist sees only cardiology, the paediatrician only children. Better, but the receptionist is slow, does not gauge urgency and does not know the state of the rooms: they may send you to the cardiologist in corridor A when the one in B is free. This is the equivalent of a generic L7 HTTP proxy with path-based routing, which distributes by category but with no information about internal state.

Professional triage with full awareness. A trained triage nurse who knows the catalogue of specialities, knows which bay is busy and which is free, remembers the recurring patient whose file is already open in the system (and sends them to the same doctor for continuity), applies cross-cutting policy (checks insurance cover, records allergens, redirects to paediatric A&E if the patient is a minor) and, if the cardiology room goes down because the electrocardiograph has broken, redirects to the hospital on the other side of the city. This is the piece a large hospital needs. In LLM terms it is called an inference router.

The analogy holds down to the last detail, including the “file already open” one: the patient who returns to the same doctor is exactly the client whose prompt shares a prefix with the one from five minutes ago. If the router sends it to the same replica, that replica still has the KV cache hot and the request hits the prefix cache. If it sends it to a different replica because that one was “next in round-robin”, the KV cache has to be rebuilt from scratch and TTFT doubles. The triage nurse knows this. The blind LoadBalancer does not.

The four functions of the inference router

Function 1 — Model catalogue

The router keeps a declarative catalogue mapping model identity to a concrete deployment:

models:
 - name: "llama-70b" # stable alias
 version: "v2" # canary version
 weight: 5 # 5% of the traffic
 endpoint: "vllm-llama70b-v2.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: canary
 - name: "llama-70b"
 version: "v1"
 weight: 95
 endpoint: "vllm-llama70b-v1.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: stable
 - name: "embedding-multilingual"
 version: "v1"
 weight: 100
 endpoint: "tei-bge-m3.inference.svc.cluster.local:8080"
 capabilities: [embeddings]
 lifecycle: stable

The client sends {"model": "llama-70b", "messages": [...]} without knowing there are two replica pools behind it. The router resolves. If tomorrow you migrate from vLLM to SGLang for a specific version, the client never notices; you change the endpoint in the catalogue and that is it.

What this decoupling buys you is the freedom to move topology without breaking clients. What it costs is keeping the naming convention disciplined (llama-70b is always the stable alias; llama-70b@v2 is the specific version for canary). Without that discipline, the aliases get dirty with llama-70b-prod-fixed-real-final-v3 and the catalogue stops being navigable within a few weeks.

Function 2 — Traffic splitting

The splits from the canary post (1 % → 5 % → 25 % → 100 %) materialise here, not in the inference engine. The router computes a deterministic hash of the request_id (or of the user_id, if you want stickiness) and maps it to the weight range in the catalogue. For a weight of [v1: 95, v2: 5], 5 % of the hash space falls in v2 and 95 % in v1.

Three design decisions that matter:

Hash by random request_id = independent sampling. Each request is an independent observation of the v1 vs v2 distribution. It is the right setting for statistically comparable canaries.
Hash by user_id = per-user stickiness. The same client always sees the same pool. Useful for A/B testing with persisted conversational memory, but it breaks the statistical comparability of the canary because user populations are not symmetric, a pitfall explained in the previous post.
Hash by tenant_id = hard partitioning. Tenant A goes to v1, tenant B to v2. This is the pattern for clients with different SLAs, or for validating v2 in an internal tenant before exposing it to external clients.

Function 3 — Cross-cutting policy

Once, across all models, the router applies:

Auth: OIDC with JWT tokens validated against Keycloak / Authentik. Authorization: Bearer ... headers translated into tenant_id and roles.
Rate limit: token bucket per tenant (X req/min) or per model (Y req/min for llama-70b because it is expensive).
Quota: monthly quota of tokens consumed per tenant. The router counts gen_ai.usage.input_tokens + gen_ai.usage.output_tokens and rejects with 429 Quota exceeded when it runs out.
Pre-prompt PII redaction: Presidio or Llama Guard inline before the prompt touches the model. What the model does not see is not trained on, not logged, not leaked.
Lightweight inline guardrails: PromptGuard 2, Llama Guard 4, Granite Guardian, the ones covered in Guardrails and safety in LLMs, run in the router because their latency (30–150 ms) fits within the TTFT budget.
gen_ai.* tracing propagation: the router starts the parent span, propagates traceparent to the engine and emits the gen_ai.system, gen_ai.request.model and gen_ai.request.version attributes that OTel GenAI tracing consumes.
Semantic cache: for prompts that repeat exactly or with high semantic similarity (embedding cosine > 0.97 against the previous cache), it returns the cached answer without touching the engine. Typical saving in RAG with frequently asked questions: 20–40 % of requests.

Function 4 — Failover and degradation

The router knows the health state of each endpoint (active health probes every 5–15 s, recent TTFT latency) and decides:

If v2 returns persistent 5xx or does not respond, the circuit breaker opens: the router redirects the traffic that was going to v2 towards v1 until the probes go green again. This is automatic canary rollback in its simplest form.
If the whole cluster is saturated (all replicas reporting num_requests_waiting > N for T seconds), the router returns 503 Service Unavailable with Retry-After: 30 instead of queueing forever. It is better to tell the client “come back in 30 seconds” than to keep them waiting four minutes and then time out.
If there is multi-region or multi-cluster, cross-cluster failover via DNS or L7: the primary region goes down and the secondary router takes over.

The non-obvious piece: prefix-aware routing

This is the function a conventional LoadBalancer cannot do and the one that justifies an LLM-specific router beyond the four generic ones.

The KV cache in vLLM, SGLang and TensorRT-LLM can reuse common prefixes across requests, see KV cache. Specifically:

vLLM with --enable-prefix-caching: it detects that the current request shares a prefix (of a length that is a multiple of the block size, 16 tokens by default) with a previous request whose pages are still in HBM, and reuses those pages instead of reprocessing them.
SGLang with RadixAttention: it structures the cache as a radix tree indexed by tokens; each request hits the common path of the tree and only computes the tail.
TensorRT-LLM: a similar feature, called KV cache reuse.

The prefix cache hit rate is the key metric: every token that hits is a token that is not processed in prefill, reducing TTFT in direct proportion. For a typical RAG system, with a 400-token system prompt plus 2,000 tokens of retrieved documents plus a 50-token user question, the common prefix (system_prompt + docs) is 2,400 of the 2,450 total tokens. If the cache hits, prefill only processes 50 tokens instead of 2,450: TTFT falls to roughly a twentieth.

But the cache lives per replica, not globally. If two requests with the same 2,400-token prefix land on different replicas, both do the full prefill: the first one’s cache is no use to the second. The second pays the full cost.

With blind round-robin (any conventional LB), requests are spread uniformly across N replicas. For a cluster of 4 replicas and 1,000 requests with the same system_prompt + docs, each replica receives ~250 requests, but all 4 do their own “first prefill” and the following 249 benefit within their replica. The global hit rate is decent but not optimal. For traffic with many different system prompts and little intra-prefix repetition, the hit rate hovers around 5–15 %.

With prefix-aware routing, the router computes a hash of the prompt prefix (the first N tokens, or the system_prompt declared in messages[0]) and keeps an affinity table of hash → replica. All requests with the same prefix land on the same replica. The first pays the full prefill; the following 999 hit the cache. Global hit rate: 60–85 %.

The cost of implementing it: the router has to parse the request body (not just the HTTP header), apply a lightweight tokeniser or a byte-based hash, and maintain an LRU/consistent-hash affinity table that rebalances when a replica comes in or out. That is server work, not generic proxy work. vLLM Production Stack router implements it natively. NVIDIA Dynamo does too. LiteLLM has a beta in its enterprise version. Envoy AI Gateway is adding it as an experimental filter.

Correction (September 2026). The previous sentence about LiteLLM is false: prefix-aware routing does not exist in any version, open or paid, verified against the code of 1.102.0. The detail, what LiteLLM does have, and the real state of the field are in Prefix routing: what LiteLLM does not do.

The operational difference for a production RAG system: with prefix-aware routing, the same cluster serves 2–4× more requests without adding GPUs, simply because prefill disappears in most cases.

Token-aware load balancing

The second non-obvious piece. Classic round-robin splits by number of requests; but a 50-token prompt and an 8,000-token one cost radically different amounts (a factor of ~160× in prefill). Splitting equally by count severely unbalances the real load.

Token-aware load balancing adds up the expected prefill tokens (prompt length) and expected decode tokens (the client’s max_tokens) per active replica, and sends the new request to the replica with the lowest accumulated load. It is what both vLLM Production Stack and NVIDIA Dynamo implement as the default strategy when it is enabled.

The metric feeding the calculation is, once again, vllm:num_requests_running and vllm:gpu_cache_usage_perc, see GPU observability for LLM inference, ideally complemented with an estimator of the incoming prompt’s tokens. Mature routers use tiktoken or the model’s real tokeniser to count prompt tokens before choosing a replica.

Comparison of concrete pieces (May 2026)

Piece	LLM awareness	Prefix-aware	Token-aware LB	Multi-model	Semantic cache	Plug & play
LiteLLM Proxy	High	Beta (enterprise)	Yes	Excellent	Yes (Redis)	Very high
vLLM Production Stack router	vLLM-specific	Yes, native	Yes	vLLM only	No (external)	Medium
NVIDIA Dynamo router	High + disagg-aware	Yes	Yes	vLLM/SGLang/TRT-LLM	No (external)	Low
Envoy AI Gateway	Medium (filters)	Experimental	Yes	Yes	Via filter	Medium
Kong AI Gateway	Medium (plugins)	No	Yes	Yes	Yes (plugin)	Medium
KGateway	Medium	Roadmap	Yes	Yes	Roadmap	Low (CNCF incubating)
NGINX + custom Lua	Manual	No	Manual	Manual	No	Low (build it yourself)

LiteLLM Proxy is the default option to start with. OpenAI-compatible, simple YAML, supports the commercial providers plus any self-hosted OpenAI-compatible backend. The OSS version covers the four basic functions and semantic cache; prefix-aware routing and the enterprise version add advanced multi-tenancy.

vLLM Production Stack router is the right option if the fleet is 100 % vLLM. Aware of the KV cache, the prefix and the LoRA loaded per replica. It integrates better with native vLLM metrics.

NVIDIA Dynamo router is the most complete production-grade option, especially if you operate disaggregated serving (separate prefill workers and decode workers). It requires an NVIDIA-aligned stack.

Envoy AI Gateway and Kong AI Gateway are the options if the organisation already has Envoy or Kong as its corporate gateway and wants to extend it with LLM awareness without introducing another new piece.

Minimal manifest: LiteLLM Proxy on a generic cluster

apiVersion: v1
kind: ConfigMap
metadata: { name: litellm-config, namespace: inference }
data:
 config.yaml: |
 model_list:
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v1.inference.svc:8000/v1
 weight: 95
 model_info:
 version: v1
 lifecycle: stable
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v2.inference.svc:8000/v1
 weight: 5
 model_info:
 version: v2
 lifecycle: canary
 - model_name: embedding-multilingual
 litellm_params:
 model: openai/bge-m3
 api_base: http://tei-bge-m3.inference.svc:8080
 router_settings:
 routing_strategy: least-busy # basic token-aware
 num_retries: 1
 timeout: 60
 general_settings:
 master_key: "os.environ/LITELLM_MASTER_KEY"
 database_url: "os.environ/DATABASE_URL"
 litellm_settings:
 cache: true
 cache_params:
 type: redis
 host: redis.inference.svc
 port: 6379
 similarity_threshold: 0.97
 success_callback: ["langfuse"]
 failure_callback: ["langfuse"]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: litellm-router, namespace: inference }
spec:
 replicas: 3
 selector: { matchLabels: { app: litellm } }
 template:
 metadata: { labels: { app: litellm } }
 spec:
 containers:
 - name: litellm
 image: ghcr.io/berriai/litellm:v1.55.0
 args: ["--config=/config/config.yaml", "--port=4000", "--num_workers=4"]
 ports: [{ containerPort: 4000, name: http }, { containerPort: 4000, name: metrics }]
 env:
 - { name: LITELLM_MASTER_KEY, valueFrom: { secretKeyRef: { name: litellm-secret, key: master_key } } }
 - { name: DATABASE_URL, valueFrom: { secretKeyRef: { name: litellm-secret, key: db_url } } }
 - { name: LANGFUSE_PUBLIC_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: public } } }
 - { name: LANGFUSE_SECRET_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: secret } } }
 volumeMounts: [{ name: config, mountPath: /config }]
 readinessProbe: { httpGet: { path: /health, port: 4000 } }
 volumes: [{ name: config, configMap: { name: litellm-config } }]
---
apiVersion: v1
kind: Service
metadata: { name: litellm-router, namespace: inference }
spec:
 selector: { app: litellm }
 ports: [{ name: http, port: 80, targetPort: 4000 }]
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: litellm-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: litellm } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s

The end client points at litellm-router.inference.svc:80/v1/chat/completions, sets model=llama-70b, and the router decides on each request whether it goes to v1 (95 %) or v2 (5 %), applies the rate limit, looks in the semantic cache, propagates tracing to Langfuse, and translates from OpenAI-compatible to the OpenAI-compatible API of the destination vLLM. Three router replicas for HA and so the gateway itself can scale horizontally with KEDA if needed, see LLM autoscaling on Kubernetes.

Four operational pitfalls

Pitfall 1 — the router becomes a SPoF if it is not replicated. Three or more replicas of the router itself, behind a LoadBalancer Service (this one genuinely L4) with healthchecks. A single router replica means every configuration deploy closes the whole service for a few seconds.

Pitfall 2 — the router’s latency adds to the model’s. Each function adds milliseconds: body parsing (5–10 ms), JWT auth (2–5 ms), rate limit (1–2 ms), PII redaction with Presidio (20–80 ms), inline guardrails with Llama Guard (50–150 ms), prefix hash (5–10 ms), token counting with a tokeniser (10–30 ms). In total, 100–300 ms of overhead before touching the engine. If the model’s TTFT is 400 ms and the router’s is 200 ms, the client sees 600 ms, so it is worth measuring what each function costs and disabling the non-critical ones in the low-latency path.

Pitfall 3 — the catalogue drifts from the real state of the cluster. The router believes vllm-llama70b-v2 exists because it is in its YAML, but the deployment was withdrawn three days ago and nobody updated the config. The router returns 502 on 5 % of the traffic. Fix: validate the catalogue against kubectl get svc in CI; no endpoint in the catalogue may point at a non-existent Service. Or better: the router discovers the available endpoints dynamically via a label selector (app=vllm,model=llama-70b) and applies the catalogue weights over the ones that are alive.

Pitfall 4 — semantic cache with an outdated embedding. The semantic cache compares the new prompt’s embedding against the embeddings of cached prompts. If you update the embedding model (see RAG corpus curation), the distances are computed in a different space and the cache stops working correctly (false hits or false misses). Policy: the cache is invalidated when the embedding model changes; generations are never mixed.

Fit within the stack and maturity

In the seven-layer stack, the router is layer 1: the front door that precedes the inference engine (layer 2), the KV cache + PagedAttention (layer 3) and the rest. It is the only piece that sees all the traffic from the outside; any policy not applied here gets duplicated N times in the engines.

In the five maturity levels, the router appears from level 3 (MANAGED) onwards: without OIDC + RBAC + cert-manager + default-deny NetworkPolicy, the router has nobody to authenticate and nobody to apply quotas to; before level 3, what you should build is a minimal proxy with no pretension of a catalogue. Platforms that try to have a polished router at level 1 end up with a large YAML nobody maintains.

In the seven deployment phases, the router is what closes F6: the last atomic step that puts the cluster into production. Without a router, F6 does not finish, because the catalogue, the quotas, the canaries and the failovers are a necessary condition for opening production traffic.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB, the inference router consumes modest resources: 3 replicas of the router pod (2 CPU cores, 4 GiB of memory each) are enough to sustain thousands of RPS because its work is light (parsing, hashing, routing, no inference). The router lives on non-GPU nodes of the cluster (control plane nodes or general workload nodes), and never consumes nvidia.com/gpu.

Traffic volume that a LiteLLM with 3 replicas and 4 workers each will sustain: 2,000–5,000 RPS routing to a vLLM backend, with 80–150 ms of overhead on the full path (auth + rate limit + cache check + propagation). If more is needed, scaling the router with KEDA on litellm:requests_per_second is trivial.

For larger clusters (16+ GPU nodes), consider the vLLM Production Stack router or the NVIDIA Dynamo router, which are more complex but squeeze out the prefix-aware routing and token-aware LB that LiteLLM OSS does not cover. For multi-region clusters, Envoy AI Gateway with Istio Service Mesh is the standard choice.

What we have not covered (upcoming articles)

Deep comparison of LiteLLM vs vLLM PStack vs Dynamo, with prefix-aware benchmarks on a real on-premise cluster.
Semantic cache with Redis Stack + RedisVL: hit rate, false positives, TTL policy.
Multi-region routing: how the router decides between the DC1 and DC2 clusters based on latency, health and load.
AI Gateway specific features: token-bucket cost-based rate limiting (penalising long prompts), guardrails policy engine in the router.
Migration path: how to introduce a router into a cluster that already has clients pointing straight at the vLLM service, with no downtime.

References

LiteLLM project — litellm.ai (documentación de Proxy, routing strategies, semantic cache).
vLLM Production Stack — github.com/vllm-project/production-stack (router con prefix-aware nativo).
NVIDIA Dynamo — developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/ (router production-grade con disaggregated-aware).
Envoy AI Gateway — gateway.envoyproxy.io/docs/tasks/ai-gateway/ (proyecto en gestación dentro de Envoy).
Kong AI Gateway — konghq.com/products/kong-ai-gateway (proxy enterprise con plugin LLM).
KGateway — kgateway.dev (alternativa CNCF en gestación).
Zheng et al. — SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) — RadixAttention y prefix caching.
vLLM project — Automatic Prefix Caching (docs.vllm.ai/en/latest/features/automatic_prefix_caching.html).
Patel et al. — SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024) — base teórica del routing prefill/decode aware.

Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

Mon, 01 Jun 2026 16:30:00 +0200

This post complements those on LLM autoscaling on Kubernetes (the autoscaler coexists with the rollout and must respect it), GPU observability for LLM inference (the metrics that act as gates come from there), Evals for LLMs (the eval that decides whether the new model is ready), LLM-as-judge (the technique that puts the “quality” into the canary gate) and Retrain: closing the loop (the previous step that the new model comes out of).

TL;DR

Promoting a new version of an LLM model to the production cluster without cutting traffic or breaking the SLO demands progressive deployment. The three canonical strategies, blue-green, canary and shadow, answer different questions and have different costs. Blue-green: a complete new pool brought up in parallel, atomic load balancer switch. Instant rollback (point back at the old pool); it demands twice the GPUs during the window. Canary: traffic is shared progressively between the old version and the new one (1 % → 5 % → 25 % → 100 %), measuring regression gates at each step; it consumes incrementally less hardware but exposes real users to the new model from the first percentage point. Shadow / mirror: the old model serves 100 % of the real traffic to the client and, in parallel, a copy of every request goes to the new model without returning its response to the user; it isolates you from quality risk but spends the new model’s GPU on responses nobody consumes, and it does not work well with long SSE streaming. The choice depends on three factors: available GPU budget, service criticality and the availability of a fast automatic eval. The five regression metrics any LLM canary should gate on are: TTFT P95, error rate (HTTP 5xx + premature finish_reason="length"), quality score with LLM-as-judge over a golden set, statistical drift of output embeddings (Wasserstein or KL against the baseline distribution) and cost per request (tokens/s and kW/request). On Kubernetes, Argo Rollouts manages traffic and the AnalysisTemplates as automatic gates; Flagger is the more opinionated alternative. vLLM v1 does not support hot model swap robustly as of May 2026, so the rollout unit is the whole replica (deployment v2 alongside deployment v1). The three specific pitfalls: LB sticky sessions break the statistical comparability of the canary (a client A always lands on the new one, B on the old one, so the populations are not equivalent); semantic eval with LLM-as-judge takes 2–8 seconds per sample and is no use as a real-time gate (it is used in post-analysis or offline before promotion); SSE streaming complicates shadowing because the new model’s response has to be discarded without affecting the old one’s. This post includes a minimal Argo Rollouts manifest applicable to a generic cluster with the NVIDIA GPU Operator.

You are here: DEPLOY (and the transition to RETRAIN)

A new model does not appear in the cluster by magic: it comes from the retrain loop or from a weights provider update. The step between “I have an artifact that passed offline eval” and “it is serving 100 % of the traffic” is exactly this post.

The analogy: opening night of a play

A theatre company is about to premiere a new version of a play that has been running successfully for a year. The company knows several hard facts: the current audience pays for a consistent experience; a bad opening damages the business for months; but premiering nothing leaves the company obsolete against the competition.

The three opening routes the management can choose are the same three of the LLM rollout.

Dress rehearsal behind closed doors (shadow / mirror). The new actors perform the whole play to an empty theatre. There is no audience; nobody buys a ticket. Three full run-throughs are enough to check continuity, timing and the chemistry of the cast. It is expensive because there are salaries and theatre rent, but it does not expose the audience to risk. Useful when the new cast is untested and the director wants to see how it holds up over a complete performance before selling it. In LLM terms: the new model processes every real request in parallel with the old one but its responses are discarded; you spend the new model’s GPU on responses nobody sees.

Cast by performance, alternating (canary). Instead of changing the whole cast at once, Thursday performances belong to the new cast, Friday ones to the old one, Saturday ones half and half. The management reads the visitors’ book comments and the audience numbers performance by performance, deciding after two weeks whether to promote the new cast to permanent or withdraw it. Cheaper than the dress rehearsal because the performances still sell tickets, but it exposes a real audience to risk from the first Thursday. In LLM terms: traffic is shared progressively between the old version and the new one, measuring gates at each step.

Double company with an atomic switch (blue-green). The company hires the new cast, prepares it for a month behind closed doors, and one Saturday announces: “from the next opening night every performance is with the new cast”. If the first performance goes badly, they go back to the old cast in 24 hours, but during that month of preparation double salaries are paid to both companies. In LLM terms: two complete pools of the same size, instant LB switch from one to the other, rollback in seconds if the metrics break.

The analogy also supports the decision: the choice depends on how critical the play is for the business (criticality of the LLM service), how much budget there is to sustain two casts at once (GPU budget), and how much confidence there is in the new cast from the chamber rehearsals (offline eval prior to the canary).

The three strategies in detail

Blue-green

The operator keeps two replica pools identical in size: the blue one (production version v1) and the green one (candidate version v2). When v2 is validated offline (eval passed, smoke tests), the LoadBalancer switch redirects 100 % of the traffic from blue to green in a single step. If the SLO metrics break, the switch goes back in seconds.

Cost: 2× GPUs for the whole window (preparing v2 + the post-switch observation window). For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), preparing the blue-green needs 16 extra GPUs for 1–3 days.

Risk: the switch is atomic. If v2 has a problem that did not show up in offline eval but does show up at scale (for example, edge cases that only appear at 200 RPS), 100 % of users notice it at once. Rollback is instant, but the requests of the first minute after the switch were already affected. So blue-green is preferable when you have high confidence in v2 (a minor change: same architecture, same format, just a new version of the weights) and you prioritise immediate rollback over gradual exposure.

Canary

The operator deploys v2 with a small number of replicas (typically 1) next to the v1 pool. The LoadBalancer shares the traffic out progressively on a schedule: 1 % for 30 minutes → 5 % for 1 hour → 25 % for 2 hours → 50 % for 4 hours → 100 %. Between each step, an analysis gate evaluates regression metrics over the traffic that is already landing on v2. If the gate fails, the rollback withdraws traffic from v2 automatically and leaves v1 serving everything.

Cost: incremental. At the start (1 % of traffic) one v2 replica is enough; at 50 % you need half as many v2 replicas as v1 has in total. Peak extra GPU during the canary: about 30–50 % above baseline.

Risk: real users are seeing v2 from the first 1 %. If v2 produces responses with degraded quality but normal TTFT and error rate, the affected users perceive the degradation without the gate detecting it (unless the gate includes quality drift, which is slow). So canary is preferable when you have medium confidence in v2 (a significant change: different architecture or training) and you accept that a low percentage of users will be guinea pigs.

Shadow / mirror

The LoadBalancer sends 100 % of the real traffic to v1 (which answers the client) and duplicates every request towards v2 (whose response is discarded or stored for offline analysis). The client never sees v2; it is never exposed to the risk.

Cost: 100 % extra compute for v2 with no direct user value for the whole shadow window. For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), a shadow of the same size consumes 16 extra GPUs full time.

Risk: shadow is the safest for the user. But it has two serious limitations: (a) if v2 has a bottleneck that makes the copied request to the shadow take a long time, the shadowing proxy can consume LB connections; it must be out-of-band (asynchronous); (b) long SSE streaming complicates mirroring because you have to keep two parallel streams and discard one while the other flows to the client. Common pattern: shadow only non-streaming requests (short completions, classification), manual offline eval of the streaming requests.

The five regression metrics that act as a gate

Without automatic gates, the “canary” is just a nice name for “manual rollout with a variable percentage”. The gates are the piece that turns the canary into a defensible operation.

Metric 1 — TTFT P95. Comparison of the new model’s P95 against the baseline (v1) P95 in 5-minute windows. Gate: ttft_p95(v2) / ttft_p95(v1) < 1.10. It detects prefill latency regressions (a slower new model) or engine problems (suboptimal config). Source: vllm:time_to_first_token_seconds_bucket (see GPU observability for LLM inference).

Metric 2 — Error rate. Sum of HTTP 5xx + unexpected 4xx + rate of premature finish_reason="length" (responses cut short because the new model does not generate EOS). Gate: error_rate(v2) - error_rate(v1) < 0.01 (1 percentage point). It detects engine crashes, a broken tokenizer, generation problems. Source: vllm:request_success_total{status=...}.

Metric 3 — Quality score (LLM-as-judge). Over a golden set of 200–1,000 representative prompts, v1 and v2 are run offline and a judge model (typically larger: GPT-4 class, Claude, local Llama 405B) scores each pair. Typical gate: mean_score(v2) >= mean_score(v1) - 0.05. This metric is not measured in real time during the canary: judge inference takes 2–8 seconds per sample and does not scale as an inline gate. It is used as an offline pre-promotion gate (before starting the canary) and as a post-mortem over a sample of real traffic captured during the canary. See LLM-as-judge for the mechanics.

Metric 4 — Statistical output drift. For every request that lands on v2 during the canary, embed the response with a lightweight embedding model (e5, BGE) and compare the distribution of v2 embeddings against the distribution of the v1 baseline over the same window. Usable metrics: Wasserstein distance, KL divergence, or more simply, comparing means and variances per dimension. Gate: normalised distance < a calibrated threshold (typically Wasserstein < 0.15). It detects subtle changes in style, length and vocabulary that LLM-as-judge does not capture without going through it too. It is fast: the lightweight embedding takes around 50 ms per response.

Metric 5 — Cost per request. Output tokens per request and kW per request. Gate: cost_per_request(v2) / cost_per_request(v1) < 1.20. It detects new models that generate significantly longer responses or that consume more energy for the same load (quantisation degradation, optimisations failing). Without this gate, an “update” can silently double the bill.

Metric	Type	Measurement latency	Typical gate	Detection
TTFT P95	Quantitative	5 min	`< 110% baseline`	Latency regression
Error rate	Quantitative	1 min	`< 1pp over baseline`	Crashes, generation broken
Quality (LLM-judge)	Offline semantic	hours, over golden set	`> baseline − 0.05`	Functional quality
Statistical drift	Statistical	~5 min	Wasserstein < 0.15	Style, length, vocabulary
Cost per request	Quantitative	5 min	`< 120% baseline`	Economic/energy efficiency

The mechanics on Kubernetes: Argo Rollouts

Argo Rollouts extends the standard Kubernetes Deployment with a new Rollout resource that orchestrates traffic progression and automatic analyses. It integrates with any service mesh (Istio, Linkerd) or ingress controller that supports traffic splitting (NGINX, Traefik, Gateway API).

A minimal example of a 1 → 5 → 25 → 100 % canary with TTFT and error rate gates:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: vllm-llama70b }
spec:
 replicas: 4
 strategy:
 canary:
 canaryService: vllm-llama70b-canary
 stableService: vllm-llama70b-stable
 trafficRouting:
 nginx:
 stableIngress: vllm-llama70b-ingress
 steps:
 - setWeight: 1
 - pause: { duration: 30m }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 5
 - pause: { duration: 1h }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 25
 - pause: { duration: 2h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 50
 - pause: { duration: 4h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 100
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata: { labels: { app: vllm-llama70b } }
 spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args: [ --model=/models/llama-70b-fp8-v2 ]  # new version
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: ttft-error-gate }
spec:
 metrics:
 - name: ttft-p95-ratio
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 1.10
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
 /
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))
 - name: error-rate-diff
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 0.01
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 sum(rate(vllm:request_total{version="v2",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v2"}[5m]))
 -
 sum(rate(vllm:request_total{version="v1",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v1"}[5m]))

If any of the AnalysisTemplates fails, Argo Rollouts backs out automatically: it sets weight=0 on the canary, alerts the operator and keeps v1 serving 100 %. Human operation is reduced to investigating the failure and deciding whether to relaunch or abort.

Flagger offers a more opinionated alternative: weight progression is automatic as a function of metric success instead of a fixed pause; the operator defines a target (maxWeight: 100, stepWeight: 10, metrics: [...]) and Flagger raises or lowers it according to behaviour. Both are mature as of May 2026; the choice is usually dictated by which service mesh is already in the cluster.

The vLLM detail: why there is no “hot swap” of the model

As of May 2026, vLLM v1 does not support hot swapping the model inside the same replica without restarting the engine. The --model argument is evaluated at start-up; changing it requires re-instantiating the LLMEngine, which restarts connections and discards the KV cache. So the rollout unit is the whole replica: you do not do “v1 loads the new model on one of its GPUs” but “a v2 replica is brought up next to a v1 replica and traffic is shared out via the LB”.

TensorRT-LLM with Triton has a similar mechanism: changing the model requires a reload of the Triton backend. SGLang does not support robust hot swap either. The operational consequence: LLM rollout is always going to cost extra GPUs during the window, and the choice between blue-green, canary and shadow is exactly the question of how many extra and for how long.

The three pitfalls specific to LLM rollout

Pitfall 1 — sticky sessions break canary comparability. If the LoadBalancer does session affinity by client IP (common in NGINX and Traefik with loadbalancer.kubernetes.io/session-affinity: ClientIP), a user A always lands on v2 while B always lands on v1. Load distributions, prompt profiles and client behaviour are not random between the two pools, which statistically invalidates any gate comparison. Solution: for a canary, disable session affinity (sessionAffinity: None) or use affinity by random request-id. If the client app requires stickiness for functional reasons (conversational memory persisted in cache), canary is not the right strategy; use blue-green or shadow.

Pitfall 2 — LLM-as-judge is not an inline real-time gate. The temptation to use quality score as a live gate is strong, but the judge’s latency (2–8 s per sample) makes it unfeasible to evaluate more than a 1–2 % sampling of the traffic, and the results arrive minutes late. Operational solutions: (a) offline pre-canary eval over a golden set as a prerequisite for starting (if it fails, the canary does not even begin); (b) during the canary, capture requests + responses from v2 in real time and run the judge asynchronously in a batch job that finishes before the next step; (c) use statistical embedding drift as a fast proxy for inline quality, and reserve the judge for intermediate gates between steps.

Pitfall 3 — SSE streaming complicates shadowing. Classic traffic mirroring (NGINX mirror, Istio MirrorPolicy) is designed for request/response HTTP: it copies the request, lets the primary server answer the client, and duplicates the request to the secondary, discarding the response. With SSE, the secondary’s response is a continuous stream of several seconds, and keeping two streams in parallel doubles the load on the proxy. Solutions: (a) shadow only non-streaming requests (chat without streaming, embeddings, classification, batch eval), (b) shadow the streaming traffic but with a short timeout on the secondary (discard the shadow if it takes longer than 30 s), (c) replace the shadow with a small-weight canary (1 %), which does handle streaming well.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB = 16 GPUs, serving Llama 70B FP8 with TP=4 (4 possible replicas, one per node):

Blue-green: impossible to keep two complete pools of 4 replicas without extra GPUs. Practical solution: blue-green with reduced pools (2 v1 replicas + 2 v2 replicas) during the window, with accepted capacity degradation (half the sustained RPS SLO), or having a parallel cluster (another node) reserved for rollouts.
Canary: feasible. Start with 3 v1 replicas + 1 v2 replica (25 % nominal weight, though the traffic weight also varies). Move to 2 v1 + 2 v2 at 50 %, then 1 v1 + 3 v2, finally 0 v1 + 4 v2.
Shadow: awkward because of the GPU cost. Reserve it for pre-canary validation of major changes, during a short window (4–8 hours) with shadowed traffic limited to a sample (10–20 % of requests, not 100 %).

For clusters of 8 GPU nodes, the three patterns are all sustainable. The operational rule: the rollout budget is typically 25–30 % of the cluster’s sustained capacity, and buying for the peak plus that head-room makes the capacity planning numbers add up.

What we have not covered (upcoming articles)

Multi-region rollouts: how to coordinate a canary when the cluster is geographically distributed.
A/B testing of prompts (not of models): the same model with two different system prompts, measuring conversion.
Embedding rollback: changing the embedding model of a RAG system means re-embedding the whole corpus, so the canary mechanics are different. See RAG corpus curation.
Feature flags for LLM: granularity per tenant or per feature within the same model.
End-to-end continuous deployment: integration with the retrain pipeline so a new adapter is promoted automatically after passing evals.

References

Argo Rollouts project — argoproj.io/argo-rollouts (Rollout and AnalysisTemplate CRDs).
Flagger project — fluxcd.io/flagger (alternative with automatic progression).
Istio — Traffic Mirroring (mirror configurable at VirtualService level).
NGINX Ingress — Canary annotations (nginx.ingress.kubernetes.io/canary-*).
vLLM project — issue tracker on hot model swap (status as of May 2026: in design, not production-ready).
Hou et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024) — reference on goodput metrics applicable to canary gates.
Bürkner et al. — Statistical methods for detecting model drift in production (various articles on Wasserstein and KL in ML monitoring).

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

Mon, 01 Jun 2026 16:00:00 +0200

This post complements those on GPU observability for LLM inference (where the metrics that feed the HPA come from), Capacity planning (what ceiling and what head-room the autoscaler assumes) and Continuous batching (which explains why num_requests_waiting is the primary metric).

TL;DR

Classic Kubernetes autoscaling, HPA on cpu or memory, is no use for LLM inference. The reason: the vLLM pod consumes little CPU (the GPU does the work) and the process RSS is flat; both metrics can sit at 30 % while the GPU is saturated and the request queue grows unchecked. The four viable signals that do respond to the real load are: vllm:num_requests_waiting (the queue, the primary metric), vllm:gpu_cache_usage_perc (pressure on the KV cache pool), TTFT P95 via the vllm:time_to_first_token_seconds_bucket histogram (the SLO guarantee) and the batch fill ratio num_requests_running / max_num_seqs (utilisation of the concurrency ceiling). For an HPA to consume Prometheus metrics you need an adapter; as of May 2026 there are two mature options: prometheus-adapter (cluster-wide sigma, static configuration, external.metrics.k8s.io output) and KEDA (ScaledObject with a Prometheus trigger, configurable polling, optional scale to zero, cron integration). KEDA is the dominant option for LLM on a generic cluster because it solves the “warm pool + cron + engine metric” pattern in a single CRD. The dominant operational challenge is not the scaling logic but the cold start: a vLLM pod with Llama 70B BF16 (140 GB) takes between 90 seconds (model pre-cached on a local PV) and 6 minutes (image pull + model download from object store) to serve the first token. The five levers that cut it down are an image pre-pulled via DaemonSet, the model cached on a regional PV or tmpfs, a warm pool with minReplicaCount > 0, predictive scaling via KEDA cron when the traffic pattern is predictable (offices 9–18 h), and parallel model download. The three specific LLM scale-down pitfalls: cutting streaming SSE connections halfway through a response (graceful drain with terminationGracePeriodSeconds ≥ 60 s), scale-out/in oscillation from a badly calibrated stabilization window, and forgetting that the HPA only scales pods; GPU nodes are scaled with cluster-autoscaler over labelled nodepools. This post includes the minimal YAML manifests.

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

An artisan bakery has three wood-fired ovens. Each oven takes 25 minutes to reach temperature from cold. Once hot, it bakes bread continuously with a run of 18 minutes per batch. The manager wants to maximise bread sold per day without burning wood for nothing, and she knows three things: that there is a demand peak at 7:30 every morning, that Mondays sell almost nothing, and that when the bread on the counter runs out the customers go to the supermarket next door.

The cheap strategy, lighting ovens when there is a queue in the shop, does not work. By the time the queue grows and the manager lights the second oven, that oven will not be ready until 25 minutes later; the customers in that window are lost. The “queue at the counter” signal arrives late.

The smart strategy: light the second oven at 6:55, ahead of the predictable 7:30 peak, and leave it running until 10:00 even though the queue drops at 8:15. Keep the third oven off from Monday to Wednesday because demand does not arrive; light it proactively on Thursdays at 12:00 because historically it goes up. Keep a stock of pre-proved raw dough in the chiller so that when the oven is ready the bread goes in within 30 seconds and there is no two-hour proving wait.

Autoscaling an LLM inference cluster works the same way:

Lighting ovens from cold = reactive scale-out when the queue grows (slow, loses customers).
Proactive cron = predictive scaling when the pattern is known (working hours, expected peaks).
Pre-proved dough = warm pool of replicas with the model loaded but under 0 load.
Switching off ovens with no bread inside = scale-down respecting the active streams (you do not shut the oven with bread in it).

The key metric, “how many customers are in the queue”, is called num_requests_waiting. The metric that says “the oven is about to run out of dough for new loaves” is called gpu_cache_usage_perc. And the quality-of-service metric, “how long the first loaf takes to come out when a new customer walks in”, is called TTFT.

Why HPA on CPU is no use

The classic Kubernetes HPA looks at the pod’s resource.cpu. For a conventional HTTP service, Node.js or a REST API, CPU moves linearly with traffic and the HPA scales with reasonable accuracy. For a vLLM or SGLang pod on GPU, the pod’s CPU typically lives between 5 % and 15 % regardless of whether the GPU is at 30 % or 99 % load: the real work is done by the device, not the process. Result: the CPU-based HPA never triggers scale-out even though the GPU is bursting, and clients pile up in the queue until TTFT P95 crosses the SLO. The operator discovers the problem through the TTFT alert, not through the HPA.

memory is no use either: the vLLM process RSS is flat after startup (model plus buffers loaded in one go); it does not reflect the real pressure on the GPU. The only things that rise and fall with the useful inference load are metrics the engine publishes explicitly: request queue, KV cache pool, SLO latencies. Without an adapter that makes them visible to the HPA, autoscaling is blind.

The four viable signals

Signal 1, vllm:num_requests_waiting (queue). It is the most direct metric: how many requests are waiting to enter the batch. It reacts the instant the target concurrency saturates. It is robust against model changes (the number of requests is the same concept whether it is Llama 7B or 70B). It is the primary metric of the LLM HPA. Typical threshold: target = 5 requests waiting on average; if the queue grows above 5 sustained for 2 minutes, scale out.

Signal 2, vllm:gpu_cache_usage_perc (KV pool). It moves before the queue does: the KV pool fills up while the batch slots are still free, until the engine starts rejecting new requests for OOM prevention and the queue forms. It is therefore predictive: it triggers scale-out before the client notices degradation. Typical threshold: target = 0.85 (85 % of pool used).

Signal 3, TTFT P95. The contractual guarantee. If TTFT P95 leaves the SLO, scale out even though the queue and the KV pool look reasonable (there may be a spike of long prompts). It is reactive, it leaves the SLO before your HPA reacts, but it works as a final guardrail.

Signal 4, batch fill ratio. The num_requests_running / max_num_seqs quotient (the latter is engine config, not a metric). Useful for scale-down: if the ratio stays below 0.4 for 10 minutes, there is spare capacity and replicas can be reduced safely.

The recommended policy combines all four: the queue and the KV pool trigger scale-out (whichever arrives first), TTFT confirms it as a guardrail, and the batch fill ratio manages scale-down. Implementing that in a single HPA demands external metrics; KEDA makes this manageable.

The wiring: KEDA as the Prometheus adapter

KEDA introduces two main CRDs: TriggerAuthentication (how to authenticate against the source) and ScaledObject (which deployment to scale with which triggers). For a vLLM deployment with Prometheus as the source:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-llama70b-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama70b
 minReplicaCount: 2 # warm pool
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300 # 5 min before scale-down
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600 # large window to avoid oscillation
 policies:
 - type: Pods
 value: 1
 periodSeconds: 120
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - type: Pods
 value: 2
 periodSeconds: 60
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue_depth
 threshold: "5"
 query: |
 avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv_cache
 threshold: "0.85"
 query: |
 avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_ttft_p95
 threshold: "1.5"
 query: |
 histogram_quantile(0.95,
 sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{deployment="vllm-llama70b"}[5m])))

Three non-obvious operational details:

minReplicaCount: 2. This is the warm pool. Keeping at least two replicas guarantees availability if a node is lost and absorbs spikes without waiting for the cold start of the first scale-up. Dropping it to 0 saves GPU off-peak but introduces 90 s–6 min of latency for the first new client.

stabilizationWindowSeconds: 600 on scale-down. Ten minutes. Models are not nginx: if a replica closes prematurely and two minutes later there is another peak, the cold start of a new pod is what the client waits. Better to keep extra replicas around for twice as long as you would for a normal web service.

scaleUp: stabilizationWindowSeconds: 30. Thirty seconds. Scale-out has to be fast: the new pod’s cold start adds its own delay, and if on top of that the HPA waits several more minutes before triggering, the SLO is already broken.

The big operational problem: cold start

A vLLM pod loading Llama 70B goes through these phases before serving the first token:

Phase	Typical time	Can be accelerated with
Image pull (4–6 GB)	30–90 s	DaemonSet pre-pull
Model download (140 GB BF16)	60–300 s	Cached regional PV, S3 + multi-thread
Loading the model into HBM	30–90 s	tmpfs or local NVMe
CUDA graph capture	20–60 s	`--enforce-eager` (slower at runtime but fast startup)
PagedAttention warmup	5–15 s	—
Health check ready	10–30 s	probe tuning

Total without optimisation: 4–10 minutes. That is how long a new replica takes to absorb traffic. With all the levers combined: 45–90 seconds. The difference between those two numbers is the main platform work for LLM autoscaling.

The five levers

Lever 1, pre-pulled image. A trivial DaemonSet runs ctr image pull (or crictl pull) on the GPU nodes as soon as they join the cluster. The inference engine image stays on disk; new pods skip the 30–90 s of pull. Cost: around 6 GB of disk per node.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: vllm-image-warmer }
spec:
 selector: { matchLabels: { app: vllm-warmer } }
 template:
 metadata: { labels: { app: vllm-warmer } }
 spec:
 nodeSelector: { workload: gpu }
 initContainers:
 - name: pull
 image: vllm/vllm-openai:v0.10.0
 command: ["/bin/true"]
 containers:
 - name: pause
 image: registry.k8s.io/pause:3.10

Lever 2, model on a regional PV. The model download (140 GB BF16 or 35 GB FP8) from central object storage is the dominant component of the cold start. Caching the model on a zone or rack PV, Rook-Ceph RBD or local NVMe provisioned by the operator, cuts 60–300 s down to 5–15 s. The antipattern: downloading the model on every startup from external S3.

volumeMounts:
 - name: model-cache
 mountPath: /models
 readOnly: true
volumes:
 - name: model-cache
 persistentVolumeClaim:
 claimName: llama70b-fp8-pvc  # RWX shared, filled offline

Lever 3, warm pool. minReplicaCount > 0 keeps pre-loaded replicas idle. The cost is idle GPU; the benefit is 0 s of cold start for the first client of a peak. For production clusters with continuous traffic: a warm pool of 2–3 replicas. For nightly batch clusters with 0 traffic: warm pool 0 and accept the cold start, or KEDA with a cron that switches on 10 minutes earlier.

Lever 4, predictive scaling with cron. When the pattern is predictable (offices 9–18 h):

triggers:
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5" # 8:30 Monday–Friday
 end: "0 19 * * 1-5" # 19:00
 desiredReplicas: "6"

Combined with reactive triggers. The HPA scales according to the maximum of the signals: if the cron asks for 6 and the queue asks for 10, the result is 10.

Lever 5, parallel download and an efficient format. For PVs that are not pre-loaded, tools such as nvidia-modelmanager, s5cmd or aria2c parallelise the model download. Going from serial download (~150 MB/s) to parallel with 8 threads (~1.2 GB/s) divides the time by 8. And formats such as safetensors load into HBM faster than the original PyTorch pickle.

When to scale nodes, not just pods

The HPA scales pods. If the cluster has no free GPU nodes, the new pod stays Pending for lack of resources. To scale nodes, you need cluster-autoscaler with a specific GPU nodepool, labelled:

# nodepool config (Karpenter or cluster-autoscaler equivalent)
labels:
 workload: gpu
 gpu-model: h100-sxm-80gb
taints:
 - key: nvidia.com/gpu
 effect: NoSchedule
limits:
 min: 2 nodes
 max: 8 nodes

Without this, the HPA can ask for 10 replicas but the cluster only delivers those that fit on nodes already up. The cold start of a new node (bare metal or cloud provisioning, PXE, OS boot, NVIDIA drivers, cluster join) is much longer than the cold start of a pod: typically 5–15 minutes on preconfigured bare metal, 30–60 minutes on real provisioning. For on-premise clusters, the nodepool must be always sized to the expected maximum, and the “scaling” is only on the pod side. The concept of reactive node scale-out only applies to clouds; on-premise you have to buy for the peak.

Three pitfalls specific to LLM scale-down

Pitfall 1, cutting streaming SSE connections. When a replica enters Terminating, Kubernetes sends SIGTERM to the pod and, by default, kills it 30 seconds later. For vLLM that means cutting streaming SSE connections halfway through the response. The client gets a 502 error with the partial output lost. Solution: terminationGracePeriodSeconds: 120 plus a preStop hook that tells the engine not to accept new requests but to finish those in flight:

spec:
 terminationGracePeriodSeconds: 120
 containers:
 - name: vllm
 lifecycle:
 preStop:
 httpGet:
 path: /shutdown
 port: 8000

This requires the engine to expose a graceful shutdown endpoint; vLLM v1 supports it via --enable-graceful-shutdown. Without it, scale-down breaks the SLO even though the metrics do not capture it (cut requests never enter the TTFT histogram).

Pitfall 2, scale-up/scale-down oscillation. If the scale-down stabilizationWindowSeconds is short (~60 s default), the next drop in the queue triggers scale-down, and two minutes later the next peak triggers scale-up. The system oscillates, pays repeated cold starts, and never reaches a stable regime. Solution: scale-down with a window of 10 minutes minimum and conservative policies (type: Pods, value: 1, periodSeconds: 120, at most one replica fewer every 2 minutes).

Pitfall 3, vllm:num_requests_waiting with avg when there is rebalancing. If two replicas are unbalanced (one with queue 20, the other with queue 0), avg gives 10 and the HPA triggers scale-out when the right move would be to rebalance via the load balancer. To detect it: add an alert on stddev(vllm:num_requests_waiting) per deployment. If the dispersion is high, the problem is not capacity but routing.

Full example manifest

For a vLLM deployment with Llama 70B FP8 on 4×H100 SXM per replica, KEDA with warm pool 2:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama70b
 namespace: inference
spec:
 replicas: 2 # managed by KEDA afterwards
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata:
 labels: { app: vllm-llama70b, deployment: vllm-llama70b }
 spec:
 terminationGracePeriodSeconds: 120
 nodeSelector: { workload: gpu, gpu-model: h100-sxm-80gb }
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args:
 - --model=/models/llama-3.3-70b-fp8
 - --tensor-parallel-size=4
 - --max-num-seqs=64
 - --enable-prefix-caching
 - --enable-graceful-shutdown
 ports:
 - { name: http, containerPort: 8000 }
 - { name: metrics, containerPort: 8000 }
 resources:
 limits:
 nvidia.com/gpu: "4"
 memory: 200Gi
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 60
 periodSeconds: 10
 failureThreshold: 30 # tolerates the warmup
 lifecycle:
 preStop:
 httpGet: { path: /shutdown, port: 8000 }
 volumeMounts:
 - { name: model-cache, mountPath: /models, readOnly: true }
 - { name: dshm, mountPath: /dev/shm }
 volumes:
 - name: model-cache
 persistentVolumeClaim: { claimName: llama70b-fp8-pvc }
 - name: dshm
 emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: vllm-llama70b-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: vllm-llama70b } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: vllm-llama70b-scaler, namespace: inference }
spec:
 scaleTargetRef: { name: vllm-llama70b }
 minReplicaCount: 2
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600
 policies:
 - { type: Pods, value: 1, periodSeconds: 120 }
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - { type: Pods, value: 2, periodSeconds: 60 }
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue
 threshold: "5"
 query: avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv
 threshold: "0.85"
 query: avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5"
 end: "0 19 * * 1-5"
 desiredReplicas: "6"

This set is the minimum viable one for LLM autoscaling on a generic cluster with the NVIDIA GPU Operator. Each team adapts it to its own SLO.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB per node, 4 GPU nodes:

Each node hosts one vLLM replica with TP=4 and Llama 70B FP8 (one model per node, they are not shared).
Warm pool of 2 replicas off-peak; the KEDA cron raises it to 4 during working hours.
Cluster-autoscaler does not apply (4 physical nodes bought; scaling is pods only). The number of concurrent replicas is at most the number of available nodes (if each replica uses all 4 GPUs of the whole node).
If the sizing requires more simultaneous replicas than nodes, there are two routes: (a) lower the TP of each replica so two fit per node, (b) expand the physical nodepool. The decision is dictated by capacity planning, see Capacity planning for on-premise LLM inference.

KEDA event volume: around 5 evaluations/min per ScaledObject. For 10 models served in parallel, 3,000 evaluations/h. Manageable with one KEDA operator per cluster.

What we have not covered (upcoming articles)

Cluster-autoscaler for on-premise GPU nodes: how to orchestrate bare metal provisioning (Tinkerbell, Metal³) as a function of demand.
Multi-cluster autoscaling: scaling across clusters in different DCs for geographic resilience.
Cost-aware autoscaling: prioritising nodes by hourly energy cost (on clusters with indexed tariffs).
Predictive ML-based scaling: instead of a static cron, training a model that predicts demand 30 minutes ahead.
Multi-tenant quotas and fairness: KEDA with namespace quotas so that one tenant does not monopolise the HPA.

References

KEDA project — keda.sh (documentación oficial de triggers Prometheus y cron).
Kubernetes — Horizontal Pod Autoscaler walkthrough (kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale).
NVIDIA — GPU Operator on Kubernetes (Helm chart oficial con DaemonSet de drivers y DCGM).
vLLM project — production_monitoring/ (métricas Prometheus expuestas por el servidor).
Karpenter — NodePool spec (etiquetado y taints para nodepools GPU).
Cluster Autoscaler — Scaling GPU nodes (caveats de descubrimiento de recursos GPU).
Kubernetes — Pod lifecycle and termination (preStop, terminationGracePeriodSeconds).

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

Mon, 01 Jun 2026 15:30:00 +0200

This post complements those on LLM tracing with OpenTelemetry GenAI (the tracing layer above the metrics), Capacity planning (what was sized and what must be watched) and Continuous batching (the mechanism that explains several of the engine metrics).

TL;DR

Observability for an LLM inference cluster is built on two complementary sources: the GPU hardware metrics exposed by the DCGM (Data Center GPU Manager) Exporter, part of the NVIDIA GPU Operator, and the metrics from the inference engine (vLLM, SGLang, TensorRT-LLM) exposed on a Prometheus-compatible /metrics. Neither is enough on its own. The classic nvidia-smi metric called GPU utilization is misleading for LLMs: it reads high whenever any kernel is running, without distinguishing tensor cores burning from SMs waiting on HBM. The full cockpit has twelve DCGM metrics in four families (compute: DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE; memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL; thermal-power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS; health: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_RETIRED_DBE) and five vLLM engine metrics (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds). Each one has a defensible green/amber/red threshold, a PromQL query for alerting, and at least one common false reading that confuses the junior operator. The six critical alerts that any production cluster must fire are: HBM > 92 %, thermal or power throttle, XID error, ECC double-bit, KV cache pool > 95 %, and TTFT P95 outside SLO for 5 minutes. The point of having this panel: that the operator on shift diagnoses the origin of a degradation in under five minutes, without opening an SSH console to the GPUs. When that holds, the cluster has moved to professional operation; until then, it is operated on intuition.

You are here: OBSERVE (the other half of tracing)

Tracing, already covered in LLM tracing with OpenTelemetry GenAI, answers what happened in this specific request. Metrics answer what is happening across the cluster in aggregate. They are complementary: an alert on the metrics side tells you “the cluster is degrading”, tracing tells you “and here is the specific trace that proves it”. A cluster without tracing but with metrics operates; a cluster without metrics but with tracing does not operate, it debugs.

The analogy: the cockpit of a modern aircraft

In a modern commercial aircraft, the pilot’s instrument panel has more than 70 active indicators. If there were only one, the altimeter, say, the aircraft would fly into the ground at the first moment of low visibility. You need the altimeter and the attitude indicator, and the airspeed indicator, and the turn indicator, and the fuel gauge, and the oil pressure gauges for each engine, and the turbine exit temperatures. Each answers a different question. And together they cover the operational question: is the aircraft healthy, is it where it should be, and is it going where we want?

Observability for an LLM inference cluster works the same way. A single metric, “GPU utilization 99 %”, answers nothing useful. It is like looking only at the car’s odometer to diagnose why the engine is making a noise. The full cockpit is twelve instruments on the hardware side plus five on the inference engine side, organised into families that answer different questions:

Compute and efficiency: are the tensor cores doing the work we expect, or are they waiting?
Memory: is there VRAM left for new requests, or are we on the edge of OOM?
Thermal and power: is the hardware healthy, or is it silently limiting throughput?
Health and errors: is there hardware degradation under way (ECC, XID, NVLink)?
Inference engine: is the queue growing, is the KV pool saturated, is the SLO being met?

The first four answer “is the GPU fine?”. The fifth answers “is it delivering the service we promised?”. The two questions are different and both must have an answer at a glance.

Why `nvidia-smi` `GPU-Util` misleads with LLMs

The classic metric that appears in nvidia-smi as GPU-Util corresponds to DCGM_FI_DEV_GPU_UTIL. Its official definition: “percentage of time during which one or more kernels were executing on the GPU”. The problem with LLMs: the decode phase is memory-bound, not compute-bound. When the inference engine decodes token by token, the GPU spends 90 % of the time waiting for the HBM to finish delivering the model weights and the KV cache. There is a kernel running (the HBM read), so GPU-Util reports values close to 100 %. But the tensor cores are idle: the bottleneck is memory, not compute.

Practical result: the operator sees “GPU-Util 99 %” in Grafana and assumes “GPU saturated, no more load can go in”. But the reality may be “compute at 25 %, HBM saturated at 95 %”, which changes the operational decisions (quantisation, batch size, parallelism). The classic metric lies by oversimplifying.

The right move is to look at the three DCGM profiling metrics of the _FI_PROF_* subsystem:

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps over the maximum per SM. Is there parallel work?
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores effectively active. Is compute working?
DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring. Is memory saturated?

A typical decode-bound GPU running Llama 70B on an H100 shows: SM occupancy 35–55 %, tensor active 15–30 %, DRAM active 80–95 %. That is the real “saturated GPU” for LLMs. The three together tell the regimes apart; none of them alone says anything actionable.

How they are wired up in production

The platform side is covered in Five maturity levels (level 4, the GPU plane) and Seven deployment phases (phase F5). For the observer, the key pieces are:

NVIDIA GPU Operator. Helm manifests that deploy on each GPU node: drivers, container toolkit, MIG manager and DCGM Exporter. The last one exposes /metrics in Prometheus format with all the DCGM_FI_* listed above. It is scraped from the cluster’s internal Prometheus.

Inference engine. vLLM exposes /metrics on port 8000 (default) with vllm:* metrics. SGLang exposes it too, with the sglang: prefix. TensorRT-LLM exposes it through Triton Inference Server with the nv_inference: prefix. The basic naming convention is similar across the three engines; the thresholds and queries in this post assume vLLM, but they translate.

ServiceMonitor / PodMonitor. The Prometheus operator resource that says what to scrape. Minimal example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: vllm-inference
spec:
 selector:
 matchLabels: { app: vllm }
 podMetricsEndpoints:
 - port: metrics
 interval: 15s

Dashboards. The NVIDIA operator publishes reference Grafana dashboards for DCGM in nvidia/dcgm-exporter (the official repo). vLLM publishes one in vllm-project/vllm (the examples/ folder). Both work as a base; each team adds the panels specific to its own SLO.

The twelve DCGM metrics organised by family

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps per SM over the maximum possible. Value between 0 and 1.

Green: 0.30–0.70 (typical LLM decode regime).
Amber: < 0.20 sustained (batch too small, GPU under-used in parallelism).
Red: 0.95 sustained with low DRAM_ACTIVE (pathological kernel saturating the SMs).

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores executing. The key metric for “is compute producing?”.

Green in prefill: 50–80 %.
Green in decode: 15–30 % (decode is memory-bound, this is not a symptom of a problem).
Red: < 5 % sustained in prefill, or the engine is not using the tensor cores at all (bad config, incompatible format).

DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring data. The key metric for detecting memory saturation.

Green in decode: 60–85 %.
Amber: > 90 % sustained (HBM is a firm bottleneck, which explains high TPOT).
Red: > 95 % sustained with KV cache pool < 70 % (something other than the engine is asking for HBM; investigate leaks).

Family 2 — Memory

DCGM_FI_DEV_FB_USED: Frame Buffer (HBM) used, in MiB.

Green: 70–85 % of total.
Amber: 86–92 %.
Red: > 92 % (risk of OOM on the next paged-attention allocation).

PromQL for the cluster-wide percentage: 100 * sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_TOTAL).

DCGM_FI_DEV_FB_FREE: free Frame Buffer. Complementary to the previous one; useful for absolute alerts (< 4096 MiB free).

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: aggregate NVLink bandwidth in MB/s. For TP (tensor parallel) topologies that cross GPUs over NVLink, this metric reveals whether the parallelism split is saturating the bus.

Green: varies with topology. On 4×H100 SXM with NVLink 4.0, theoretical capacity is 450 GB/s per GPU. A typical TP=4 regime: 50–150 GB/s sustained.
Red: > 90 % of capacity sustained (check whether the model would fit with lower TP or with pipeline parallel).

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP: die temperature in °C.

Green: < 75 °C.
Amber: 75–82 °C.
Red: > 83 °C (close to the H100’s automatic thermal throttle; check ventilation, air flow, rack inlet temperature).

DCGM_FI_DEV_POWER_USAGE: draw in watts. For the H100 SXM, nominal TDP is 700 W. Useful for three things: spotting an unusually low workload (suspect idle or a stall), billing real energy cost, and firing alerts if the draw approaches the PDU limit.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: encoded bitmap with the active throttle reasons. This is the metric that silently explains TPOT degradations.

Relevant bits:

0x0000000000000001: Idle (not a problem).
0x0000000000000002: App clocks setting.
0x0000000000000004: SW Power Cap (software limit, e.g. from nvidia-smi -pl).
0x0000000000000008: HW Slowdown.
0x0000000000000010: Sync Boost (NVIDIA Sync).
0x0000000000000020: SW Thermal Slowdown (software thermal limit).
0x0000000000000040: HW Thermal Slowdown (hardware thermal limit, an emergency).
0x0000000000000080: HW Power Brake Slowdown (PSU voltage drop).
0x0000000000000100: Display Clock Setting.

Any throttle other than Idle with a sustained value > 0 is an alert. TPOT degradation with DRAM_ACTIVE already high and thermal throttle active is the classic “the rack is badly ventilated, it is not the engine’s fault”.

Family 4 — Health

DCGM_FI_DEV_XID_ERRORS: cumulative counter of driver XID errors. XIDs are critical event codes that NVIDIA documents exhaustively (XID 13: graphics engine exception; XID 31: GPU memory page fault; XID 43: reset channel verif error; XID 79: GPU has fallen off the bus; XID 95: uncontained ECC error; and so on). Any increment is an immediate alert: many XIDs require a node reset or an RMA of the GPU.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC errors (uncorrectable). Unlike single-bit ones (which ECC corrects silently and which are counted in DCGM_FI_DEV_ECC_SBE_*), double-bit errors corrupt data. Any value > 0 is a critical alert: the GPU must be drained and inspected.

DCGM_FI_DEV_RETIRED_DBE: physical HBM pages retired because of accumulated double-bit errors. NVIDIA retires faulty pages automatically to prevent future corruption. More than 4–8 retired pages on one GPU suggests silicon degradation: document it and plan a replacement in the next maintenance window.

The five inference engine metrics (vLLM)

The DCGM metrics answer “is the GPU healthy?”. The engine metrics answer “is the service meeting the SLO?”. Without them, you know the hardware works but you do not know whether the clients are happy.

vllm:num_requests_running: requests currently in the batch. If it reaches the configured --max-num-seqs and does not come down, the engine is saturated on concurrency (check VRAM and rebalance through the autoscaler, see Autoscaling LLM on Kubernetes).

vllm:num_requests_waiting: requests queued, not yet in the batch. Any value > 0 sustained over minutes means the cluster is not scaling with the load. This is the primary metric for HPA.

vllm:gpu_cache_usage_perc: % of the KV cache pool in use.

Green: 50–80 %.
Amber: 80–92 %.
Red: > 92 % (risk of preempt-on-OOM: vLLM will drop requests to free memory, which raises TTFT visibly).

vllm:time_to_first_token_seconds: histogram of TTFT per request. Consumed as histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket[5m]))). Compared against the TTFT P95 SLO, it fires the primary service alert.

vllm:time_per_output_token_seconds: histogram of TPOT. Equivalent to the previous one but for streaming smoothness. Compared against the TPOT P95 SLO, it fires the secondary alert.

The six alerts that must page in production

Any serious production cluster fires these six alerts to a channel with an on-call rotation. Without them, the SLO is met by luck, not by process.

groups:
 - name: gpu-llm-critical
 rules:
 - alert: GpuHbmNearOom
 expr: 100 * (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 92
 for: 2m
 labels: { severity: critical }
 annotations:
 summary: "HBM on {{ $labels.gpu }} at {{ $value }}% — OOM risk"

 - alert: GpuThermalOrPowerThrottle
 expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0) and ignoring(reason) (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 1)
 for: 1m
 labels: { severity: warning }
 annotations:
 summary: "GPU {{ $labels.gpu }} throttling (reasons={{ $value }})"

 - alert: GpuXidErrorDetected
 expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
 labels: { severity: critical }
 annotations:
 summary: "XID error on GPU {{ $labels.gpu }} — investigate immediately"

 - alert: GpuEccDoubleBit
 expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
 labels: { severity: critical }
 annotations:
 summary: "ECC double-bit on GPU {{ $labels.gpu }} — drain node"

 - alert: VllmKvCachePoolNearFull
 expr: vllm:gpu_cache_usage_perc > 0.95
 for: 3m
 labels: { severity: warning }
 annotations:
 summary: "KV cache pool > 95% on {{ $labels.instance }}"

 - alert: VllmTtftP95OutOfSlo
 expr: histogram_quantile(0.95, sum by(le, instance)(rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.5
 for: 5m
 labels: { severity: warning }
 annotations:
 summary: "TTFT P95 over SLO ({{ $value }}s > 1.5s)"

These six cover 80 % of the incidents that affect the SLO. The remaining 20 % demand investigation with tracing (see LLM tracing with OpenTelemetry GenAI).

Master table: thresholds and queries

Metric	Green	Amber	Red	Base query (PromQL)
SM occupancy	0.30–0.70	0.15–0.30	< 0.10 sustained	`DCGM_FI_PROF_SM_OCCUPANCY`
Tensor active (decode)	15–30 %	< 10 %	< 3 %	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
DRAM active	60–85 %	85–95 %	> 95 % with low KV	`DCGM_FI_PROF_DRAM_ACTIVE`
FB used	70–85 %	86–92 %	> 92 %	`100 * DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL`
NVLink BW	< 70 % cap	70–90 % cap	> 90 % cap	`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`
GPU temp	< 75 °C	75–82 °C	> 83 °C	`DCGM_FI_DEV_GPU_TEMP`
Power usage	< 90 % TDP	90–98 % TDP	> 98 % TDP	`DCGM_FI_DEV_POWER_USAGE`
Throttle reasons	0 or Idle	App/SW	HW Therm/Power	`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`
XID errors	no change	—	any delta	`increase(DCGM_FI_DEV_XID_ERRORS[5m])`
ECC DBE	0	—	> 0	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
Retired pages	< 4	4–8	> 8	`DCGM_FI_DEV_RETIRED_DBE`
KV cache used	50–80 %	80–92 %	> 92 %	`vllm:gpu_cache_usage_perc`
Requests waiting	0	1–5 sustained	> 10 sustained	`vllm:num_requests_waiting`
TTFT P95	< SLO	80–100 % SLO	> SLO	see alert query above
TPOT P95	< SLO	80–100 % SLO	> SLO	`histogram_quantile(0.95, sum by(le)(rate(vllm:time_per_output_token_seconds_bucket[5m])))`

Three pitfalls that confuse the junior operator

Pitfall 1 — “GPU-Util at 99 % = saturated”. As explained at the start, DCGM_FI_DEV_GPU_UTIL lights up with any kernel. The right move is to look at the three _PROF_* metrics (SM occupancy, tensor active, DRAM active) together. GPU util 99 % + tensor active 8 % + DRAM active 92 % means “saturated by memory, not compute”; GPU util 99 % + tensor active 75 % + DRAM active 50 % means “saturated by compute, prefill heavy”. The two situations call for different levers.

Pitfall 2 — confusing single-bit ECC (SBE) with double-bit (DBE). Single-bit errors are corrected silently and are unavoidable in any HBM under load (cosmic radiation, voltage fluctuations). An SBE counter growing slowly is not an alert, it is physics. DBE is: it corrupts data. Telling the two metrics apart avoids false alarms and false negatives in equal measure.

Pitfall 3 — alerting on num_requests_waiting > 0 without context. An instantaneous value of 1 or 2 during a spike is normal. What matters is a sustained queue: use for: 5m with a threshold of 3–5. Without that window, the system floods the alert channel with noise.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter deployed through the NVIDIA GPU Operator, one DaemonSet per GPU node.
Internal Prometheus with 30-day retention for high-frequency metrics, 1 year for downsampled ones (Thanos/Mimir if the volume justifies it).
Grafana with three standard dashboards: GPU hardware (DCGM), engine (vLLM), SLO (TTFT/TPOT/RPS against written targets).
Alertmanager with an on-call rotation and rate limiting through silences grouped by node.
Cardinality kept under control: gpu (local id), node, pod, model. Do not add request_id or other high-cardinality labels to metrics; that is tracing’s job.

Estimated volume for a 16-GPU cluster scraped every 15 s: around 2 million samples/min, around 25 GB/day of raw Prometheus. Manageable with one Prometheus per cluster plus retention; if the team scales beyond 64 GPUs, consider a Thanos sidecar or VictoriaMetrics. See OSS LLMOps tooling catalogue for equivalent alternatives.

What we have not covered (upcoming articles)

Tracing of LLM workloads: already covered in LLM tracing with OpenTelemetry GenAI.
Autoscaling based on these metrics: see Autoscaling LLM on Kubernetes.
Incident response runbooks: how each of these alerts translates into concrete action (drain, restart, RMA, scaling, rollback).
Cost accounting: using DCGM_FI_DEV_POWER_USAGE and vllm:request_success_total for per-tenant cost showback.
Multi-tenant fairness monitoring: when several tenants share a cluster, which metrics detect that one of them is hogging the KV cache.

References

NVIDIA — DCGM Exporter (repo nvidia/dcgm-exporter, documented metrics and units).
NVIDIA — DCGM Field Identifiers reference (complete list of DCGM_FI_*).
NVIDIA — XID Errors documentation (catalogue of XID codes and remediation procedures).
NVIDIA — NVIDIA GPU Operator (official Helm chart).
vLLM project — examples/production_monitoring/ (reference PromQL and Grafana dashboards).
Prometheus — Histogram and summary best practices (for building defensible percentile queries).
NVIDIA — H100 Tensor Core GPU datasheet (TDP, HBM bandwidth, NVLink capacities).

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

Operations on lo0 — Tech Blog

Langfuse v4, day 2 (5 of 10): the worker's thirty-nine queues, and the three that decide whether you lose data

TL;DR

You are here: OBSERVE, day 2

The analogy: the sorting hub

The map of the thirty-nine

The sharding that does not shard

Concurrencies, and the ones set to one on purpose

An event’s path, from API to ClickHouse

Retries, and the bin nobody looks at

The metrics, with their exact names

The two probes, and the one that can restart you in a loop

The crons

Splitting roles, which is what all this is for

Checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

TL;DR

You are here: OBSERVE, day 2

The analogy: changing the track gauge without closing the line

The three-rung ladder

The three combinations that refuse to start

What breaks in each mode

The three steps, in order

The ClickHouse migrations backfill nothing

The five-step chain that does backfill history

The failure that does not block

Where the ten minutes come from

The forty-eight-hour TTL

What can and cannot be undone

How this looks in code

Migration checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (3 of 10): two well-known agents instrumented, Open Deep Research and GPT Researcher

TL;DR

You are here: OBSERVE, day 2

The analogy: two factories, one audit

The two agents, and why these

Open Deep Research: the shape of the graph

What one request costs, measured

Zero AGENT observations

Instrumenting it without touching the repository

GPT Researcher: two products in one repository

The myth of the fresh config

The standard mode: eight calls, eight traces

The human node that stays open

What to look at in the trace once it arrives

Checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

TL;DR

You are here: OBSERVE, day 2

The analogy: the file that gets photocopied whole at every signature

Which agentic platform is the most used

How LangGraph gets into Langfuse

The observation type is decided by the node name

What a turn costs, measured

What this means for ClickHouse

Interrupts: two traces where there should be one

Thread, session, user, and what is only read at the root

The filter that drops your own spans

The model and the cost with a gateway in the middle

How this looks in code

Agent instrumentation checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (1 of 10): the data model changed underneath, and what really goes into a trace

TL;DR

You are here: OBSERVE, day 2

The `litellm_jwtauth` fields that decide security