Agents on lo0 — Tech Blog

Langfuse v4, day 2 (3 of 10): two well-known agents instrumented, Open Deep Research and GPT Researcher

Mon, 14 Sep 2026 07:00:00 +0200

Third article in a series of ten about operating Langfuse v4 in production. The first walked through what goes into a trace; the second measured what an agent turn costs over a minimal graph. This one applies the same to two agents people deploy. Code read on 14 September 2026: Open Deep Research at commit 1b7d2e8 and GPT Researcher at 6f99857, with Langfuse 4.15.2 and LangGraph 1.2.11.

TL;DR

One Open Deep Research request with the default configuration is 1,416 observations and 8.1 MB. Measured over a faithful replica of its topology with five concurrent units, six supervisor iterations and ten react calls per researcher, which are its defaults. It all lands in a single trace. And it is a floor, because on the bench the search results are three hundred characters and real ones are much larger.

None of those 1,416 is of type AGENT. Its nine nodes are called clarify_with_user, write_research_brief, research_supervisor, final_report_generation, supervisor, supervisor_tools, researcher, researcher_tools and compress_research. None contains the string the handler looks for.

The parallel fan-out does not break the trace, and the fear that it does is unfounded. Open Deep Research forwards the config to its subgraphs and GPT Researcher passes a fresh one carrying only tags. I tested both patterns: one trace in both cases, because LangChain rebuilds the config from the context variable.

GPT Researcher is two products in one repository. The multi-agent mode is LangGraph and traces like any graph. The standard mode has no graph, and there every model call opens its own trace: eight calls, eight traces, measured.

Its human node does not use interrupt(), it waits on a websocket. The node’s observation stays open while the person thinks, so its duration measures human latency. The 95th percentile of that graph means nothing unless that node is excluded.

Instrumenting both requires no change to their repositories. For Open Deep Research it is enough to import the compiled graph and pass the handler through config. For GPT Researcher’s standard mode you have to wrap with the SDK, because there is nowhere to hook callbacks.

You are here: OBSERVE, day 2

The previous article produced a formula over a toy graph: 4 + 5n observations per turn, with the byte volume growing with the square of the steps. A formula over a toy is good for understanding the mechanism and no good for sizing anything.

This article applies it to two real agents, chosen because they are well known and because they break in different ways. The series grows to ten articles and the migration one moves along one place.

The analogy: two factories, one audit

A traceability audit in a factory that runs the whole process on its own lines is tedious but complete: every station leaves a record, and the auditor reconstructs the part from beginning to end. The volume of paper is enormous and that is the only problem.

The second factory subcontracts half the process. The auditor sees material going into a building and finished product coming out, with times and quantities, and has nothing about what happens inside. There is little paper. The problem is not the paper either.

Open Deep Research is the first factory and the problem is volume. GPT Researcher, in its standard mode, is the second, and the problem is that the trace does not carry what you need to know. Both are fixable, with different tools, and confusing them leads to buying disk when what is missing is instrumentation.

The two agents, and why these

Open Deep Research (langchain-ai/open_deep_research, 12.7 k stars) is LangChain’s reference research agent. All its work goes through LangChain, so callback instrumentation sees it whole.

GPT Researcher (assafelovic/gpt-researcher, 28.4 k stars) is the best-known open source research agent and it is not from LangChain. Its multi-agent mode uses LangGraph; its standard mode, which is what the API and the web interface run, does not.

One is in-house and the other is not, which for a comparison is what matters.

Open Deep Research: the shape of the graph

Three graphs, two of them nested as subgraphs (src/open_deep_research/deep_researcher.py):

The root graph, deep_researcher (line 701), with four nodes: clarify_with_user, write_research_brief, research_supervisor and final_report_generation.
The supervisor subgraph (line 353), with supervisor and supervisor_tools, iterating between them.
The researcher subgraph (line 589), with researcher, researcher_tools and compress_research, which is a classic react loop.

The multiplication lives in supervisor_tools. When the supervisor decides to delegate, that node launches one researcher subgraph per unit and awaits them together (line 305):

research_tasks = [
 researcher_subgraph.ainvoke({
 "researcher_messages": [HumanMessage(content=tool_call["args"]["research_topic"])],
 "research_topic": tool_call["args"]["research_topic"]
 }, config)
 for tool_call in allowed_conduct_research_calls
]

tool_results = await asyncio.gather(*research_tasks)

The detail that decides whether the trace comes out whole is that config at the end: the node receives LangGraph’s configuration and forwards it to the subgraphs, so the callbacks travel and every researcher hangs off the right place.

The three ceilings in the default configuration (src/open_deep_research/configuration.py) are max_concurrent_research_units = 5, max_researcher_iterations = 6 and max_react_tool_calls = 10. They multiply each other.

What one request costs, measured

I replicated that topology with fake models, the same technique as the previous article: three graphs, the same nesting, the same fan-out with asyncio.gather, three-hundred-character search results and two-hundred-character model replies. No network and no GPU, so it reproduces anywhere.

Units	Iterations	React calls	Observations	generation	chain	Attribute KB	Largest observation
1	1	1	26	8	18	56.8 KB	3.7 KB
2	2	3	92	26	66	257.3 KB	4.6 KB
5	6	10	1,416	370	1,046	8,108 KB	11.0 KB

The last row is the repository’s default configuration. One research request, one person pressing a button once, leaves 1,416 observations and eight megabytes in the observability backend. In a single trace, under a single identifier.

Those eight megabytes are a floor, not a ceiling. On the bench each search result takes three hundred characters; a real Tavily result carrying page content takes between ten and a hundred times more, and that text enters the history and gets re-serialised in every later observation of the same researcher, which is the previous article’s quadratic growth acting on much larger text.

Put into capacity arithmetic: a hundred research requests a day, which for a small team is nothing, is 141,600 observations and on the order of a gigabyte a day uncompressed, with toy searches. With real searches, tens of gigabytes. It is another order of magnitude compared with a chat assistant, and it is the reason this article comes before the capacity one.

Zero AGENT observations

Among those 1,416 observations there is not one of type AGENT. The previous article explained why: the handler decides that type by looking for the string agent in the class path or the run name. Open Deep Research’s nine nodes are clarify_with_user, write_research_brief, research_supervisor, final_report_generation, supervisor, supervisor_tools, researcher, researcher_tools and compress_research.

They are good names. They describe what each node does and they leave out a word that adds nothing for whoever reads the code. The result is that LangChain’s reference research agent, seen from Langfuse, is a tree of chains.

It is fixable from outside, without touching the repository, by naming the subgraph invocation or renaming the two nodes that represent a model decision, which are supervisor and researcher. Without that, any dashboard grouping by observation type will not tell decision work apart from plumbing.

Instrumenting it without touching the repository

Open Deep Research ships as a LangGraph Server graph, with its langgraph.json pointing at deep_researcher. There are two ways to get the handler in, depending on how you run it.

If you invoke it from your own code, the compiled graph is importable and the config is enough:

from langfuse import Langfuse, propagate_attributes
from langfuse.langchain import CallbackHandler
from open_deep_research.deep_researcher import deep_researcher

trace_id = Langfuse.create_trace_id(seed=f"{request_id}")
handler = CallbackHandler(trace_context={"trace_id": trace_id})

with propagate_attributes(
 trace_name="deep-research",
 user_id=user,
 session_id=request_id,
 tags=["odr", "research"],
):
 result = await deep_researcher.ainvoke(
 {"messages": [HumanMessage(question)]},
 config={
 "callbacks": [handler],
 "configurable": {
 "thread_id": request_id,
 "max_concurrent_research_units": 3,
 "max_researcher_iterations": 4,
 },
 },
 )

Lowering those two ceilings from 5 and 6 to 3 and 4 is not an observability recommendation, it is a product decision that also divides the trace volume by three. Better taken at the same time.

If you deploy it on LangGraph Server there is no invocation of yours to put the config into, so the handler gets bound at compile time. The pattern is to wrap the exported graph:

# instrumented_graph.py, and point langgraph.json here
from langfuse.langchain import CallbackHandler
from open_deep_research.deep_researcher import deep_researcher

graph = deep_researcher.with_config({"callbacks": [CallbackHandler()]})

That route loses the per-request seeded trace identifier, because the handler is created once. If the agent uses interrupts, that loss matters, for the reason the previous article measured.

GPT Researcher: two products in one repository

Here is the interesting part, and it is a trap you fall into with any agent that has grown in layers.

The repository holds the gpt_researcher package, which is the engine, and the multi_agents package, which is an orchestration on top. Only the second uses LangGraph.

The orchestrator (multi_agents/agents/orchestrator.py:66) has eight nodes: browser, planner, human, researcher, writer, fact_checker, visualizer and publisher, with two conditional loops, one for human review of the plan and one for fact checking. The researcher node launches in parallel one editor subgraph per section (multi_agents/agents/editor.py:134), and that subgraph has its own revision loop between reviewer and reviser.

All of that traces like any graph. And the models are called with self.llm.ainvoke(messages, **kwargs) from the generic provider (gpt_researcher/llm_provider/generic/base.py:365), which is a LangChain chat model with no explicit config, so it picks up the ambient configuration while running inside a node. The generations show up.

The standard mode is another thing. The GPTResearcher class has no graph, no nodes and does not go through LangGraph at any point. It is the mode the API and the web interface use, that is, the one most people run.

The myth of the fresh config

Before going on, something that looks like a bug and is not has to be dismantled, because it shows up in many agents.

When the researcher node launches the section subgraphs, it does it like this (multi_agents/agents/editor.py:74):

final_drafts = [
 chain.ainvoke(self._create_task_input(research_state, query, title),
 config={"tags": ["gpt-researcher"]})
 for query in queries
]
research_results = [result["draft"] for result in await asyncio.gather(*final_drafts)]

That config is fresh and carries only tags. The method does not even receive the node’s configuration, so it could not forward it. The intuitive conclusion is that the subgraphs lose the callbacks and every section ends up in its own trace.

It does not happen. I set both patterns side by side, Open Deep Research’s forwarding the config and GPT Researcher’s passing a fresh one, and both give one trace with one root span. The reason is that LangChain does not take the config you pass as is: it starts from the ambient configuration held in a context variable and overlays the keys you bring. The callbacks are still there, and context variables are copied into the tasks asyncio.gather creates.

What does get lost are the parent’s tags, replaced by the new list. That is a filtering nuisance, not a broken trace.

The standard mode: eight calls, eight traces

With no graph there is no root run, and with no root run every model call is its own tree. Measured with eight standalone calls to a chat model, with the handler passed by hand on each one:


Observations	8
Distinct traces	8
Root spans	8
Types	8 generation

Eight traces of one observation each, with no relation between them. None of them knows it belongs to the same piece of research. And that is the favourable case, because the standard mode offers no public place to put that handler: the **kwargs reach ainvoke from create_chat_completion (gpt_researcher/utils/llm.py:120), but the path crosses several internal layers that are not meant as an extension point.

The route that works without patching the repository is to wrap from outside with the SDK and let the inner calls hang off that:

from langfuse import get_client, observe, propagate_attributes
from gpt_researcher import GPTResearcher

@observe(name="gpt-researcher", as_type="agent")
async def research(question: str, request_id: str, user: str) -> str:
 with propagate_attributes(user_id=user, session_id=request_id, tags=["gptr"]):
 researcher = GPTResearcher(query=question, report_type="research_report")
 await researcher.conduct_research()
 report = await researcher.write_report()

 client = get_client()
 client.update_current_span(
 output={"length": len(report), "sources": len(researcher.get_source_urls())},
 metadata={"gptr_costs": researcher.get_costs()},
 )
 return report

With that there is one trace per piece of research, with an agent observation at the root and the generations hanging inside, in the same tree. Searches and document chunking do not appear, because they do not go through LangChain. If they are needed, they get added with start_as_current_observation(as_type="tool") around your own calls, which is the first of the three ways out of the span filter the previous article described.

Worth noting researcher.get_costs(). GPT Researcher keeps its own cost accounting, and putting it in as metadata is the only thing that lets you reconcile it later with what Langfuse computes. When the two figures disagree, it is almost always the gateway’s model alias.

The human node that stays open

The multi-agent mode has a human node that asks for an opinion on the plan before researching. It does not use LangGraph’s interrupt(). It waits inside the node, reading from the websocket (multi_agents/agents/human.py).

The two approaches have opposite consequences for the trace. With interrupt(), the graph hands control back and the trace splits in two, which is what the previous article measured. Waiting inside the node, the trace is single but that node’s observation stays open for as long as the person takes, minutes or hours.

Neither is wrong. What is wrong is the dashboard that averages durations without knowing which of the two it has in front of it. With the wait inside the node, the 95th percentile of graph duration measures how long a human takes to answer a message, not how long the agent takes. Alert on that metric and the alert fires at lunchtime.

The practical rule is to exclude human-wait nodes by name from any latency metric, and to measure that wait separately, which tends to be an interesting product metric in its own right.

What to look at in the trace once it arrives

With both agents instrumented, and before building any dashboard, there are four checks that are done once and save weeks:

Count the observations of a typical request. If it comes out above a thousand, this platform’s problem is going to be volume and sampling has to be decided before anything else.
Look for more than one root span. More than one means there is work being traced outside the tree, and you want to know which before there are thousands.
Check that generations carry cost. If they come in at zero, it is the model alias, fixed by registering that name with its price.
Sort observations by duration and remove the ones waiting on a person. What stays at the top is where the time really is.

Checklist

The graph is invoked with a trace identifier seeded per request, not a random one.
The agent’s concurrency and iteration ceilings are set deliberately, knowing they multiply trace volume.
Nodes representing a model decision carry agent in the name, or it has been accepted in writing that there will be no observations of that type.
Work that does not go through LangChain is wrapped with the SDK, or it has been decided that it need not be visible.
Human-wait nodes are identified and excluded from latency metrics.
The agent’s own cost accounting, if it has one, goes in as metadata so it can be reconciled.
There is a measured figure of observations per request, taken from your own installation and not from this article.

Traps

A research agent’s defaults are demo values, not production ones. Five units times six iterations times ten react calls multiply, and whoever chose them was thinking about report quality, not about your ClickHouse.

One repository can hold two agents with two different instrumentation stories. Instrumenting the mode the README documents and not the one the API runs is a mistake with no symptom: the traces arrive, just of something else.

Passing a fresh config to a subgraph does not break the trace. It breaks the tags. The tree survives because the callbacks travel through a context variable.

A node waiting on a person contaminates any duration statistic. And not obviously, because the node finishes cleanly and flags no error.

Good node names and good observation classification are conflicting goals. The winner is whoever writes the agent’s code, who is normally not you.

Eight megabytes per request with three-hundred-character searches. With real results, the number your installation produces will not look like this article’s, and it will be larger.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented (this article): Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Open Deep Research source, commit 1b7d2e8, read on 14 September 2026: src/open_deep_research/deep_researcher.py and configuration.py.
GPT Researcher source, commit 6f99857, read on 14 September 2026: multi_agents/agents/orchestrator.py, multi_agents/agents/editor.py, multi_agents/agents/human.py, gpt_researcher/llm_provider/generic/base.py and gpt_researcher/utils/llm.py.
Own measurement of 14 September 2026 over replicas of both topologies, with LangGraph 1.2.11, langchain-core 1.6.3, Langfuse SDK 4.15.2, InMemorySpanExporter and fake models, no network.
GitHub stars for both repositories, checked on 14 September 2026.

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

Sun, 13 Sep 2026 07:00:00 +0200

Second article in a series of ten about operating Langfuse v4 in production. The first one, what really goes into a trace, walked through the data model by reading the server code. This one measures the other side. Verified against Langfuse 4.15.2 (Python SDK), LangGraph 1.2.11 and langchain-core 1.6.3, with figures taken on 13 September 2026 on a bench with no network.

TL;DR

One agent turn is four observations plus five per tool call. Measured with an in-memory exporter over a minimal ReAct graph: zero tools, 4 observations; one, 9; five, 29; ten, 54; forty, 204. The formula holds exactly across the whole range.

The byte volume grows quadratically, not linearly. The same turn goes from 12.2 KB with one tool call to 1.87 MB with forty. 88 % of those bytes is the input field, because every observation re-serialises the full message history as it stands at that moment.

Tool definitions travel attached to every model call. The handler appends them to the input field as messages with role tool. Measured: 364 bytes per tool. With twenty tools that is 7.3 KB on every single step of the loop.

The AGENT observation type is assigned by string match on the name. A node called planificador comes out as CHAIN; the same node called agente_planificador comes out as AGENT. The logic is literally looking for agent in the class path or in the node name (langfuse/langchain/CallbackHandler.py:397).

An interrupt with a resume produces two separate traces. Verified with invoke, with stream and with subgraphs, and also reusing the same handler object. The SDK ships machinery to reattach the trace, but it only fires if the root run finishes with a control-flow exception, and in LangGraph 1.2.11 the interrupt comes back through the return value.

The fix fits in one line and is measured. Passing trace_context={"trace_id": Langfuse.create_trace_id(seed=thread_id)} when building the handler leaves the interrupt and the resume in the same trace, even with different handlers and therefore with different replicas.

The span processor drops by default whatever it does not recognise. It only exports spans from the SDK’s own tracer, spans carrying some gen_ai.* attribute, or spans from a closed list of thirty-five instrumentation prefixes (langfuse/_client/span_filter.py:11). Your own code’s spans do not arrive unless they come through one of those three doors.

You are here: OBSERVE, day 2

The series was planned with eight articles and has grown to ten. The reason is ordering: the next two are about migrating and about sizing ClickHouse, and both need an input figure that nobody publishes, which is how many observations and how many bytes a real unit of work produces. A chat produces a trace with two observations. An agent produces two hundred. Sizing with the chat figure leads to a cluster that falls over the day someone connects the agent.

So before migrating, measure. And to measure there has to be in front what people actually deploy.

The analogy: the file that gets photocopied whole at every signature

A procurement file passes across six desks. At each desk somebody reads what is there, adds a sheet and passes it on.

There are two ways to leave a record of that route. The first is to write in a register which desk touched it, when, and which sheet it added. Six short entries. The second is to photocopy the whole file on the way into each desk and on the way out, and to archive the twelve copies. The register takes one sheet. The copies take, at the sixth desk, six times what they took at the first, and in total they take something on the order of the square of the number of desks.

LangChain’s callback instrumentation does the second thing. It does not record the delta of each step, it records the complete state on the way into and out of each step. With a chat that makes no difference, because the state is short and there is one step. With a forty-step agent, the archive weighs almost two megabytes per file processed.

This is not a defect that needs fixing. It is what lets you open a trace and see exactly what the model saw at step 17 without reconstructing anything. But it drives the sizing, and the number has to be known before signing the contract.

Which agentic platform is the most used

The question has two answers depending on the metric, and both are defensible.

By PyPI downloads over the last thirty days, checked on 13 September 2026:

Package	Downloads / 30 days
`langgraph`	52.3 M
`strands-agents`	35.2 M
`openai-agents`	22.9 M
`crewai`	19.1 M
`google-adk`	14.5 M
`pydantic-ai`	8.0 M
`llama-index`	5.0 M
`agno`	1.9 M
`autogen-agentchat`	0.6 M

By GitHub stars the order flips at the top: CrewAI is at 58.4 k, LangGraph at 38.1 k and the OpenAI Agents SDK at 28.6 k.

PyPI downloads are inflated by continuous integration and by images that reinstall on every build, so they indicate presence in pipelines rather than agents in production. Stars measure interest, not deployment. With both together, LangGraph is the one that shows up in more pipelines and the one with the most worked-through instrumentation path in Langfuse, so it is the one instrumented here. What follows about volume applies equally to any platform instrumented through LangChain callbacks, because the cost comes from the callback model, not from the graph.

How LangGraph gets into Langfuse

The route is worth stating, because the previous article left two open and only one is used here.

langchain-core 1.6.3 does not import OpenTelemetry anywhere. The check is direct against the installed wheel and returns not one file. LangGraph does not emit OTLP spans on its own either. What exists is LangChain’s callback system, and the Langfuse CallbackHandler hooks into it.

That handler does not talk to the classic ingestion API. It creates observations with the v4 SDK, which underneath is OpenTelemetry: LangfuseSpanProcessor sets up an OTLPSpanExporter against {base_url}/api/public/otel/v1/traces (langfuse/_client/span_processor.py:123). So it is OTLP, but generated by the SDK, not by third-party auto-instrumentation.

The difference matters for one concrete reason left hanging in the previous article. Because the data passes through the SDK calls, the classic masking function does apply: input, output and metadata all go through _process_media_and_apply_mask before becoming attributes (langfuse/_client/span.py:534). With auto-instrumentation of the OpenInference or OpenLLMetry kind, the messages travel in span events and masking does not touch them. For an agent handling customer data inside its tools, that difference decides the route.

The trade-off is that a callback handler only sees what goes through LangChain. HTTP calls a tool makes on its own, the database query, the subprocess, none of that shows up. And if you instrument your code with OpenTelemetry so that it does show up, you run into the filter described further down.

The observation type is decided by the node name

The first article insisted that v4 has ten observation types and that using the right one changes what the interface knows how to group. The LangChain handler emits six of those ten: tool, retriever, generation, agent, chain and span. It never emits event, embedding, evaluator or guardrail.

The assignment is direct for four of them: tool callback, type tool; retriever callback, type retriever; model callback, type generation. For chain callbacks, which is what every node of a LangGraph graph is, the criterion is this (langfuse/langchain/CallbackHandler.py:393):

elif callback_type == "chain":
 # Detect if it's an agent by examining class path or name
 if serialized and "id" in serialized:
 class_path = serialized["id"]
 if any("agent" in part.lower() for part in class_path):
 return "agent"

 name = self.get_langchain_run_name(serialized, **kwargs)
 if "agent" in name.lower():
 return "agent"

 return "chain"

Measured over the same graph with the node renamed and everything else identical:

Node name	Tool calls	Observations	Types
`planificador`	10	54	11 generation, 33 chain, 10 tool
`agente_planificador`	10	54	11 generation, 22 chain, 11 agent, 10 tool

Eleven observations change type because of eleven characters in a function name. This has two practical consequences. The first is that any dashboard counting AGENT observations is counting naming conventions. The second is that the interface draws the agent graph when the trace contains any observation of a type other than span, event or generation, and chain qualifies, so the drawing comes out either way. What changes is what you can filter and aggregate afterwards.

The operational recommendation is boring and it works: name the nodes that represent a model decision with a name containing agent, and leave the rest as chains. It is a naming convention promoted to data schema, which is not pretty, but it is what there is and it is stable.

What a turn costs, measured

The setup is a minimal ReAct graph: one node that calls the model, a ToolNode with one tool, a conditional edge with tools_condition and an in-memory checkpointer. The model is langchain-core’s GenericFakeChatModel, which returns a fixed list of messages, so neither network nor GPU is needed and the experiment reproduces on any laptop. The exporter is OpenTelemetry’s InMemorySpanExporter, passed to the client through the span_exporter parameter, so observations are counted with no Langfuse server in front.

Result, with the node named planificador:

Tool calls	Observations	generation	chain	tool	Attribute bytes
0	4	1	3	0	4.2 KB
1	9	2	6	1	12.2 KB
5	29	6	18	5	64.1 KB
10	54	11	33	10	173.5 KB
20	104	21	63	20	541.4 KB
30	154	31	93	30	1,108 KB
40	204	41	123	40	1,874 KB

The observation count is exactly 4 + 5n. The five per loop iteration are: the model node, the model call, the conditional edge, the tools node and the tool. The four fixed ones are the root graph, the first partial pass and the close.

The bytes do not follow that line. From 10 to 20 steps the volume multiplies by 3.1; from 20 to 40, by 3.5. That is the quadratic growth expected when each of the n steps drags along a history whose length is proportional to n.

The breakdown by field confirms it. With forty steps, out of 1,709 KB measured in that run, 1,502 KB sit in langfuse.observation.input and only 56 KB in the output. 88 % of what is ingested is the same history repeated at different stages of progress. The largest single observation in that turn is 22.2 KB, well below the 9.5 MB per OTLP span ceiling, so the problem is not the size limit: it is the aggregate.

And inside the input there is a second multiplier. The handler appends each tool’s definition to the generation’s input field, as messages with role tool (langfuse/langchain/CallbackHandler.py:1190). Measured with three-parameter tools and a one-line description:

Bound tools	Input field bytes of one generation
0	34
1	398
5	1,854
10	3,674
20	7,334

That is 364 bytes per tool, on every generation, always. An agent with twenty tools and forty steps ingests 7.3 KB × 41 generations = 300 KB in tool schemas alone, which never change over the course of the turn.

What this means for ClickHouse

A honest caveat belongs here. I compressed the full payload of those runs with ZSTD level 3, which is what events_full uses, and got ratios between 29× and 86×, improving the longer the turn. That number is inflated by how synthetic the bench is: my tool always returns the same string, so the redundancy is higher than in production.

What does hold is the shape. Repeating the history is exactly the pattern columnar compression absorbs well, so the ClickHouse disk is not where it hurts. Where it hurts is earlier: in the bandwidth towards the OTLP endpoint, in the worker CPU that decompresses and flattens, and in the ingestion queue, all of which see uncompressed bytes. The first article noted that the queue that saturates first is langfuse.queue.ingestion.depth, and this is why it saturates.

Quick sizing arithmetic, with the measured figures and an average ten-step turn:

1,000 turns a day is 54,000 observations and around 170 MB uncompressed a day.
10,000 turns a day is 540,000 observations and 1.7 GB uncompressed a day.
If the average turn goes from ten steps to twenty, observations double and bytes triple.

Sensitivity to the number of steps is the parameter to watch, more than the number of users. A prompt change that makes the agent take three more turns around the loop multiplies the observability bill with nobody touching the infrastructure. Article 6 of the series comes back to this with ClickHouse system tables to measure the real bytes per observation in your own installation, which is the only thing that replaces this estimate.

Interrupts: two traces where there should be one

A production agent with human approval uses interrupt(), hands control back, waits for a person and resumes with Command(resume=...). The operational question is whether the trace survives that pause.

The SDK ships explicit machinery for the case. There is a store of pending trace contexts keyed by thread_id, capped at 1,024 entries with eviction of the oldest (MAX_PENDING_RESUME_TRACE_CONTEXTS), and a function that recognises a Command carrying resume and recovers the stored context.

Measured, it does not fire in this combination of versions:

Case	Same trace?
Same handler object, `invoke`	No
Same handler object, `stream`	No
Interrupt inside a subgraph	No
Fresh handler for the resume	No

The reason is in the code and it is consistent. The pending context is only stored from on_chain_error when the failing run is the root one and the resulting level is DEFAULT, which is what happens with LangGraph’s control-flow exceptions (CallbackHandler.py:866). In LangGraph 1.2.11 the interrupt does not leave the root as an exception: the graph returns normally, with the interrupt in the return value. The root run ends through on_chain_end and the branch that stores the context never runs. The interrupted node’s observation does come out right: level DEFAULT and the Interrupt in the status message, not marked as an error, which is the correct behaviour.

The fix is verified and does not depend on the SDK changing:

from langfuse import Langfuse
from langfuse.langchain import CallbackHandler

trace_id = Langfuse.create_trace_id(seed=thread_id)
handler = CallbackHandler(trace_context={"trace_id": trace_id})

With that, the interrupt and the resume land in the same trace even if the handler is a different object and even if another replica serves the resume, because the identifier derives from the thread_id and not from in-memory state. It is the same technique the documentation recommends for joining several agents into one trace, applied to joining an agent with itself.

With one nuance to decide deliberately. Seeding with the bare thread_id puts every turn of that conversation into a single trace, which grows without bound while the conversation stays alive. If what you want is one trace per turn with the resume attached to the right turn, seed with thread_id plus the turn number, and leave conversation grouping to the session identifier, which is what it is for. Since v4 has no trace table, a huge trace is not a huge row: it is a filter returning many rows, and what suffers is the interface when opening it.

Thread, session, user, and what is only read at the root

The handler recognises five metadata keys: langfuse_session_id, langfuse_user_id, langfuse_trace_name, langfuse_tags and langfuse_prompt. The first four are read only when parent_run_id is None, that is, only on the root run (CallbackHandler.py:587). Putting them in the config used to invoke a subgraph does nothing.

The recommended route, and the one that works without surprises, is the context manager:

from langfuse import propagate_attributes

with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["production", "support-l1"],
):
 graph.invoke(payload, config=cfg)

What LangGraph contributes on its own arrives anyway. In the measured run these show up as observation metadata: langgraph_node, langgraph_step, langgraph_triggers, langgraph_path, langgraph_checkpoint_ns, checkpoint_ns and thread_id, plus ls_provider, ls_model_type and ls_integration. The thread_id is also promoted to trace metadata.

That is fine, but keeping in mind the first article: in events_core what is indexed is metadata_names, that is the key names, not the values. Filtering by langgraph_node = "planificador" is a scan. If you are going to segment by node routinely, the right place is the observation name, which is where the node already appears, or a tag, assuming that tags are not indexed either. The cheap field for slicing by node in v4 is the observation name itself.

The session identifier deserves an explicit decision. LangGraph’s thread_id and Langfuse’s session_id are the same concept and they do not connect on their own. Mapping them one to one is the reasonable thing, and it is also what makes the whole conversation navigable in the interface even when each turn is a separate trace.

The filter that drops your own spans

This is the finding that surprises most people while setting it up. The Langfuse span processor does not export everything that passes through the TracerProvider. It exports what satisfies one of these three conditions (langfuse/_client/span_filter.py:104):

It was created by the Langfuse SDK’s own tracer.
It carries at least one attribute starting with gen_ai.
Its instrumentation scope matches one of thirty-five prefixes in a closed list.

The list covers the expected and a bit more: openinference, litellm, haystack, langsmith, strands-agents, pydantic-ai, autogen-core, vllm and a couple of dozen opentelemetry.instrumentation.* entries.

The consequence for an agent is direct. If you instrument your tools with OpenTelemetry by hand so that the ERP call or the Postgres query shows up inside the agent’s trace, those spans are dropped silently, with no log and no error. There are three ways out, in order of cleanliness:

Create those observations with the Langfuse SDK, with start_as_current_observation(as_type="tool"), which is what the rest of this article assumes.
Put some gen_ai.* attribute on the span, which is abusing the convention to smuggle the span through, so only if the span does describe an interaction with a model.
Send those spans to your normal OpenTelemetry collector instead of Langfuse, and correlate by trace identifier. For a sovereign platform with its own collector, this is the sensible option: Langfuse keeps the model side, the collector keeps the system side, and the trace identifier joins them.

The behaviour being silent is what makes this take an afternoon to find. It belongs in the runbook.

The model and the cost with a gateway in the middle

One detail that affects anyone putting LiteLLM in front, which is the setup described in the operational pair and in GPU multi-tenancy.

The model name the handler puts on the generation comes first from the ls_model_name metadata and, failing that, from the component’s serialisation (CallbackHandler.py:1297). With a ChatOpenAI pointed at the gateway, that name is whatever alias you defined in LiteLLM, for example qwen-32b-internal. Langfuse computes cost from its own model table, which does not know that alias, so the generation arrives with tokens and without cost.

It is fixed by defining the model in Langfuse under that same name with its price per million tokens, which is also the only way for the cost to reflect what your GPU costs rather than what somebody else’s API costs. It is an afternoon of work and it stops the agentic platform’s cost panel from sitting at zero forever.

How this looks in code

The minimum to have in place for a LangGraph agent to produce useful traces in Langfuse v4:

import os
from typing import Annotated, TypedDict

from langfuse import Langfuse, get_client, propagate_attributes
from langfuse.langchain import CallbackHandler
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition

# 1. Client. Masking applies because the data goes through the SDK.
def mask(*, data, **kwargs):
 return redact_pii(data)

Langfuse(
 public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
 secret_key=os.environ["LANGFUSE_SECRET_KEY"],
 host=os.environ["LANGFUSE_HOST"],
 mask=mask,
 environment="prod", # lowercase, dashes, 40 chars, must not start with "langfuse"
 sample_rate=float(os.getenv("LANGFUSE_SAMPLE_RATE", "1.0")),
)

# 2. Graph. The decision node carries "agent" in its name on purpose.
class State(TypedDict):
 messages: Annotated[list, add_messages]

graph = StateGraph(State)
graph.add_node("agent_planner", model_node)
graph.add_node("tools", ToolNode(TOOLS))
graph.add_edge(START, "agent_planner")
graph.add_conditional_edges("agent_planner", tools_condition)
graph.add_edge("tools", "agent_planner")
app = graph.compile(checkpointer=checkpointer)

# 3. Per turn: deterministic trace seeded with thread and turn, session = thread.
def run_turn(thread_id: str, turn: int, text: str, user: str):
 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {
 "callbacks": [handler],
 "configurable": {"thread_id": thread_id},
 }
 with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["agent", "support"],
 ):
 out = app.invoke({"messages": [HumanMessage(text)]}, config=cfg)
 return trace_id, out

# 4. Resume after human approval: same seed, same trace.
def resume_turn(thread_id: str, turn: int, decision):
 from langgraph.types import Command

 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {"callbacks": [handler], "configurable": {"thread_id": thread_id}}
 return app.invoke(Command(resume=decision), config=cfg)

# 5. Score the turn. Scores do not travel over OTLP: they have their own route.
get_client().create_score(
 trace_id=trace_id,
 name="resolved",
 value=1,
 data_type="BOOLEAN",
)

Two notes on what is not in the example. sample_rate is head sampling and per trace, so it samples whole turns, which is what you want: half an agent trace is no use to anybody. And if the process is a web server, the client is built once at startup, not per request, whereas the handler can be per turn with no problem, because with the seeded identifier it no longer holds state worth preserving.

Agent instrumentation checklist

Nodes representing a model decision carry agent in the name. The rest do not.
Every turn has a deterministic trace identifier seeded with thread and turn number.
LangGraph’s thread_id goes as Langfuse’s session_id, always.
Trace attributes are set with propagate_attributes, not as metadata on a subgraph.
The gateway’s model alias is registered in the Langfuse model table, with a price.
Tools doing work outside LangChain are instrumented with the Langfuse SDK, not with bare OpenTelemetry.
There is an alert on ingestion queue depth, and the average number of steps per turn is known.
Someone has measured, on the actual installation, how many observations and how many bytes a real turn produces. The figures in this article come from a synthetic bench.
Sampling is set through an environment variable, so it can be lowered without deploying code.

Traps

Type AGENT does not mean there is an agent. It means somebody wrote agent in the name. And the other way around: a whole graph without that string produces not one agent-type observation.

Putting langfuse_session_id in a subgraph’s metadata does nothing. It is only read on the root run.

An interrupt splits the trace in two. Unless you seed the identifier. And with several replicas, the handler’s in-memory state is never going to save you, because the resume can land on another pod.

Your OpenTelemetry spans do not arrive. The default filter drops them without warning if the scope is not in the list and there are no gen_ai.* attributes.

The number to watch is not users, it is steps per turn. Volume grows with the square of the steps, and the steps are decided by the prompt, not by the infrastructure.

You store megabytes and the interface shows you a thousand characters. The materialised view’s truncation to two hundred characters and the thousand-character read ceiling are still there. You pay for the full ingestion and see a clipping, unless you open the specific observation.

Cost comes out at zero with the gateway in front. Model alias unknown to the price table.

flush_at above the queue size blows up when constructing the client. With an OpenTelemetry ValueError that does not mention Langfuse anywhere.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front (this article): instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Own measurement of 13 September 2026: LangGraph 1.2.11, langchain-core 1.6.3 and Langfuse Python SDK 4.15.2, with OpenTelemetry’s InMemorySpanExporter and GenericFakeChatModel, no network.
Langfuse SDK 4.15.2 source: langfuse/langchain/CallbackHandler.py, langfuse/_client/span_filter.py, langfuse/_client/span_processor.py, langfuse/_client/span.py.
PyPI downloads over the last thirty days via pypistats, checked on 13 September 2026.
GitHub stars for LangGraph, CrewAI and the OpenAI Agents SDK, checked on 13 September 2026.

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

Sat, 12 Sep 2026 09:00:00 +0200

A follow-up to Keycloak in an AI platform, which flagged the gap: resource indicators are not supported and protected resource metadata belongs to the MCP server. This post tries to close it. Verified against revision 2026-07-28 of the specification, the official Python and TypeScript SDKs, Keycloak 26.7.3 and LiteLLM 1.102.0.

TL;DR

The specification splits three obligations and the authorisation server covers only one. The MCP server must publish its protected resource metadata. The client must send the resource indicator in both requests, and must do so even if the authorisation server does not support it. The MCP server must validate that the token was issued for it. Keycloak does not understand the indicator and considers the metadata someone else’s, so all three end up in the resource layer.

The Python SDK mounts the metadata on its own, but degraded. As soon as the resource server URL is configured, the route appears. What that automatic route publishes carries a single authorisation server, reuses as its scope catalogue the ones the middleware requires, and leaves the name and the documentation null. To publish a complete document you have to mount the route by hand.

Audience validation comes switched off in Python and does not exist in TypeScript. In Python, if the resource URL is configured and the corresponding flag is not enabled, the SDK emits a deprecation warning and behaves as if it were disabled; the docstring itself promises that version 3 will set it to true. In TypeScript the bearer verification does three things, and none of them is looking at the audience. Neither of the two ships a signature verifier with public keys.

Token passthrough is forbidden in writing, and the substitute has a name. The specification says the MCP server must not accept or relay tokens that were not issued for it, and that if it calls upstream APIs the token must be a different one. The mechanism is token exchange, which Keycloak supports in its standard version with a limitation you need to know: the audience is a client identifier, not a resource URL.

There is an official extension that changes the whole picture, and it is stable. Enterprise-managed authorisation replaces the redirect to the MCP authorisation server with an exchange at the corporate identity provider, which evaluates the policy and issues a grant based on an identity assertion. It has been stable since June 2026 and there are servers in production.

Keycloak implements it halfway and in experimental. It only knows how to act as receiver, not as issuer, behind a feature flag, and against draft 01 when the working group is on 04. The official documentation literally says not to use it in production.

LiteLLM already implements it on the client side and does not document it. In 1.102.0 there is an authentication mode towards MCP servers that runs the two stages of that grant. There is not a single documentation file that mentions it.

You are here: the resource side

The previous post walked through the identity provider. This one walks through what sits on the other side of the token, which is where the MCP specification puts almost all its normative obligations.

The natural reading order is When MCP grows up for the basic setup, Keycloak in an AI platform for the identity piece, and this one for the missing layer. The MCP gateway seen from the inside is in its own post.

The analogy: the visa and the border checkpoint

A consulate issues visas. A border post checks them. They are two different offices, and the classic mistake consists of assuming that because the visa is genuine it is good for entering through any door.

The visa has written on it which country it is good for. That is the token’s audience, and here the problem appears: the consulate we work with does not know how to write the destination the traveller asks for, because it does not understand that field on the form. It knows how to write a destination if asked for it under a different name, by way of a pre-agreed stamp, which is the workaround of scopes and the audience mapper.

And then there is the other half, the one almost nobody builds. The border post has to exist, it has to announce where it is and from which consulates it accepts visas, and it has to read the destination written on the visa before letting anyone through. If the border post limits itself to checking that the stamp is genuine and does not look at the destination, any valid visa from the same consulate is good for entering. That is exactly what the two official SDKs do by default.

Part 1. The split of obligations

From revision 2026-07-28, with the normative words as they are:

Obligation	Whose	Status with Keycloak
Publish protected resource metadata (RFC 9728)	MCP server, MUST	Out of its scope, by design
Send the resource indicator (RFC 8707) in authorisation and in token	Client, MUST, even if the authorisation server does not support it	It does not understand it
Validate that the token was issued for oneself	MCP server, MUST	The resource’s responsibility
Not accept or relay other tokens	MCP server, MUST NOT	The resource’s responsibility
Validate the issuer of the authorisation response (RFC 9207)	Client, MUST; authorisation server SHOULD emit it	Supported
Consent for each dynamically registered client	MCP proxy with a static identifier, MUST	Configurable

The most confusing part of that split is the second line. The specification obliges the client to send the parameter regardless of whether the authorisation server supports it. With Keycloak, that parameter is lost: the official documentation says it cannot recognise it, and the standard behaviour towards an unknown parameter is to ignore it. The client complies, the token comes out, and what does not come out is the correct audience. The failure appears later and somewhere else, which is the worst kind of failure.

It is worth knowing where that work stands. The matter is issue 14355 of the project, open, with a milestone in 26.8.0. There was a complete implementation in pull request 35711, with a mapper of its own and an extension point for resolving resources, which went back to draft in October 2025 because of a total lack of tests and because it was decided to redo it in phases, starting with a single resource per client. In March 2026 a new issue was opened for experimental support, also with a milestone in 26.8.0 and with no associated pull request. In 26.7.x there is no feature flag that enables it.

What is new in the revision and affects the setup

Four things changed with respect to the previous revision and are worth collecting:

RFC 9207. A new section on validating the authorisation response. The client must record the issuer from the validated metadata document, and the comparison is literal: you are not allowed to normalise case, elide the default port, add or remove the trailing slash or re-encode characters before comparing.
Offline tokens leave the catalogue. A new section on refresh tokens: MCP servers should not include the corresponding scope either in the challenge header or in the supported scopes of their metadata.
Dynamic registration becomes deprecated. It goes from optional to optional and deprecated, retained for compatibility with authorisation servers that do not support client identifier metadata documents. And a new requirement is added to registration: the application type is mandatory, and a client must not reuse credentials from another authorisation server, it has to register again.
Step-up elevation is rewritten. The menu of three server strategies disappears and is replaced by two rules: the challenge’s scope attribute describes what is needed for the requested resource, with no obligation to include what has already been granted, and scope accumulation becomes the client’s responsibility. With a new requirement for the server: it must take scope hierarchies into account, where a broad one implies the narrow ones.

Part 2. What the SDKs give, and what they do not

Here is the uncomfortable part, and it is the main reason for writing this post. Verified by reading the code of the Python SDK and of the TypeScript 2.0.0-alpha one.

Protected resource metadata

In Python the complete model exists and it mounts on its own. The metadata class carries the RFC fields: resource, authorisation servers with a minimum of one, key set URL, supported scopes, bearer methods with a default value of header, name, documentation, policy, terms, and the client certificate and DPoP fields. The handler serves the document with a one-hour cache directive, and the URL is built by inserting the well-known path in front of the resource path, as the RFC requires.

As soon as the resource server URL is configured, the route appears with no further work. But what that automatic route publishes is degraded in three points: it passes a single authorisation server, it takes as its scope catalogue the ones the middleware requires (which are not the same as the ones the resource supports), and it passes neither the name nor the documentation, which come out null. To publish a complete document you have to call the route creation function by hand.

In TypeScript it exists and does not mount on its own. The function that builds the document emits only five fields: resource, authorisation servers, supported scopes, name and documentation. It does not emit the supported bearer methods, nor the key set URL, nor anything about DPoP. And serving it is explicit: either you call the response function from your own handler, or you mount the metadata router. There is no point in the SDK that does it on its own.

Audience validation

This is the one to fix on day one.

In Python it comes disabled. The token verifier is a single-method protocol, that is to say a hole the implementer fills. The access token’s resource field is the indicator you set yourself. And the comparison only happens if you ask for it: there is a function that normalises as a URL and ignores the trailing slash, but the resource server URL reaches it empty unless the validation flag is enabled.

The detail to read twice is in the configuration: if the resource URL is set and the flag is not, the SDK raises a deprecation warning and behaves as if the flag were false. The docstring says version 3 will set the default to true. Until then, a Python server configured with authorisation accepts tokens issued for another resource, unless the implementer enables the flag or validates the audience inside their own verifier.

In TypeScript it is not validated at all. Bearer token verification does exactly three things: strip the prefix, check the required scopes, and require that the expiry is present and not past. A search for audience or key set across the server and middleware packages returns no validation. It is all left in the hands of whichever verifier you plug in.

Neither of the two ships a signature verifier with public keys. In Python the only example is introspection against the authorisation server, and its audience check sits behind a flag that also comes out false. In TypeScript the interface is empty. That is: the component that validates the token, which is the one holding up the most important normative requirement of the specification, is your own code in both cases.

The authentication challenge

Python builds the challenge with the error and its description, and adds the metadata URL only if it is configured. It returns 401 with invalid token and 403 with insufficient scope, both from the same function, so both carry the metadata URL. What it never emits is the scope parameter, which revision 2026-07-28 asks to be included, and which is exactly what the client needs in order to know what to ask for.

TypeScript does emit the scope when there are required scopes, in addition to the metadata URL, and it correctly maps invalid token to 401 and insufficient scope to 403.

That is, each SDK gets a different half right. The Python one validates better and warns worse; the TypeScript one warns better and does not validate.

Scopes, which are not per tool

In both SDKs scopes are a static list at the transport mounting level, not per tool. There is no per-tool scope declaration and no step-up elevation help on the server side. Emitting a 403 with the specific scope that call requires, which is what the specification describes, is your own code.

That is the origin of almost all the fine-grained authorisation you have to build, and it is why there is a Part 6.

Which revision each one announces

Python is already on 2026-07-28. The revision appears in the known versions and in the list of modern versions, described as the ones that use the stateless envelope per request. The server discovery method is registered with a default handler, and the method and name headers are validated against the body.

TypeScript has it, but in a separate list. The latest protocol version constant is still 2025-11-25, because that list is only the one for the initial handshake. The modern era lives in a separate module, with a comment explaining the reason: keeping them deliberately separate so that adding a revision there never leaks a modern version string into a 2025-era handshake.

And it is worth recalling the mismatch that already came up in the MCP gateway post: LiteLLM 1.102.0 still announces 2025-06-18.

Part 3. Building the protected resource

With the above, the list of what you have to write is short and concrete.

1. A token verifier that validates the signature and the audience. Against the realm’s key set, checking issuer, expiry and that the audience contains the MCP server’s canonical URL and only things that concern it. Here the side effect of the Keycloak workaround comes in: if a client asks for two scopes from two different resources, out comes a token with two audiences, because each mapper contributes its own to the array. There is no client policy condition or executor in 26.7 that limits the number of audiences per token. The practical defence is the resource’s: reject tokens whose audience includes other resources, instead of merely checking that its own is present.

2. The complete metadata document, with all the authorisation servers you accept, the real catalogue of scopes the resource understands, and the name and documentation filled in. In Python, mounting the route by hand instead of leaving the automatic one. In TypeScript, mounting it, full stop.

3. The authentication challenge with the scope. In Python you have to add it, because the SDK does not emit it. And you have to take scope hierarchies into account, which is a new requirement of the revision.

4. The decision between local validation and introspection. Validating the signature locally costs no network, but revocation does not take effect until the token expires: there are no revocation lists for signed tokens. Introspection costs a round trip per request and only confidential clients can invoke it. The reasonable combination is a short token life and introspection on the operations that change state.

5. Binding state to the user. The new revision is explicit about this, because with protocol sessions gone, state is carried in handlers that travel as an ordinary tool argument. Servers must verify every incoming request and must not treat possession of a handler as authentication; and they should bind the handler to the user on the server side, for example storing the state under a key that combines the user identifier derived from the verified token with the handler, and reject the handler if someone else presents it. It is a change in the way of working compared with the sessions of before.

Part 4. The hop from the gateway to the MCP server

The real shape in an inference platform is not client against MCP server: it is client against gateway, and gateway against MCP server. That second hop is where it is decided whether the architecture is correct.

The specification leaves no room. The MCP server must not accept or relay tokens that were not issued for it. And if it makes requests to upstream APIs, it may act as an OAuth client of theirs, but the token it uses there is a different token, issued by the upstream authorisation server, and it must not forward the one it received.

The standard mechanism for that is token exchange, and here you need to know three Keycloak details.

First, the permission model changed. The old version required fine-grained administration permissions and an explicit exchange authorisation on the target client. The standard version, supported since 26.2, does not require them: it is enough for the requesting client to be confidential and to have the corresponding switch enabled. In exchange there is a surprising condition: the subject token has to carry the requesting client in its audience, unless it exchanges its own token. That is, the gateway needs to appear in the audience of the user token, which is achieved with an audience mapper in a default scope of the gateway. Only then can it narrow the audience to that of the specific MCP server.

Second, the audience is a client identifier, not a resource URL. The parameter filters audiences, that is to say it narrows them, which is what you want. But it takes the identifier of a client registered in the realm. The practical consequence is that the MCP server has to be registered as a client, and its identifier should be its canonical URL, so that the resulting audience matches what the resource validates. It is a naming trick, and it has to be documented for whoever comes after. The documentation itself acknowledges it: token exchange does not yet support the resource parameter.

Third, the subject token type is limited. The standard version only accepts access tokens as subject.

There is also a new experimental feature in 26.7 that adds a parameterised scope type for validating whether the requesting user is authorised to act on behalf of another. Interesting for delegation, but experimental.

What the gateway does today

In LiteLLM 1.102.0 the authentication modes towards MCP servers number twelve, with a default value of none. The ones that matter for this discussion are four.

oauth2_token_exchange implements the standard exchange. It sends the correct grant type, the subject token and its type, and the audience, never the resource. The omission is deliberate according to the code itself: fabricating a target risks an invalid target error. It caches the resulting token under a key that combines the subject token with the whole configuration, and with a lifetime equal to the token’s minus one minute. And it blocks, never degrades: with no incoming token it returns 401, which the edge turns into a challenge with the resource metadata URL; a rejection from the identity provider returns 401; a gateway configuration error returns 500; a transport failure, 503.

true_passthrough and oauth_delegate forward the client’s authorisation header as is. That is token passthrough, by the name the specification gives it, and its legitimate use is narrow: when the token the client presents was already issued for the target MCP server and the gateway is a mere transport. Outside that case, it is non-compliant.

oauth2_id_jag is the surprise, and it deserves a part of its own.

Part 5. The enterprise extension, which changes the picture

In June 2026 the MCP project published an authorisation extension that solves a different problem from the one the standard flow solves, and which in an organisation with an identity provider of its own is the real problem.

The normal flow is per user: each employee authorises each client against each MCP server. It works for consumer applications and it does not work in a company, because onboarding a person requires authorising dozens of services one by one and offboarding requires revoking them one by one.

The io.modelcontextprotocol/enterprise-managed-authorization extension inverts that. It is in the stable specifications directory of the extensions repository, it comes from SEP-990, and the flow is this:

The MCP client authenticates the user against the corporate identity provider through the normal flow, and stores the identity assertion, which may be an OpenID identity token or a SAML assertion.
When the server indicates that enterprise-managed authorisation is needed, the client exchanges that assertion at the corporate provider for an authorisation grant based on an identity assertion. The provider evaluates the organisation’s policy there: group membership, roles, conditional access.
The client presents that grant to the MCP authorisation server and obtains an access token.
The sentence that defines the extension, verbatim: the user is not redirected to the authorisation endpoint of the MCP authorisation server.

The MCP authorisation server validates the signature against the corporate provider’s key set, plus audience, issuer and expiry, and uses the subject as the user’s stable identifier, with the email as a fallback for linking earlier accounts.

What that buys is exactly what a compliance file asks for: policy in a single place, an auditable decision at the identity provider, and centralised revocation that takes effect across all clients at once. The employee who loses access stops receiving grants, without touching any server.

The underlying standard is a draft from the IETF’s OAuth working group, under the name of JWT identity assertion authorisation grant, revision 04 of May 2026, signed by people from Okta, Ping Identity and an independent author. It profiles identity chaining across trust domains by combining token exchange with the JWT profile.

And here comes the problem

Keycloak implements it only halfway. It has its own documentation page, behind a feature flag. And three limitations you need in front of you before designing anything:

It only acts as receiver. It accepts assertions issued by an external provider and issues local tokens. Native support for acting as issuer, the documentation says, is not yet fully implemented. In the extension’s flow, the issuer is the corporate provider: if that provider is Keycloak, the missing piece is exactly the one you need.
It is based on draft 01 while the working group is on 04.
The official documentation says not to use it in production.

The asymmetry is striking and worth pointing out: Keycloak does not support a standard published eight years ago such as resource indicators, and does partially support a draft from a few months back.

What is already in the gateway, undocumented

In LiteLLM 1.102.0, the oauth2_id_jag mode runs the two stages of the flow. In the first it asks the corporate provider for a token exchange with the requested token type specific to this grant, sending the user’s identity token as subject and, optionally, audience, resource and scope. In the second it presents the resulting assertion to the resource’s authorisation server with the JWT bearer grant type. In both the gateway authenticates as a client with a JWT signed by private key.

Two operational warnings that come out of the code and are nowhere else:

The subject comes from the caller’s identity token, or from the one captured at login. And only the generic OIDC provider captures assertions: with Google, Microsoft or SAML there is none, and all users fail. The code raises a warning when loading the configuration.
A badly configured server refuses with a 500 instead of falling back to the static credential. It is the right decision, and it is worth knowing before it happens.

And the main warning: there is no documentation. Not a single file in the repository that mentions this mode. Anyone using it is reading code, like this post.

Part 6. Per-tool authorisation

None of the previous pieces answers the question the business ends up asking: whether this person may call this tool with these arguments. The identity provider gives roles and scopes, and with that you do not model a graph of relations, as was already argued in the previous post.

What is published today, with sources:

OpenFGA is the only one with a dedicated product guide, updated on 9 September 2026. The pattern is a tool type with an invocation relation, checked on each call, and a listing query so the client only sees the tools it may invoke. It is a pattern and a modelling language, not a reference implementation.
Pomerium ships a per-tool policy criterion, with exact, prefix, suffix and list matching.
Kong offers it as a product, with per-tool control lists.
Traefik Hub goes as far as parameter-level restrictions, which is the finest published grain.
Envoy has an MCP filter that extracts protocol attributes for fine-grained access control, declared to be in active development and with no OAuth of its own: the hook is external authorisation towards a policy engine.

The lowest-friction option in a platform that already has a service mesh is external authorisation against a policy engine. The most expressive one, the relations model.

Part 7. What still has no standard

Two gaps the specification does not cover and that have to be closed from outside.

Tool definition change. Neither revision 2026-07-28 nor the SDKs pin a hash of the description or of the input schema. The best practices page covers confused deputy, token passthrough, server-side request forgery in discovery, state handler hijacking, server confusion and validation of the authorisation URL scheme, but there is no integrity or pinning requirement. The tool that does it is the Invariant Labs scanner, under the Apache 2.0 licence, which pins hashes to detect definition change after approval and also ships a proxy mode with guardrails. It already came up in the MCP gateway post and it remains the only answer.

The public guidance that does get into this is the NSA’s on MCP security design considerations, from May 2026, done with Carnegie Mellon’s software engineering institute. Two of its observations are directly actionable: that authorisation in MCP is optional and that the specification imposes no requirement on token lifecycle management, so expiry and rotation are left in the organisation’s hands; and the warning against dynamic tool discovery with no origin verification and no authorisation checks, which is the most direct acknowledgement of the risk in a state guidance document.

Part 8. Who covers what

With all of the above, the decision table for the layer you put in front of the MCP server:

Piece	Publishes resource metadata	Validates audience	Token exchange	Per-tool authorisation
Python SDK, automatic route	Yes, degraded	Not by default	No	No
TypeScript SDK	Does not mount on its own	No	No	No
Traefik Hub (commercial)	Yes, automatic	Not documented	Not documented	Yes, down to parameter
Kong, OAuth plugin for MCP	Not documented	Yes	Not documented	Yes, per-tool lists
Pomerium	Partial	Not documented	Not documented	Yes, per-tool criterion
agentgateway (Solo.io)	Yes	Yes, by JWT policy	Yes, including the assertion grant	Yes, with expressions
mcp-context-forge (IBM)	Not in release notes	Not documented	Yes, since 1.0.6	Yes, per-role control
Envoy, MCP filter	No	No	No	Via external authorisation
LiteLLM 1.102.0	Yes, and authorisation server metadata too	Not on the incoming JWT	Yes, and the assertion grant	Yes, per-key and per-team permissions

Two warnings about that table. The Kong plugin requires a minimum version of 3.12, is in technical preview and needs a licence for its AI edition; it also introduces a breaking change in 3.13, which starts treating all traffic as MCP in order to close a possible authentication bypass. And on oauth2-proxy I found no primary documentation either for or against protected resource mode: I am not claiming it does not have it, I am claiming it is not documented where I could look.

Reference configuration

Keycloak: the per-resource scope, optional, with its audience. That it is optional and not default is the key piece, because it is what makes the client have to ask for it explicitly and makes it work as a substitute for the resource indicator:

kcadm.sh create client-scopes -r plataforma \
 -s name=mcp:inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

And the MCP server registered as a client with its canonical URL as identifier, so that token exchange produces the right audience:

kcadm.sh create clients -r plataforma \
 -s clientId=https://mcp.ejemplo.es/inventario \
 -s enabled=true -s publicClient=false -s consentRequired=true

The gateway, with token exchange instead of token passthrough:

mcp_servers:
 inventario:
 url: "https://mcp.ejemplo.es/inventario"
 transport: "http"
 auth_type: "oauth2_token_exchange"
 client_id: "litellm-gateway"
 client_secret: os.environ/GW_SECRET
 token_exchange_endpoint: "https://sso.ejemplo.es/realms/plataforma/protocol/openid-connect/token"
 audience: "https://mcp.ejemplo.es/inventario"
 subject_token_type: "urn:ietf:params:oauth:token-type:access_token"
 token_exchange_profile: "rfc8693"

The resource verifier, which is what nobody hands you ready-made. The essential bit is the last check, the one that rejects other resources’ audiences instead of settling for finding its own:

RECURSO = "https://mcp.ejemplo.es/inventario"

async def verificar(token: str) -> AccessToken | None:
 claims = jwt.decode(
 token, await jwks(),
 algorithms=["RS256"],
 audience=RECURSO, # validates that I am there
 issuer=ISSUER,
 )
 aud = claims["aud"]
 aud = [aud] if isinstance(aud, str) else aud
 # and that nobody else is: the scopes workaround allows
 # tokens with two audiences, and that reopens the confused deputy
 if set(aud) - {RECURSO, CLIENTE_GATEWAY}:
 return None
 return AccessToken(
 token=token,
 client_id=claims["azp"],
 scopes=claims.get("scope", "").split(),
 expires_at=claims["exp"],
 resource=RECURSO,
 )

And when mounting the server, the flag that does not come set:

auth = AuthSettings(
 issuer_url=ISSUER,
 resource_server_url=RECURSO,
 validate_token_resource=True, # without this there is only a warning
 required_scopes=["mcp:inventario"],
)

Checklist

Enable audience validation in the Python SDK, or write it into the verifier if it is TypeScript. Without that, the central normative requirement is not met.
Reject tokens with other resources’ audiences, not just check that your own is present.
Mount the metadata document by hand, with all the authorisation servers and the real catalogue of scopes.
Add the scope parameter to the challenge, in Python, and account for scope hierarchies.
Register each MCP server as a Keycloak client with its canonical URL as identifier, so that token exchange produces the right audience.
Replace any token passthrough mode with token exchange, except the narrow case where the token was already issued for the target.
Check that the gateway appears in the audience of the user token, or the exchange will fail.
Bind state handlers to the user on the server side, and never treat them as authentication.
Decide between local validation and introspection knowing that the first delays revocation until expiry, and shorten the token life accordingly.
Pin a hash of the description and the input schema from outside, because the specification does not cover it.
Put a policy engine in for per-tool authorisation, and do not try to model it with roles.
If you are considering the enterprise extension, check first who issues the grant: Keycloak today only knows how to receive it, in experimental and against an earlier draft.

Traps and things that are not what they look like

The Python SDK mounts the metadata document on its own, and that is why it looks solved. What it publishes carries a single authorisation server and no name and no documentation.

The audience validation deprecation warning does not enable anything. It warns and carries on behaving as disabled.

The TypeScript SDK does not validate the audience at all. It is not configuration, it is that there is no code for it.

Python’s challenge does not carry the scope, so a client that receives a 403 does not know what to ask for.

The scopes workaround allows two audiences in the same token. Each mapper contributes its own, and there is no client policy that limits it. The defence is in the resource.

The token exchange audience is a client identifier, not a URL. If the MCP server is not registered with its URL as identifier, the resulting audience will not match what the resource validates.

Token exchange requires the gateway to be in the audience of the user token. It is the condition that wastes the most time the first time round.

Dynamic client registration is deprecated as of this revision, and the substitute, client identifier metadata documents, is experimental in Keycloak and has an open bug with documents carrying unknown fields, which blocks login with real clients.

LiteLLM’s identity assertion grant depends on the generic OIDC provider. With Google, Microsoft or SAML no assertion is captured and all users fail.

Step-up elevation is no longer the server’s responsibility. Scope accumulation moved to the client in this revision.

The authentication elevation signal goes in a 401, not a 403. Insufficient scope is 403; asking for a higher authentication level is a different thing and a different code.

Closing

The summary of this post fits in one sentence: the identity provider is the cheap half of the problem, and the expensive half is the protected resource, which no SDK hands you ready-made and which in the two official ones comes with audience validation disabled or absent.

From that comes an order of work. First the token verifier, with the audience validated in both directions, because without it everything else is decoration. Second, the complete metadata document, mounted by hand. Third, replacing any token passthrough with exchange, with the MCP server registered with its URL as identifier. Fourth, a policy engine for what roles do not model.

And a scheduling decision worth taking with the information in front of you. The enterprise-managed authorisation extension is the right route for an organisation with a provider of its own, it is stable and it has servers in production behind it. What is not ready is Keycloak as issuer. Anyone who wants that flow today has two honest options: delegate the mechanics to a gateway that implements it, or wait. Putting it into production with a feature the official documentation marks as experimental and expressly advises against is not one of them.

Sources

Especificación MCP, revisión 2026-07-28: autorización, registro de clientes y buenas prácticas de seguridad.
Extensión de autorización gestionada por la empresa y su especificación estable, del SEP-990. Anuncio de 18 de junio de 2026.
draft-ietf-oauth-identity-assertion-authz-grant, revisión 04 de 21 de mayo de 2026, grupo de trabajo OAuth del IETF.
Código de los SDK oficiales de MCP: python-sdk y typescript-sdk 2.0.0-alfa, clonados el 12 de septiembre de 2026.
Integración de Keycloak con Model Context Protocol, intercambio de tokens y concesión de autorización por aserción de identidad, consultados el 12 de septiembre de 2026.
Incidencias de Keycloak 14355, 47117 y 51039, y petición de cambios 35711.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
NSA y SEI, Model Context Protocol: Security Design Considerations, mayo de 2026.
OpenFGA, autorización de servidores MCP, actualizado el 9 de septiembre de 2026.
mcp-scan de Invariant Labs, Apache 2.0.
Documentación de agentgateway, Traefik Hub, Kong y Pomerium, consultadas el 12 de septiembre de 2026.

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

Sat, 12 Sep 2026 06:00:00 +0200

Seventh article in the operational track of the control layer. The fourth dealt with the coexistence of human traffic and agentic traffic on a fixed fleet. This one deals with the prior question: how big the fleet has to be, and how big the gateway in front of it. Verified against LiteLLM 1.102.0 (commit of 11 September 2026) and vLLM 0.29.0, with load data published between May and September 2026.

TL;DR

Until this year, sizing for agents meant extrapolating a chat profile and adding margin. In 2026 the first production-scale characterisations were published and the profile turned out to be something else. These are the seven conclusions.

The median prompt of an agentic call is 68,000 tokens, of which 63,000 arrive cached. The measurement covers 13.5 million sessions, 3.2 million users, 760.5 million model calls and 44.9 trillion prompt tokens in one week of June 2026. The median input-to-output ratio exceeds 275 to 1. Any calculation done on a profile of 800 prompt tokens and 250 output tokens describes a different system.

The load is prefill-dominant when counting tokens and decode-dominant when counting time. The genuinely new tokens that have to be prefilled on each turn are only between 1.5 and 7.3 times the output, not 275. With the prefix cache working, the split of execution time runs from 91 % to 98.6 % in decode. You size by memory bandwidth and by KV cache capacity, not by prefill FLOPs.

The sizing risk is not the average, it is the collapse of the prefix cache. The median hit rate within a turn is 98 %, it falls to 55 % at the boundary between turns and to 8 % when the client changes model. With dynamically retrieved context, prefix reuse drops to 1 %. One miss turns 142,000 prefix tokens into real prefill. Session affinity per replica stops being an optimisation and becomes a capacity requirement.

The intra-turn concurrency measured in production is 1.15 at the median and 1.4 at the 90th percentile. 93 % of tool batches contain a single invocation. The image of the agent opening twenty calls in parallel does not appear in the data. The real multiplier comes from the duty cycle and from the fact that 87 % of calls are started by the agent, not by the person.

The gateway is sized per worker, and each worker is a universe. One vCPU and 4 GiB per worker, with requests equal to limits. The limit counters, the caches and the cooldowns live in the worker’s memory unless there is Redis, so a limit of 100 requests per minute with four workers and three replicas allows 1,200. The Prisma pool is also per process: the number of connections open against Postgres is workers times connection_limit.

The CPU cost that grows with agentic load is the token counter. It serialises the entire tool catalogue on every request and tokenises it. The project itself measured 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens, and the operation passes through a concurrency limiter of four per worker. With agents sending long catalogues and six-figure contexts, that limit is the ceiling before the CPU is.

Autoscaling by memory does not work on this piece and the startup probe takes five minutes. The official chart deliberately leaves memory without a target because Prisma’s query engine never returns the maximum reached. That leaves CPU at 60 %, with a startupProbe of 300 seconds that makes any new replica arrive late. For agents there are two better metrics, and they are in /metrics.

You are here: the arithmetic that comes before buying

Classic capacity planning starts from a chat SLO and crosses two budgets, the VRAM one and the time one. That method is still correct. What changes with agents are the inputs: the prompt length, the number of calls per unit of human work, the split between prefill and decode, and the very definition of an SLO. This article replaces those inputs with measured figures and drags the result through to two numbers: gateway replicas and GPUs.

Out of scope is the splitting of traffic once it is built, which was already covered by the coexistence of humans and agents and prefix routing.

The analogy: the repair shop and the front desk

An appliance repair shop sizes its staff by counting faults, not phone calls. The difference matters because the person who brings in the washing machine occupies the front desk for three minutes, while the technician who takes it apart goes back to consult the machine’s data sheet fourteen times over the afternoon. If the shop manager sizes by visits to the front desk, they end up short of technicians and overstaffed at reception.

The inference platform has the same asymmetry. The person sends one turn. The agent sends fifteen calls for that turn, and each one drags along the complete file of everything done so far. The front desk is the gateway and the technicians are the GPUs, and the usual mistake is to size both with the same counter.

There is a second detail the analogy captures well. The technician does not read the whole data sheet each time: they have it open on the bench. That is the prefix cache. The day somebody moves them to a different bench between consultations, they have to fetch it from the archive again, and the work multiplies without the number of faults having increased.

Part 1. The real load profile

For two years, any capacity calculation for agents was done on assumptions. In 2026 three pieces of work appeared that measure the matter, and it is worth reading them together because they measure different scaffolds and their figures are not averageable.

The three sources

The characterisation at scale was published by a research team working on GitHub Copilot production telemetry (arXiv 2608.00101, 30 July 2026): 13.5 million sessions, 3.2 million users, 95.1 million user turns, 760.5 million model calls and 44.9 trillion prompt tokens, in one week of June 2026. It is the largest sample published to date.

The characterisation by benchmark suites comes from an architecture team (arXiv 2605.26297, 25 May 2026), which instruments five of them (ADE-Bench, DABStep, GAIA, SWE-bench Pro and Terminal-Bench 2.0) and measures the internal split of time.

The production measurement on a different scaffold was published by the vLLM project itself together with AgentX (vLLM blog, 8 September 2026).

The three disagree on turns per session: 15 calls at Copilot’s median, 43 turns in AgentX, 62.4 in SWE-bench Pro with Qwen. It is not a contradiction, they are scaffolds with different loops. The sizing lesson is that the number of calls per unit of human work depends on the client that connects, and that is why you have to measure it at home before fixing the size.

The figures you can actually use

Quantity	Measured value	Source
Median prompt per call	68K tokens, 63K of them cached	2608.00101
Input:output ratio (median)	above 275:1	2608.00101
Median session	3 user turns, 15 calls, 4.2 min	2608.00101
Mean session	6.1 turns, 40.6 calls, 62.6 min	2608.00101
Calls per turn (median)	4.5	2608.00101
Calls started by the agent	87 %	2608.00101
Range across user profiles	50× (23K to 1.1M tokens per turn)	2608.00101
Median input context	142K tokens; output 444 tokens per turn	AgentX
Accumulated context in SWE-bench Pro	mean 68.7 to 80.1K, maximum 146 to 166K	2605.26297
Reasoning tokens over the output	29.0 % to 67.6 %	2605.26297

Two operational observations come out of that and appear in no sizing guide.

The first: agents that fail accumulate up to 1.8 times more average context than those that finish well (2605.26297). The cost does not correlate with success, it correlates with difficulty and with getting stuck. A sizing done on the task that goes well falls short.

The second: the range across user profiles is 50 to 1. A user who only converses spends 23,000 tokens per turn; one who leaves the agent in a long loop spends 1.1 million. Segmenting by usage profile pays off more than any difficulty-based routing, because the variance is in the people, not in the requests.

What to measure before calculating

Three of your own figures, taken from a week of real traffic, are worth more than any table in this article:

Model calls per hour of human work, by user profile.
Prompt length distribution (median and 95th percentile), not the mean.
Aggregate prefix cache hit rate, with the time series, not the single number.

All three come from what the gateway already stores. In LiteLLM_SpendLogs you have the tokens per request and the key and team identifiers; in proxy_server_request you have the tools block of each call; and in vLLM’s /metrics you have vllm:prefix_cache_queries and vllm:prefix_cache_hits.

Part 2. Prefill, decode and the cache trap

Here is the finding that corrects almost everyone’s intuition, including mine when I started this article.

Counting tokens, agentic load is overwhelmingly input: 275 to 1. The apparently obvious conclusion would be to size by prefill capacity, that is by FLOPs. It is false.

What has to be prefilled on each turn is not the whole prompt, but what was added since the previous turn. That addition, measured, sits between 1.5 and 7.3 times the output length (2605.26297). The rest is served by the prefix cache. And the real split of the model’s execution time comes out like this: between 91.0 % and 98.6 % in decode, between 1.4 % and 9.0 % in prefill (2605.26297).

The hit rate that sustains that split is high and has been measured in three places: theoretical from 87.9 % to 99.3 % and empirical from 84.6 % to 99.5 % (2605.26297); above 96 % in production (AgentX); and a median of 98 % within a turn in Copilot (2608.00101).

Where it falls over

The same Copilot work measures the degradation, and it is the most actionable figure in the article:

98 % median hit rate within a turn. The detail matters: the first call of a turn hits around 45 %, the second rises to 86 % and from the third onwards it stabilises between 92 % and 94 %.
55 % at the boundary between turns, with idle gaps of 4.1 minutes at the median.
8 % when the client changes model.

And there is a fourth, worse regime: with dynamically retrieved context, that is when the scaffold injects search or RAG results that change on every iteration, prefix reuse falls to 1 % or less (arXiv 2608.15127, 15 August 2026).

A cache miss with a context of 142,000 tokens turns a prefill of a few thousand tokens into a six-figure one. The work multiplies by a factor of between 20 and 60 without the number of requests having changed. That is the difference between a fleet that meets its targets and one that seizes up mid-afternoon.

Two sizing rules come from that:

Without session affinity towards the same replica, the calculated sizing is worthless. LiteLLM’s concrete mechanism is session_affinity, which is not active by default and which was already covered in the prefix routing article. Without it, the calculation has to be redone assuming a prefill regime, which is between 20 and 60 times more expensive.
Changing model halfway through a session costs the entire cache. Any difficulty-based routing policy that moves a live conversation from one model group to another pays an 8 % hit rate on the next call. If you want to route by difficulty, do it at the start of the session.

Co-batching, which is the other effect nobody measures

Mixing a short request into the same batch as long-context requests degrades the short one’s inter-token latency. The measured figure: one long request in the batch raises time per token by 38.3 %; two raise it by 79.7 % (2608.15127). It is the technical reason why separating human and agent pools improves the human experience more than the capacity split suggests, and it is also the reason not to raise --max-num-seqs above what the KV budget allows.

Part 3. The KV budget, with the numbers of the agentic case

The formula does not change with respect to classic capacity planning, but the inputs do, and the result is surprising:

$$\text{KV}_{\text{bytes/token}} = 2 \times n_{\text{layers}} \times n_{\text{kv heads}} \times d_{\text{head}} \times \text{bytes}_{\text{elem}}$$

For a 70B-class geometry (80 layers, 8 KV heads with grouped attention, head dimension 128, 2-byte elements) that gives 0.3125 MiB per token. With the median agentic context of 142,000 tokens:

$$142{,}000 \times 0.3125\ \text{MiB} \approx 43.3\ \text{GiB per sequence}$$

For a 30B-class geometry (48 layers, everything else the same) that gives 0.1875 MiB per token, that is around 26.0 GiB per sequence.

It is worth pausing on what that means. An 80 GB H100 with the weights of a 70B model in FP8 already taking up space does not have 43 GiB free for a single sequence. The conclusion is harsh and has to be written down: serving real 70B-class agentic context demands tensor parallelism of at least 2 and a quantised KV cache. And the number of simultaneous sequences that fit, which is what sets --max-num-seqs, comes from dividing the available KV by that figure, not from a chat heuristic.

$$\text{sequences} \approx \frac{\text{VRAM}_{\text{KV available}}}{\text{KV}_{\text{bytes/token}} \times \text{max context}}$$

With FP8 in the KV, the divisor halves. It is the cheapest capacity lever there is in this load profile, and its cost in quality is validated with evals, not assumed.

Part 4. From people to concurrent requests

This is where most calculations overshoot, because it is assumed that an agent equals many simultaneous requests. The data says otherwise.

The concurrency of calls within a turn, measured in production, is 1.15 at the median and 1.4 at the 90th percentile, and 93 % of tool batches contain a single invocation (2608.00101). Intra-turn parallelism is small. Subagents appear in 44 % of AgentX sessions, with a median of 4 deployments among those that use them, so there is a long tail, but it is not the usual regime.

What does multiply is the duty cycle. The mean session lasts 62.6 minutes and contains 40.6 calls. At 444 output tokens and an interactivity of around 60 tokens per second, each call occupies around 7.4 seconds of decode, which gives between 5 and 7 active minutes out of those 62.6.

$$C = D \times \rho \times A$$

where $D$ is the people working, $\rho$ the duty cycle and $A$ the amplification. With $\rho \approx 0.10$ to $0.12$ derived from the previous figures, twenty developers give between 2 and 2.4 concurrent sequences in the steady state. It is the peak, not the average, that has to be sized for.

The amplification

Three factors published separately:

Tool failures: 9 % of turns trigger them, with a compute amplification of 4 times (2608.00101).
Subagents: 44 % of sessions, median of 4 deployments (AgentX).
Internal variance: the same task from the same agent varies by up to 30 times in cost within a single trace (2608.15127).

The product of the first two gives $(1 + 0.09 \times 3) \times (1 + 0.44 \times 3) \approx 2.9$. Nobody publishes that product as a single figure, so it is flagged as my own derivation, but an amplification factor of 3 is the reasonable design provision and it fits the measured variance.

For the twenty developers in the example: between 6 and 7 concurrent design sequences, with 142K contexts. At 43.3 GiB of KV per sequence in the 70B class, that is around 300 GiB of KV cache, that is four H100s dedicated to KV alone with the weights on top. With KV in FP8, half. With a 30B-class model, a third.

That calculation, done with the chat profile of 800 prompt tokens, would have given a single GPU. That is the distance between the two regimes.

Part 5. The gateway, which also has to be sized

The proxy consumes no GPU, but it has its own arithmetic and it is the part that produces the most surprises in production, because almost all of its state is per process.

A worker is a universe

The default value of --num_workers is 1 (litellm/constants.py:57). The usual startup is uvicorn directly with uvloop (proxy_cli.py:1477), and there are alternatives with gunicorn, hypercorn and granian.

What lives in each worker’s memory, and therefore multiplies:

The in-memory cache, limited to 200 elements with a TTL of 600 seconds (litellm/caching/in_memory_cache.py:27). With many deployments, the cooldowns and the counters compete for that same space and get evicted before they expire.
The key authentication cache, with a TTL of 60 seconds (proxy_server.py:1650), adjustable with general_settings.user_api_key_cache_ttl.
The limit and budget counters when there is no Redis (parallel_request_limiter_v3.py, local_only=True paths).
The router cooldowns, with DEFAULT_COOLDOWN_TIME_SECONDS = 5 (constants.py:75).
A periodic task scheduler of its own per worker (proxy_server.py:9674), coordinated between pods with a lock in Redis.

The practical consequence: a limit of 100 requests per minute with four workers and three replicas allows up to 1,200 without Redis. With Redis, the v3 limiter registers seven Lua scripts at startup (parallel_request_limiter_v3.py:608) and the limits become global.

One detail of the v3 limiter that matters with agentic load: LITELLM_TPM_TOKEN_RESERVATION_ENABLED comes enabled by default and reserves estimated tokens before the call, which adds a round trip to Redis per request. And there is an asyncio.Lock per instance that serialises the atomic batch checks (:665), documented by the code itself as a 99th-percentile risk under contention.

If Redis goes down, requests do not fail: there is a circuit breaker with a threshold of 5 failures and a recovery of 60 seconds (constants.py:456), and the proxy degrades to local memory. What is lost silently is the globality of the limits.

Postgres

The pool is also per process. The default values are database_connection_pool_limit = 10 and database_connection_pool_timeout = 60 (proxy_cli.py:62), injected as parameters of the Prisma URL. A pod with four workers opens forty connections. Three pods, one hundred and twenty. It is the fast route to exhausting Postgres’s max_connections without having served any traffic, and the reason the official chart offers PgBouncer inside the pod itself with maxDbConnections: 20 and maxClientConn: 1000.

On the request path there is only the authentication, which is cached for 60 seconds, and the rate limiting. Spend is deferred. The SpendLogs queue is capped at 64 MB and on overflow it discards the oldest elements with an error in the log (utils.py:6704). The write interval is proxy_batch_write_at, with a default value of 10 seconds plus a random spread of up to 5 (constants.py:1719, proxy_server.py:9704); the production guide recommends raising it to 60.

With a slow Postgres, the queue grows to those 64 MB and then loses billing data without returning any error to the client. It is the silent failure mode you have to alert on.

The token counter, which is the agentic trap

This is the piece that turns a correct sizing into an insufficient one, and it appears in no guide.

Before admitting a request, the proxy counts tokens. With chat load the cost is negligible. With agentic load it is not, for two reasons that add up.

The first is the prompt length. The project itself measured the admission counting cost at 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens (benchmarks, consulted on 12 September 2026), and published that with a counter written in Rust those figures fall to 4.9, 6.8 and 10.2 ms.

The second is the tool catalogue. The function that prepares the definitions serialises the entire catalogue to a pseudo-TypeScript and tokenises it (litellm_core_utils/token_counter.py:934), plus nine fixed tokens. The cost is linear in the size of the catalogue and it is paid on every request. That is exactly the pattern of an agent connected to an MCP gateway: a long catalogue, resent on every turn. The token bill of that catalogue was already covered in the MCP gateway article; here its other face appears, which is proxy CPU.

And on top of that there is an explicit ceiling. The counting runs in a thread with a capacity limiter whose default value is 4 per worker (token_counter.py:340, constants.py:407). With 100K prompts, four concurrent counts of 100 ms mean a theoretical ceiling of around 40 admissions per second per worker before the limiter starts queueing, and that is without having called the model yet.

What is cached is the tokeniser, with an LRU of 64 entries (utils.py:2206), not the result of the count.

Three levers, in order of return: raise TOKEN_COUNTER_MAX_CONCURRENT_COUNTS if CPU is spare, reduce the tool catalogue exposed per key, and watch the gateway overhead figure in /metrics to know whether this is happening.

Streaming and admission

There is no connection limit by default. --limit_concurrency exists but is optional, and it returns 503. The built-in admission control is only activated if general_settings.max_in_flight_requests_per_worker is defined (middleware/admission_control_middleware.py:285), with max_queued_requests_per_worker and admission_queue_timeout_seconds at 1.0 by default, rejecting with 503 and a retry-after: 1 header.

For agents, that header matters: an agentic client that retries without respecting it multiplies the load exactly when the gateway is saying it cannot take any more. The combination of gateway retries was already calculated by the fourth article and it reached 45 calls per turn in the worst case.

The default request timeout is 6,000 seconds (constants.py:512). The production guide recommends 600. With long agentic tasks, leaving it at 6,000 means a hung connection occupies a slot for an hour and a half.

Resources and autoscaling

The official chart recommends 1 vCPU and 4 GiB per worker, with requests equal to limits (helm/litellm-helm/values.yaml:235), and the production documentation suggests one worker per pod scaling horizontally. The project’s own high-performance benchmark, by contrast, uses four workers per pod with PgBouncer in front.

The HPA comes with targetCPUUtilizationPercentage: 60. And there is a comment in the chart worth reading twice: memory is deliberately left without a target because the resident consumption of Prisma’s query engine is a high-water mark that is never returned, so that an HPA on memory would add replicas and never remove them (values.yaml:260).

The other problem is startup. The startupProbe has failureThreshold: 30 with a period of 10 seconds, that is up to 300 seconds before the pod is considered ready (values.yaml:109). An HPA that reacts at 60 % CPU and takes five minutes to add capacity arrives late to any agentic burst.

The alternative, optional in the chart, is to scale by Prometheus metrics: targetRequestsPerSecond and targetTokensPerSecond over rate(litellm_proxy_total_requests_metric_total[1m]) and rate(litellm_total_tokens_metric_total[1m]). The chart itself warns that the token metric lags with long streaming responses, because tokens are accounted for on completion. For agents, that is almost always.

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

Metric	What for
`litellm_in_flight_requests`	Real saturation per worker. It is the scaling metric
`litellm_overhead_latency_metric`	Proxy cost per request; it rises with token counting
`litellm_overhead_with_guardrails_latency_metric`	The same including guardrails
`litellm_request_queue_time_seconds`	Arrival through to pre-call, includes authentication and ASGI queueing
`litellm_deployment_cooled_down`	Replicas out of the pool
`litellm_service_latency`	Redis and database latency, per service

The /health/backlog endpoint returns in_flight_requests, admitted_requests, queued_requests and rejected_requests (health_endpoints/_health_endpoints.py:1871).

What it does not expose: the depth of the SpendLogs queue or its bytes, the Prisma connections in use against the limit, nor the depth of the logging worker’s queue. All three are real failure modes and have to be instrumented from outside.

At the engine

Names verified in the vLLM 0.29.0 metrics documentation:

vllm:num_requests_running, vllm:num_requests_waiting
vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:inter_token_latency_seconds
vllm:prefix_cache_queries, vllm:prefix_cache_hits, vllm:kv_cache_usage_perc

And the ones that break dashboards on upgrade:

vllm:time_in_queue_requests is duplicated by vllm:request_queue_time_seconds.
The direct prefix cache hit rate indicator disappeared. It is now computed as rate(vllm:prefix_cache_hits) / rate(vllm:prefix_cache_queries).
vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are deprecated along with CPU swapping.
vllm:gpu_cache_usage_perc became vllm:kv_cache_usage_perc, which also breaks any HPA that uses it.

The SLO worth chasing

A 95th percentile of time to first token describes nobody’s experience when the unit of work is 43 turns. Coding tasks frequently exceed ten minutes and research tasks reach hours (2608.15127).

Two indicators serve better:

90th percentile of full-task latency, measured per session, not per request. It comes from grouping LiteLLM_SpendLogs by session identifier.
90th percentile of interactivity, in tokens per second per user. It is the one AgentX uses, with values between 58.3 and 74.2 in its runs.

To autoscale the engine, the official production-stack example uses vllm:num_requests_waiting with a threshold of 5, polling every 15 seconds and a cooldown of 360. With agentic load it is worth adding vllm:kv_cache_usage_perc as a second trigger, because with 26 to 43 GiB of KV per sequence the memory pressure arrives before the queue does.

Part 7. The engine levers, ordered by return

Verified against vLLM 0.29.0, released on 9 September 2026.

--long-prefill-token-threshold 512. The AgentX measurement attributes to it +93 % in tokens per GPU per second, with a 2.3-times improvement at the 90th percentile, by attacking the head-of-line blocking that a long prefill produces in front of short requests. It is the highest-return lever documented for this profile.

KV cache in FP8. It halves the memory budget per sequence, which is the limiting resource. Validate with evals.

--max-num-queued-reqs and --max-num-queued-tokens. New in 0.29. They are admission control in the engine, and they are the right mechanism to protect the SLO instead of letting the queue grow without limit.

max_num_batched_tokens downwards. High values improve time to first token; low values improve inter-token latency. In a regime where 91 % of the time is decode, inter-token latency is what is perceived. Chunked prefill is enabled by default whenever possible in the V1 engine.

Session affinity at the gateway. It is not an engine setting, but it is what holds up everything above. Without it, that 98 % hit rate turns into random distribution across replicas.

Two version warnings: Model Runner V2 becomes the default in 0.29 and V1 is deprecated with removal planned for 0.32; and starting with python -m vllm.entrypoints.openai.api_server is deprecated.

The full calculation, from start to finish

Case: twenty developers, a coding scaffold, a 30B-class model in the fleet, maximum context of 128K, KV in FP8.

1. Steady-state concurrency. $C = 20 \times 0.11 \times 3 \approx 6.6$ sequences.

2. KV budget per sequence. 0.1875 MiB per token in BF16, half in FP8, that is around 0.094 MiB. At 128K of context: around 12.0 GiB per sequence. For 6.6 sequences with margin, rounding to 8: 96 GiB of KV.

3. GPUs. The weights of a 30B in FP8 take up around 30 GB. On an 80 GB H100 with gpu_memory_utilization of 0.90 there are around 72 GB usable per card. With tensor parallelism of 2, the weights are split: around 15 GB per GPU, so around 57 GB of KV per GPU and around 114 GB in the replica. That covers the 96 GiB calculated, and TP=2 also adds memory bandwidth, which is the resource that dominates decode. Two H100s, one replica, TP=2. It is worth seeing the contrast: on a single H100 the full weights leave around 42 GB of KV, that is three and a half sequences, less than half of what is needed. With a second replica for availability and maintenance windows: four GPUs.

4. Requests per second at the gateway. 20 people, a mean session of 40.6 calls in 62.6 minutes, duty cycle included: of the order of 0.2 to 0.3 requests per second on average, with peaks of 2 to 3.

5. Gateway workers. The ceiling is not set by throughput but by token counting: four concurrent counts per worker, at 100 ms with long prompts. With peaks of 3 requests per second and six-figure prompts, two replicas of one worker each cover the case with margin for the rolling update, at 1 vCPU and 4 GiB each. Postgres will see 20 connections.

6. What needs switching on. Session affinity, admission control at the gateway and at the engine, --long-prefill-token-threshold, and Redis so that the limits are global.

The same calculation with the chat profile would have given one GPU and one worker. The distance is not in the safety margin, it is in the inputs.

A reference `config.yaml`

model_list:
 - model_name: agentes-30b
 litellm_params:
 model: hosted_vllm/qwen3-coder-30b
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 rpm: 600
 model_info:
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072

router_settings:
 routing_strategy: simple-shuffle
 # without this, the capacity calculation in this article is worthless
 optional_pre_call_checks: ["session_affinity"]
 num_retries: 2
 allowed_fails: 3
 cooldown_time: 30

general_settings:
 # admission control: below this the queue grows without limit
 max_in_flight_requests_per_worker: 24
 max_queued_requests_per_worker: 48
 admission_queue_timeout_seconds: 1.0
 # the spend queue discards at 64 MB; shorter interval, less backlog
 proxy_batch_write_at: 60
 enable_drain_endpoint: true
 drain_endpoint_token: os.environ/DRAIN_TOKEN
 database_connection_pool_limit: 10

litellm_settings:
 # 6000 s by default: a hung connection holds a slot for an hour and a half
 request_timeout: 600
 callbacks: ["langfuse_otel", "prometheus"]
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

Environment variables that go with it:

NUM_WORKERS=1
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS=8 # default 4; raise only if CPU is spare
LITELLM_TPM_TOKEN_RESERVATION_ENABLED=true
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS=30

And on the engine side:

vllm serve Qwen/Qwen3-Coder-30B \
 --tensor-parallel-size 2 \
 --max-model-len 131072 \
 --kv-cache-dtype fp8 \
 --long-prefill-token-threshold 512 \
 --max-num-queued-reqs 64 \
 --gpu-memory-utilization 0.90

Checklist

Measure three figures of your own over a week: calls per hour of human work and per profile, prompt length distribution, and the time series of prefix cache hit rate.
Calculate the KV per sequence with the model’s real geometry, taken from its config.json, not from a table.
Set --max-num-seqs from that budget, not from a chat heuristic.
Switch on session affinity before accepting the calculated number of GPUs.
Switch on admission control in both layers and check that the client respects retry-after.
Lower request_timeout from 6,000 to 600.
Raise proxy_batch_write_at to 60 and alert on the growth of the spend queue from outside, because there is no metric.
Leave the HPA on CPU at 60 %, never on memory, and assume the 300 seconds of the startupProbe when planning the margin.
Watch litellm_in_flight_requests and litellm_overhead_latency_metric; if the overhead rises with the size of the tool catalogue, the culprit is the token counter.
Go over the dashboards for the vLLM metric renames before upgrading.
Check that no routing policy moves a live session between model groups.
Redo the calculation when a new user profile arrives: the range across profiles is 50 to 1.

Traps and things that are not what they seem

The ratio of 275 to 1 does not mean the system is prefill-bound. The new tokens per turn are between 1.5 and 7.3 times the output. The time is between 91 % and 98.6 % in decode.

The aggregate prefix cache hit rate hides the problem. An aggregate 90 % may be 98 % intra-turn and 55 % at boundaries. You have to look at the series, not the number, and segment by request type.

vLLM’s hit rate counts the reprefill after an eviction as a hit. The counter exists internally and is not exposed, so the published number is optimistic under a memory-pressure regime.

Agents that fail cost more than those that succeed, up to 1.8 times more context. Sizing on the task that goes well leaves the calculation short.

The database pool is per process, not per pod. Four workers are forty connections.

The HPA on memory does not work with this piece, and not because of a configuration mistake: Prisma’s query engine does not return the memory.

The 300-second startupProbe turns any autoscaling into late reaction. For agentic peaks, it is better to over-provision than to trust the scaling.

Raising --max-num-seqs to “make use of the GPU” degrades perceived latency: two long requests in the batch raise time per token by 79.7 %.

The token counter has a limiter of four per worker and it appears in no sizing guide. With large tool catalogues and long contexts it is the first ceiling you hit.

The figures in this article come from three different scaffolds and are not averageable. Fifteen, forty-three and sixty-two turns per session are three systems, not three measurements of the same one.

Closing

The arithmetic of sizing has not changed. What has changed are the inputs, and now there are public measurements at scale instead of intuitions. Three figures sum up the change of regime: 68,000 tokens of median prompt, 87 % of calls started by the agent rather than by the person, and a time split that is decode for more than 90 % as long as the prefix cache hits.

From that comes an order of work that is not the usual one. First, session affinity, because without it the rest of the calculation describes a different system. Second, the KV budget with the model’s real geometry, which is what sets how many sequences fit. Third, the gateway, where the ceiling is not throughput but a token counter with four threads per worker.

And a warning about your own figures. The measured range across user profiles is 50 to 1. Any platform that does not segment its measurement by profile is averaging two different populations and producing a number that describes neither of them.

Sources

Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale, arXiv 2608.00101, 30 de julio de 2026.
Agentic AI Workload Characteristics, arXiv 2605.26297, 25 de mayo de 2026.
From LLM Inference to Agentic Workloads, arXiv 2608.15127, 15 de agosto de 2026.
vLLM x AgentX: serving agentic workloads, 8 de septiembre de 2026.
vLLM metrics design y optimization guide, consultados el 12 de septiembre de 2026.
LiteLLM benchmarks y production setup, consultados el 12 de septiembre de 2026.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Autoescalado con KEDA en vLLM production-stack, consultado el 12 de septiembre de 2026.

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

Fri, 11 Sep 2026 01:45:00 +0200

Sixth post in the operational track on the control layer. The previous five treat the gateway as the piece the tokens pass through: the pair with Langfuse, day 2, virtual keys, humans and agents and prefix routing. This one is about the other door. Verified against LiteLLM 1.102.0, commit of 10 September 2026.

TL;DR

The MCP gateway is not a module, it is a complete second surface. A native MCP endpoint with four route spellings, a proxy mode with three fixed tools, four REST routes, some thirty-five administration routes and a long set of OAuth discovery endpoints. It still lives under _experimental and it is nearly 20,000 lines.

I correct what I said in the previous post: access is NOT open by default. In 1.102.0, a key with no declared permissions and no team resolves zero servers. What does open things up is a per-server flag, allow_all_keys, and that flag overrides the key’s explicit scope unless you enable a setting that is off by default.

Permissions are enforced on execution, not just on the listing. Calling a tool that did not appear in tools/list returns 403. There are three chained checks before the call, including one on the allowed arguments. The same predicate governs listing and calling, by declared design.

The hierarchy is built out of intersections, with two escapes and a fail-open. The key’s access groups are additive on top of the key and team ceiling. The organisation ceiling replaces rather than intersects when there is nothing below it. And on an indeterminate error while resolving tool permissions, the code returns “unrestricted”. It is a deliberate decision and it has to be declared in the risk analysis.

Per-tool cost is worth zero and that is why its metric disappears. There are exactly two Prometheus metrics for MCP, and the spend one is only incremented if the cost is greater than zero. With the default configuration, that series never exists. There is no listing metric, no latency metric, no error metric, no server health metric.

The cost that does matter is not attributed to MCP anywhere. It is the tokens of the tool definitions, resent on every turn inside the prompt_tokens of the model call. Five typical MCP servers add up to 58 tools and some 55,000 tokens. The tool’s spend row, with its zero cost, has nothing to do with that.

And the gateway’s semantic filter does not act on the MCP door. It only applies when it is LiteLLM calling the model. A client connected to /mcp receives the whole catalogue, filter enabled or not. And even if it did act, filtering per request rewrites the cached prefix of the prompt, which is the opposite of what Anthropic and OpenAI do with their own tool searches.

The good observability ships switched off. Without LITELLM_OTEL_V2, a tool call comes out as a generic span with the model set to MCP: name. With it enabled there is a dedicated span and correct attributes, and the arguments and the result stay hidden unless you opt in. And in neither case is trace context propagated towards the upstream MCP server: the trace is cut at the gateway.

Tool arguments are written, and they are written in the clear. They go in full into the metadata column of the spend table, without passing through the switch that governs prompt storage nor through the redaction function, which only knows about messages and response. Whoever enabled redaction for GDPR probably believes they are covered.

The 1.83 branch concentrated four serious vulnerabilities, and one of them belongs to the MCP gateway itself. The endpoint for testing a connection before saving accepted the stdio transport’s command, args and environment in the request body. A low-privilege key got command execution on the proxy host.

And the specification moved underneath. The revision in force since July 2026 removed protocol sessions, the initialisation handshake and the session header. LiteLLM 1.102.0 advertises the June 2025 revision and its version enumeration does not even contemplate the two later ones.

You are here: the door nobody chose

The gateway was chosen for the reasons in the switchboard post: licence, fit inside the cluster, maturity. What was evaluated was the inference traffic.

Later, somebody connects an agent, the agent needs tools, and the same process that is already authenticated and deployed turns out to speak MCP too. The decision to turn it into the tool door is almost never taken: it is inherited. And that door has neither the same controls, nor the same observability, nor the same cost model.

When MCP grows covered the identity of your own MCP servers. MCP from the inside covered the protocol and its instrumentation. This post covers the piece in between.

The analogy: the switchboard that also hands out keys

An office building’s telephone switchboard. Its original job is to route calls: it decides who can call outside, how much each department spends, and it leaves a record of every conference call.

Over time, the switchboard operator is also given the key cabinet. They are already at reception, they already know who everybody is, it looks like the natural place. And from that moment there are two services at the same desk with very different rules.

Calls are priced by the minute and show up on the bill. Keys have no tariff, so on the department’s bill they come out at zero, and somebody might conclude that the key cabinet costs nothing. It does cost: it costs the operator’s time, it costs having to read the whole key catalogue every time somebody asks what is available, and it costs that nobody checks that the label on a key still says what it said the day its loan was authorised.

The analogy ends at the interesting spot. When somebody takes a key, the record says which key it was and what they said they wanted it for. That record is simultaneously the evidence of the action and a piece of data that perhaps should not be stored in the clear.

The real routes

The first thing that surprises you when you look at the code is how many spellings get you to the same place.

The native endpoint

The MCP sub-application is mounted at /mcp and inside it has four mounts: the root as a wildcard, /mcp, /{name}/mcp and /sse.

And there is also a hand-declared route for bare /mcp on the main application, with a comment that explains why: the mount cannot match its own bare prefix, and the resulting 307 redirect breaks MCP clients sitting behind a proxy that terminates TLS. It is the kind of detail you discover in production on a Friday.

/mcp/sse holds a surprise. It does not use the classic SSE transport: it uses the same HTTP session manager in stateless mode. The SseServerTransport object is constructed and is not referenced on any other line of the file. It is dead code, and the /mcp/sse/messages endpoint that this object advertises does not exist as a working route.

The /{name}/mcp route resolves in a documented order: server alias, comma-separated list with a maximum of 16 elements, tool set, access group. If nothing matches, 404.

And /mcp/proxy exposes a fixed surface of three tools. It is the progressive disclosure pattern, and it will come up again further down because it is the most important mitigation in this post.

The REST routes

Four, under the /mcp-rest prefix:

Method	Route	Authentication
GET	`/mcp-rest/tools/list`	Virtual key
POST	`/mcp-rest/tools/call`	Virtual key
POST	`/mcp-rest/test/connection`	Key plus administrator role
POST	`/mcp-rest/test/tools/list`	Key plus administrator role

The variants without -rest that appear in the internal types have no router registering them. The /mcp/* wildcard absorbs them as JSON-RPC.

The last two are the stars of the security section.

Administration and discovery

Some thirty-five routes under /v1/mcp, almost all of them with virtual key authentication. Three details worth knowing:

GET /v1/mcp/registry.json carries no authentication dependency. It returns 404 if the public registry is not enabled, which is the real protection.

POST /v1/mcp/server requires the proxy administrator role. POST /v1/mcp/server/register does not: it is the proposal path, which requires a key with a team, explicitly forbids the stdio transport and leaves the entry pending review.

And on top of that there is a long set of OAuth .well-known endpoints, with dynamic client registration, authorisation, token, revocation and introspection, plus their per-server variants.

Configuration and transport

The mcp_servers schema in config.yaml has fifty-odd fields, all of them optional. A warning for anyone coming from earlier versions: spec_version no longer exists in 1.102.0.

The default transport when omitted is http, that is, streamable HTTP. The other two values are sse and stdio. The string streamable_http is not valid in the configuration file, although it is accepted as a synonym when importing connector files.

Tool prefixing uses the format {server}{separator}{tool} with - as the default separator, configurable through an environment variable read at import time. The prefix chosen is the alias, or the name, or the identifier, in that order. There is an optional short mode that derives three characters from the hash of the server identifier and resolves collisions by rehashing.

Access control, and the correction to the previous post

In the humans and agents post I closed the list of traps with this one: “access to MCP servers is open by default if no level defines a list”. Verified in 1.102.0, that is false.

The function that resolves the allowed servers does the opposite. If the team declares nothing, it inherits from the key; if the key declares nothing either, the base set is empty and the scope is marked as unbounded with zero servers. There is no branch that expands to “the whole registry” other than the administration role one.

What does open things up, and this is the important part, is something else.

`allow_all_keys`

It is a per-server flag, in configuration, in the API and in the database, defaulting to false. When set to true, that server joins the set of open servers and any key can see it.

The detail to know: by default it overrides the key’s explicit scope. A key that declares access only to server A will also see server B if B has the flag. The key’s scope is only respected if you enable general_settings.mcp_allow_all_keys_respects_mcp_scope, which is off by default.

There is a second opening case: anonymous subjects, with no user identifier, role or key, additionally receive the servers configured with authentication delegation towards the upstream and the pass-through ones. It is consistent with the design of those two modes, and it is a surface you have to know about.

`require_key_mcp_access_defined`

It lives in general_settings, defaults to false, and needs no licence. What it inverts is only the empty-key inheritance towards the team: with the flag on, a key with no list of its own gets an empty set instead of inheriting the team’s.

It does not affect keys that do declare servers, which still intersect with the team. And it does not close off the access group grants, which are additive and are the escape route you have to audit separately.

There is a sibling, require_end_user_mcp_access_defined, with the same shape and applied to the end user.

The hierarchy

The function’s docstring is normative and says that all the rules are intersections. In detail:

Key and team: intersection when both declare. Inheritance in one direction when one is empty.
Key access groups: additive union on top of the previous result.
End user: intersection if it declares anything.
Agent: intersection.
Internal user: ceiling that only narrows.
Organisation: ceiling that intersects if there are restrictions below, and that replaces by becoming the ceiling when there are none.

For subjects with no key, admitted by SSO or by the gateway, the model changes to a per-source union instead of an intersection.

The fail-open

This is the point you have to take to the risk analysis and not leave in a technical note.

On an indeterminate failure while resolving the tool permissions of a key or a JWT, the code returns “unrestricted”, that is, all tools allowed. The organisation ceiling is skipped in the same situation. For subjects with no key the behaviour is the opposite, fail-closed.

There is a well-resolved intermediate case: when a permission is named but cannot be read, a specific error is raised and it is denied in both cases.

The doctrine is documented in the class docstring, which is appreciated. And it remains a behaviour that, in a system under the ENS, you have to declare in writing instead of discovering it during an audit.

What is done well

Permissions are enforced on the call, not just on the listing. The execution path chains three checks before contacting the server: allowed or forbidden tools, per key and team tool permission, and validation of the arguments against the allowed ones. All three return 403.

Listing and calling share the same predicate, and the docstring declares it as an explicit invariant. A client that calls a tool directly when it did not appear in its listing receives 403, it does not execute. The same over REST and over the responses API route.

The x-mcp-servers and x-mcp-access-groups headers only narrow, and they fail closed if they do not resolve.

Header forwarding towards the upstream is an explicit per-server allowlist, with a centralised decision about when to strip the caller’s Authorization, and a protection against cross-forwarding: in a multi-server listing, the global header is withheld if more than one server would consume it. That is RFC 9700 applied.

The bill that never shows up

Here is, in my opinion, the most important thing in this post, and it has nothing to do with security.

Every tool the gateway aggregates takes up space in the model’s context. Name, description and parameter schema travel in the request’s tools block, on every turn, and they are paid for as input tokens of the real model.

The figures published by Anthropic in November 2025 are the best public reference there is. Five connected MCP servers add up to 58 tools and some 55,000 tokens. The breakdown: GitHub, 35 tools and some 26,000 tokens. Slack, 11 tools and some 21,000. Sentry, five and some 3,000. Grafana, five and some 3,000. Splunk, two and some 2,000.

With tool search enabled, that set drops from some 72,000 to some 8,700 tokens, and the usable context goes from 122,800 to 191,300.

The second cost of a large catalogue is not economic and is not billed: it is that the model chooses worse. It has its own section further down, because of the two it is the one that decides whether the agent is any good.

How to measure it with what is already there

The gateway stores enough data, even if it does not cross-reference it for you.

In the listing’s spend row, with call type list_mcp_tools, it enriches the metadata with allowed_server_count, tool_count_total, per_server_tool_counts and per_server_list_outcomes.

In the model call rows, the proxy_server_request column keeps the request body with the tools block inside it.

The practical method: pair the listing rows with the LLM rows of the same session by session_id, read the total number of tools from the first, and pass the tools array of the second through litellm.token_counter(model=..., tools=...), which accepts that argument. Multiply by the number of turns, because the block is resent on all of them.

For the relative cost, a quicker comparison: mean prompt_tokens of keys with MCP enabled against those without, same model and same period.

Tool selection

The previous section treats the catalogue as a bill. This one treats it as what determines whether the agent gets it right. This is ground where the talk runs well ahead of the measurement, so it is worth separating what has been measured from what gets repeated.

How many tools a model can take

Anthropic’s official figure is that Claude’s ability to pick the right tool degrades when going from 30 to 50 available tools. The same documentation gives the inverse criterion, which is the more useful one: the full catalogue with nothing else is the right option with fewer than ten tools, when all of them are used on every request, or when the definitions add up to fewer than a hundred tokens.

There is also an administrative ceiling before the semantic one. VS Code with Copilot cuts off at 128 tools per request and returns an error when you exceed it.

The full curve gets cited far more than it has been measured. The soundest primary data I found is from May 2025: varying the number of candidates from 1 to 11,100 over a registry of more than 4,400 servers, below some 30 tools the success rate is above 90 %, between 31 and 70 intermittent failures appear, and past the hundred mark the degradation becomes dominated by retrieval. It is a qualitative heat map, not a numeric table.

The figure that circulates most (78 % with ten tools, 40 % with a hundred, 13.6 % with seven hundred) comes from a conference talk with no paper, no repository and no published methodology. Do not take it to a client presentation.

And there is a warning that affects everything else. An audit of 496 tasks from four tool calling benchmarks found 18.5 % misalignment between the label and reality. In the BFCL v4 subset, 80 % of the failures assigned to the agent came from fragile state comparisons. In one of the benchmarks with an automatic judge, 23 identical re-runs gave a range from 57.9 % to 76.8 %, almost 19 points of spread, enough to reorder the ranking. Any absolute figure in this section deserves a generous margin of error.

How it fails

The work with the cleanest taxonomy evaluates 36 servers, 220 tools and 1,000 tasks against twenty frontier models, with an average of 15.2 tools exposed per task of which 4.1 are relevant.

Failure mode	Share
Not using a tool when one was needed	10.5 %
Choosing the wrong tool	9.0 %
Malformed parameters	6.9 %
Not recovering from an error	10.3 %
Cognitive failures, not tool failures	63.3 %

The interesting part is in the per-model breakdown: the dominant failure mode changes with the model, not with the size of the catalogue. In one of them, 40.1 % of the failures are simply not calling any tool. In others the malformed parameter dominates. In another, the wrong tool. There is no canonical failure to attack.

On tools that do the same thing, which is the real case of a gateway with fifteen servers, there is a good measurement. A study with ten groups of five functionally equivalent tools and a thousand query pairs puts the selection bias between 0.3 and 0.4, that is, you would have to redistribute between 30 and 40 % of the probability mass for equivalent tools to be chosen equally. The same work shows that a biased continued training takes the selection of a specific endpoint from 0.6 % to 12.8 %.

That study contributes another figure that reorders priorities: shuffling a tool’s description moves the selection distribution substantially, whereas changing only the name has minimal and inconsistent effects. Applied to the gateway’s server-tool prefixing: it costs tokens and probably does not change which tool the model goes to. There is no study isolating the effect of name spacing, and it is a real gap in the literature.

If the description is what carries weight, the state of the estate is the problem. An audit of 856 tools from 103 MCP servers found undeclared limitations in 89.8 %, missing usage guidance in 89.3 % and opaque parameters in 84.3 %. Only 2.9 % were free of problems across all their components.

The optimum is not a number

The May 2026 work I cited above proposes a chance-corrected metric and uses it to size the list adaptively. Over 370 functions it reaches 90.3 % coverage while seeing an average of 7.4 tools, against 90.8 % seeing a fixed 50.

But the same paper publishes its own counterexample, and it is honest to cite it: over another benchmark of 3,251 tools, the fixed K of 5 wins in aggregate, 64.7 % against 61.9 %. The adaptive advantage concentrates in the hard queries, where the right tool sits between sixth and twentieth by similarity: there it finds 16.7 % of the cases where the fixed K finds 0 %.

The March 2026 work over 121 tools puts the plateau at K=3, with 97.1 % that does not improve going up to K=5. With the caveat that it measures retrieval against labels, not model accuracy.

And a July 2026 work gives the formulation that seems to me the right one: ranking by score is inconsistent with optimal acquisition when costs are heterogeneous, so the optimal K is a function of cost, not a constant. Trimming from 7 to 4.4 exposed tools while keeping success is possible; fixing a universal number is not.

Filtering can make the result worse

This is the part that appears in no product blog. A June 2026 benchmark with three registry sizes (25, 100 and 250 tools), six filtering methods, seven models and 26,460 runs:

Strategy	Success	Visible tools	Tokens
Expose the whole catalogue	32.1 %	125.00	56,062
Keyword, top-5	22.1 %	4.80	3,200
Keyword, top-10	22.4 %	9.54	5,356
State-aware	24.0 %	25.95	13,522
Full causal path	24.0 %	26.24	13,697

Four of the five filtering strategies perform worse than not filtering. The fifth, which does win comfortably, exposes 0.99 tools per step and requires hand-written precondition and effect contracts for every tool, plus a search of the causal path to the goal state. It is a planner with the goal given, not a retrieval filter, and it is not comparable with anything a gateway offers.

There is more evidence in the same direction. A work over a corpus of 43,000 tools measured that the agent’s completion rate drops when using retrieved sets against hand-annotated sets: the filter introduces its own recall ceiling. And top-K breaks the tasks that need several tools: on a benchmark of 16,464 endpoints, a fine-tuned retriever reaches a Recall@3 of 68.6 % but only 39.7 % of the queries have their complete tool set within the first three. Almost 29 points of gap between “the one I needed is there” and “all the ones I needed are there”.

Filtering is not free on security either. A March 2026 work attacks the retrieval layer by injecting adversarial tools that saturate the top-k: with injection rates of 1.2 % it manages to dominate the top-k between 91 % and 97 % of the time. It is an attack surface that exposing the whole catalogue does not have.

The most elegant counterpoint comes from Anthropic’s own numbers. With the same catalogue of more than fifty tools, the gain from tool search drops from 25 points on one model to 8.6 on the next, because the unfiltered baseline rises from 49 % to 79.5 %. The margin left by filtering shrank to a third in one generation of models. Any architecture decision taken here has a short shelf life.

The prompt cache trap

Here is what makes a homemade top-K filter a bad idea even if it worked.

Tool definitions do not go just anywhere in the prompt: they go at the beginning. Anthropic documents the order of the cacheable prefix as tools, then system, then messages, with a hierarchy where each level rests on the previous one. And the invalidation rule is explicit: modifying tool definitions, be it names, descriptions or parameters, invalidates the entire cache, and a change at one level invalidates that level and all subsequent ones.

OpenAI says the same and adds a detail: the prefix must remain unchanged, and order counts. The same set of tools served in a different order is a cache miss.

With vLLM serving an open model there is no tool block of its own. The tools parameter is serialised by the chat template inside the first system message, behind the system prompt text. For automatic prefix caching the effect is identical, because it is still near the absolute start of the prompt.

The consequence is the one that matters. A filter that returns a different set of tools on the next turn does not just invalidate the tool block: it invalidates everything behind it, which is the system prompt and the whole conversation, which is exactly the part that grows with the turns. At list prices, a prefix token goes from costing 0.1 times when read from cache to 1.25 times when written. An agent with sixty thousand tokens of stable prefix goes from some six thousand equivalent tokens per turn to some seventy-five thousand, plus the prefill that gets paid again in latency.

The two big providers have separately arrived at the same solution, and it is the opposite of filtering in the prefix. Anthropic excludes deferred tools from the prefix and, when the model discovers one through search, injects it inline inside the conversation: “the prefix stays intact, so prompt caching is preserved”. OpenAI: “all tools are loaded at the end of the model’s context window (…) this allows the cache to be preserved from one request to the next”. Anthropic’s operational recommendation is the stable core pattern: leave the three to five most used tools undeferred.

The closest academic work, from January 2026, states the problem without measuring it and recommends exactly that, keeping a fixed set of reusable general-purpose functions and implementing dynamic capability through code generation.

The July 2026 MCP specification picks up both halves of the matter. Servers “should return the tools from tools/list in a deterministic order to allow caching in the client and improve the hit rate of the model’s prompt cache”. And it adds mandatory ttlMs and cacheScope in listing results, so that the client can freeze the catalogue for a known window instead of re-querying it and propagating any variation to the prompt.

Nobody has publicly measured the cost in euros of this effect. The only attempt I found is a self-published audit of ten thousand turns that puts it at 2.4 % to 3.5 % of cache misses, with non-reproducible methodology. It is a gap in the literature, and the order of magnitude in the previous paragraph comes from applying list prices, not from a measurement.

And now, LiteLLM

With all of the above in mind, the gateway’s levers read differently.

The semantic filter does not act on the MCP endpoint. It is registered as a generic LiteLLM callback, and its only dispatcher is the proxy’s precall hook, which additionally discards anything that is not completion, acompletion or aresponses. The tools/list path does not invoke that hook at any point; there is a comment in the code that admits it in passing, when explaining why the listing’s JWT signature is done separately. An external MCP client connected to /mcp receives the whole catalogue, filter enabled or not.

Lever	Serves external MCP clients	Breaks the prompt cache
Per-server `allowed_tools`	Yes	No, it is static
Name and description overrides	Yes	No, it is static
Selection by route or header	Yes	No, if it is stable per agent
`/mcp/proxy` and virtual tools	Yes	No, the surface is fixed
Semantic filter	No	Yes, it rewrites the block per request

When it does act, the filter has details you need to know before enabling it. It embeds only each tool’s description, with the name as a fallback if it is missing; the parameter schema does not go in. Of the query it embeds only the last user message: on turn twelve of an agent conversation, that might be “yes, go ahead”. The tool embeddings are cached and indexed at start-up; the query’s is computed on every request, with a synchronous call inside a coroutine.

Its failure modes are almost all open, which is the reasonable thing: if the embedding model does not respond, if nothing passes the threshold, if the names do not match the catalogue, it returns all the tools. The exception is context overflow, which fails closed with a 400, and which if it happens while building the index during start-up is memoised and makes all subsequent requests fail until you restart. The defaults are text-embedding-3-small, top_k of 10 and a threshold of 0.3, under litellm_settings.mcp_semantic_tool_filter.

/mcp/proxy does serve external clients, and that is where its real value lies. It replaces the catalogue with three tools with opaque identifiers. How search_tools searches deserves a line, because it is not what you would assume: by default it is neither a regular expression nor a semantic match, it is a keyword counter, how many tokens of the query appear as a substring in the concatenated name and description. It only moves to embeddings if you configure a model in mcp_tool_search. The result cap is hard-coded at five in this mode, and the client cannot ask for more. In exchange, call_tool goes through the same permission checks as a normal call plus a validation of the argument schema that the normal route does not do at that point.

There is a related variant: with mcp_tool_search_enabled in the key’s permissions, the catalogue is replaced by four virtual tools, and there the search one does expose top_k to the client.

The catalogue order is not deterministic, and that breaks the client’s cache without anybody filtering anything. The list of allowed servers comes from iterating a Python set of strings, so with the random hash seed the order differs between processes, between uvicorn workers and between restarts. There is no sorting anywhere in the listing path, and there is no tool cache: every tools/list queries the upstreams live, so the order within each server also depends on what the upstream returns that time. The gateway does not offer today the guarantee that the July specification recommends. The fix, sorting the catalogue before serving it, fits on one line.

Nor is there any pagination towards the client. The catalogue is returned whole, always.

And the lever almost nobody uses is the static one. Every MCP server accepts tool_name_to_display_name and tool_name_to_description, and those overrides really do replace what is sent to the client in tools/list, not just what the administrator sees in the interface. The tools/call routing translates from the new name to the real one before any permission check. With that you can trim a four-hundred-token description written by a third party, rename an ambiguous tool so it does not compete with a similar one, and do it without touching the prompt cache, because it is static and holds for all turns.

It is also the rug pull mitigation left pending from the security section: if the description the model sees is the one the operator wrote, whatever the upstream server changes stops reaching the context. Two limits. They are configured through the database or the management API, not through config.yaml. And the reverse name translation is by exact equality and is not disambiguated between servers, so two servers with the same display name route unstably, made worse by the non-deterministic order of the previous paragraph.

Alongside that, per-server allowed_tools remains the blunt and effective instrument: it filters the listing the client sees, reduces tokens, and is static.

The ladder, in order

Trim. Per-server allowed_tools, plus a per-key allowlist. Cheap, static, does not break the cache and reduces the permission surface at the same time.
Rewrite the descriptions with the overrides. That is where the evidence is, because the description carries weight and the name almost none, and along the way it pins the contract against the rug pull.
Split by route. Each agent to /{server}/mcp or to a tool set, never to the aggregated endpoint. Stable per agent, so the cache survives.
Sort the catalogue before serving it if you control the deployment, while the gateway does not do it.
/mcp/proxy when the catalogue goes past thirty or fifty tools. It is progressive disclosure and it works with external clients, which is more than you can say for the semantic filter.
The semantic filter, only when the one calling the model is LiteLLM. With a generous top_k, knowing that it embeds only the last user message and that context overflow is a closed and sticky failure.
What not to do: a home-grown top-K filter that rewrites the tools array on every turn. It costs the whole prefix and, according to four of the five measured strategies, probably makes selection worse.

The declared cost, which is worth zero

The calculation has four precedence steps: cost set by a post-call hook, per-tool cost in tool_name_to_cost_per_query, server default cost, and 0.0.

Without configuring mcp_server_cost_info, all MCP traffic is accounted at zero and consumes neither key nor team budget.

There are exactly two Prometheus metrics for MCP:

litellm_mcp_tool_calls_total
litellm_mcp_tool_call_spend_metric

Both with the same eight labels: tool name, server name, key hash, key alias, team, team alias, user and end user.

And the second is only incremented if the cost is greater than zero. With the default configuration, that time series never appears in Prometheus. It is not that it is worth zero: it is that it does not exist.

Nor is there a tools/list metric, or MCP latency, or MCP errors, or upstream server health. For that you are left with the generic request metrics.

In the reports, /spend/calculate has no MCP branch: it accepts a model and messages, or a completion response, and it always calls the LLM cost calculation. The per-team report does include MCP calls, grouped by the model column, which for these rows is worth MCP: tool_name. That is: every tool appears as if it were a model, with zero input and output tokens. Only the sessions endpoint separates the two call types explicitly.

Observability, and the trace that gets cut

There are two OpenTelemetry stacks in LiteLLM and which one runs depends on an environment variable.

Without LITELLM_OTEL_V2=true, the old logger runs, and it has no MCP handling: the only appearance of MCP in the whole file is the metadata key in a list. The tool call comes out as a generic litellm_request span with the model set to MCP: name.

With v2 enabled there is a dedicated span, with a dedicated role, client kind and a parent in the proxy’s request span. The span name is tools/call tool-name. The attributes are correct and follow the conventions:

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`mcp.method.name`	`tools/call`
`mcp.session.id`	MCP session identifier
`gen_ai.tool.name`	Tool name
`gen_ai.tool.call.arguments`	Only with content capture enabled
`gen_ai.tool.call.result`	Only with content capture enabled
`server.address`, `server.port`	From the upstream URL, redacted
`litellm.cost.total`	The cost that is probably worth zero

Content capture is off by default. Without it you see neither the arguments nor the result in the trace.

A note for anyone working with semantic conventions: mcp.tool.name does not exist. The MCP registry has exactly four attributes (mcp.method.name, mcp.session.id, mcp.protocol.version and mcp.resource.uri), all four in development, and the tool name goes in gen_ai.tool.name. LiteLLM gets it right. And the GenAI conventions were extracted into their own repository which as of today has no release at all, so there is no version number to cite.

In Langfuse, the vendor mapper returns an empty dictionary for any span that is not an LLM call. An MCP call appears as a raw span with its attributes, with no native input, output or model fields.

The trace is cut at the gateway

Incoming context is read. The traceparent the client puts in params._meta, following the specification’s propagation proposal, is extracted and turned into a link on the span, never into its parent. The reason is documented in the code: parenting to the remote trace would leave the span hanging off a trace whose root never reaches the backend. The client’s baggage is discarded on purpose to avoid forging identity attributes.

Outgoing context is not propagated. There is no propagator injection call anywhere in the tree. The headers that go out towards the upstream MCP server are built with authentication and the authorised extra headers, and nothing else.

The operational consequence: there is no end-to-end trace. If the upstream MCP server is instrumented, its spans live in another trace and there is no way to correlate them automatically.

Auditing and the conflict with privacy

A tool call leaves a normal spend row, with call type call_mcp_tool, spend equal to the cost per query, model MCP: name, zero tokens in the three token columns, a dedicated mcp_namespaced_tool_name column, and the usual key, team, user and session fields.

And in the metadata column, in JSON, the complete structure of the call: name, arguments, result, server, namespaced name, MCP session identifier, authentication mode and server resource with the URL redacted.

The problem

That structure is written into the metadata column without passing through the privacy switch. The assignment is direct and is not conditioned by the function that governs whether prompts and responses are stored, which does control the messages and response columns.

And the redaction function does not cover it either: it only touches messages and response. A search for MCP in the redaction file returns nothing.

The practical result: an operator who enables turn_off_message_logging for GDPR keeps sending the full arguments of every tool to every logging destination they have configured. To the SIEM, to Langfuse, to S3.

If an argument carries a national ID number, an address, a case number or a medical record, it has been exported.

There is an aggravating factor that does not depend on LiteLLM. The revision of the specification in force introduces an extension that allows mirroring tool arguments into HTTP headers visible to load balancers, proxies and web application firewalls. The specification recommends not marking sensitive parameters that way, and it is a recommendation, and the upstream server decides.

The fit with the ENS

The conflict is real and has a familiar shape: activity logging (op.exp.8) requires the call to be traced and non-repudiable; protection of personal data (mp.info.1) requires that log not to become a secondary repository of personal data with a different purpose. The tool argument is simultaneously the evidence of the action and the data. Redacting it destroys non-repudiation; storing it in the clear creates the repository.

The defensible way out you can build today: a hash of the argument in the audit log, the argument in the clear with encryption (mp.info.3), separate retention and its own access control. The hash sustains non-repudiation and the clear text is only opened under procedure.

There are two other measures this material touches directly. Change management (op.exp.5) because registering an MCP server and, above all, changing the description of an already approved tool are changes to the security configuration. And maintenance and updates (op.exp.4) because, as you can see in the next section, an old version of the gateway is a non-compliance with a name and a number.

Two honesty warnings. The codes follow the numbering of Annex II of RD 311/2022 and you have to check them against the text in force before citing them in a compliance document. And on NIS2: as of the date of this post the directive is still not transposed in Spain, with the draft bill on cybersecurity coordination and governance going through the process and a reasoned opinion from the Commission over the delay. What is enforceable today in the public sector and its suppliers is the ENS.

What is not logged

An administrative gap: registering, modifying and deleting MCP servers generates no audit log. There are three TODO comments in the management code that say so, one of them with an if is_audit_logging_enabled(): pass. In the same deletion block there are two more TODOs about not cleaning up orphaned permissions on keys and teams.

Security

The vulnerability you need to know about

CVE-2026-42271, published on 8 May 2026. Affects LiteLLM from 1.74.2 up to versions prior to 1.83.7. CVSS 3.1 of 8.8.

The POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list endpoints, which exist to preview an MCP server before saving it, accepted the stdio transport’s command, args and env in the request body. A low-privilege key executed arbitrary commands on the proxy host. It also affects Red Hat OpenShift AI on several branches.

It is the story of this post condensed into one line: the gateway you put in to contain MCP was the hole.

It does not come alone. The 1.83 branch concentrates three more serious ones:

CVE	Component	CVSS	Fixed in
`CVE-2026-42208`	SQL injection in key validation, with no prior authentication	9.8	1.83.7
`CVE-2026-35030`	OIDC userinfo cache keyed on `token[:20]`, collision and impersonation	9.1	1.83.0
`CVE-2026-40217`	Remote execution through bytecode rewriting in the guardrails test endpoint	8.8	no version recorded

Plus two privilege escalations fixed in 1.83.10 and 1.83.14.

The operational conclusion is short: a deployment below 1.83.14 is exposed to remote execution or to an unauthenticated SQL injection. The current branch is 1.102.0.

And a note for context: the whole MCP ecosystem has the same pattern. The unauthenticated remote execution in MCP Inspector, the command injection in mcp-remote when connecting to an untrusted server, DNS rebinding disabled by default in the Python and TypeScript SDKs until late 2025, the directory escapes in the reference servers. Connecting to an untrusted MCP server compromises the client, not just the other way round.

The rug pull nobody covers

Documented since April 2025: a server can change a tool’s description after the client has approved it. The description is what the model reads to decide when and how to use it, so changing it is changing the behaviour without asking for permission again.

The specification in force does not address it. It defines list change notification and cache freshness fields, which are recency mechanisms, not integrity ones. The closest normative warning is that clients must treat tool annotations as untrusted unless they come from trusted servers.

LiteLLM does not cover it either. The only “pinning” in the code is of the configuration’s server identifier, so that it does not change when you edit the YAML. There is no hash or comparison of the description or the input schema between successive listings. A search for related terms in the MCP module: nothing functional.

The control exists outside the gateway. mcp-scan, from the same researchers who documented the attack, implements description hash pinning precisely to detect this.

In ENS terms it is an unauthorised change to the security configuration that today goes undetected. And the home-made fix fits in a cron job: store the hash of the description and schema of every approved tool, compare against tools/list periodically, alert on the difference.

Three more holes in the code

No URL validation towards the upstream. A new server’s url field has no validator. The SSRF protection utilities exist in the tree and are used in OAuth discovery and in downloading OpenAPI specifications, but not in the egress of tools/call. A server pointing at a private range or at the cloud metadata IP is accepted as is.

The stdio transport is local execution on the proxy host. There is a command allowlist: npx, uvx, python, python3, node, docker, deno, extendable through an environment variable. The code’s own comment admits the residual risk: the permitted runtimes can execute code through arguments. With docker and npx in there, it is de facto arbitrary execution for any proxy administrator.

Two server modes skip the gateway’s authentication. Pass-through and authentication delegation to the upstream completely omit virtual key validation. They are well hardened (fail-closed if the target is mixed or unresolvable, strict comparison against truthy values, explicit blocking of the client credentials flow so as not to lend out the proxy’s service account), and the result is still that those servers have no gateway access control.

The limit that truncates in silence

The tools/list pagination stops and returns what it has accumulated in three cases: repeated cursor, one thousand pages, or expiry of a global deadline. The docstring declares it: a slow or faulty upstream produces a partial catalogue instead of an error.

The first two log a warning. The third, the deadline, exits without logging anything at all. A slow server makes part of its tools disappear from the catalogue with no trace left behind.

Correction on the slot leak

In the previous post I took issue 34534 at face value, according to which every MCP tool call acquires a max_parallel_requests slot and does not release it. Tracing the 1.102.0 code, that diagnosis no longer holds: the release path exists, on both success and failure, and the limiter is registered as a litellm callback and not just as a proxy hook.

With one honest caveat: I found no regression test covering the full cycle for MCP, and the definitive confirmation is a real proxy with max_parallel_requests: 2 and five MCP calls in a row. If anybody reproduces it today, I would like to know.

The specification moved underneath

The MCP revision in force is 2026-07-28, published on 28 July 2026, and in between there was another one, 2025-11-25. The changes in the latest one are not cosmetic:

Protocol sessions and the session header are removed. Cross-request state becomes an explicit identifier in the tool arguments.
The initialisation handshake is removed. Every request carries version and capabilities in _meta. MCP becomes stateless.
server/discover appears as a mandatory call.
Elicitation changes shape: out go server-initiated requests, in comes a multi-round pattern where the server returns an “input required” result and the client retries with the answers.
ping, logging/setLevel and the roots change notification are removed. SSE stream resumption is removed.
Standard HTTP headers are mandatory on POST, which lets a gateway or a web application firewall decide without parsing the JSON.
On authorisation: mandatory issuer validation in the client, and dynamic client registration is deprecated in favour of client identifier metadata documents.
Roots, sampling and logging are deprecated, with a minimum window of twelve months.

LiteLLM 1.102.0 advertises 2025-06-18. Its version enumeration has three values and none of them is later than that date. The literal is sent in the initialize against the upstream.

One nuance that is not entirely bad: the version the gateway advertises to its clients is not hard-coded in LiteLLM, the SDK negotiates it. And there is partial implementation of the intermediate revision, because the elicitation and sampling handlers reference it in their docstrings. That is: partial implementation of one revision while advertising an earlier one.

In practical interoperability it does not hurt yet, because the client ecosystem is still mostly on the 2025 revisions. It hurts in planning: July’s stateless redesign invalidates the gateway’s session model, and that is work ahead.

Two ecosystem notes for anyone choosing. There are clients that speak remote streamable HTTP natively (Claude and Claude Code, ChatGPT with OAuth and dynamic registration, VS Code with Copilot, Cursor, Zed, Kiro) and others that still demand stdio or a local bridge with mcp-remote, among them Windsurf, Cline and Continue.

Tool execution from the proxy itself

Besides serving MCP outwards, the proxy can consume it itself. It is enabled with an mcp block in the body, with server_url equal to the litellm_proxy sentinel, and it works both on /v1/responses and on /chat/completions.

The flow: it separates the gateway’s tools, lists them, filters by allowed, deduplicates, translates them into OpenAI function format and concatenates them. It calls the model. If there are tool calls and auto-execution is permitted, it runs them against the MCP servers. It builds the follow-up and makes one second call.

Two limits you need to know before designing on top of this.

The loop is a single hop. There is no multi-step iteration. The calls from the first response are executed and the follow-up is returned. A plan that chains tools requires the client to call again.

Auto-execution fails closed and as a block. It requires all MCP references to have approval set to “never”; a single one requiring approval disables auto-execution for the whole request.

And a third one, which is the one that bites: the multi-turn state of /v1/responses is rehydrated from the spend table. The query is literal, it looks up the session of the previous request_id and retrieves all the rows of that session ordered by time.

With disable_spend_logs: true nothing is written, the query comes out empty and the history is lost in silence, with no 4xx error. The code itself acknowledges it: it does not expect retries because deployments that do not write spend logs have nothing to wait for. It is a coupling between a performance option and a conversation feature that nobody expects to find.

Health and day-to-day operation

Health checks exist, they are on demand and they are not periodic. GET /v1/mcp/server/health accepts repeated identifiers and returns per-server status. The implementation opens a client and runs an empty operation inside the session, with a ten-second timeout.

It skips the check entirely if the server requires per-user authentication, and those always return unknown status.

The surprising part: the status, last check and error columns have existed in the database since a May 2025 migration, and the check function returns an object without writing to the database. Status is recalculated on every request. There is no periodic MCP health job.

When an upstream server is down, the behaviour is to degrade and not block. The listing fans out in parallel and each failing server contributes an empty list plus a result classified into six categories. Those results reach the client in the response’s _meta under a vendor key, without filtering the upstream’s text or URL. The reason for the design is in the code: an empty contribution with no signal makes a broken upstream indistinguishable from a healthy server with no tools.

There is no cache of the previous list. A server that is down disappears from that turn’s tools.

On registering and deregistering without restarting, there are two worlds. Database servers can be created, edited and deleted through the API, and every operation reloads the registry with an atomic swap. Propagation between replicas goes through Redis publication on the configuration change channel, if there is Redis, and through polling every 30 seconds if there is not.

Servers declared in config.yaml are loaded once, at start-up. There is no re-read. Changing them requires a restart.

And there is a precedence trap between the two worlds, with two different warnings in the code. A database row with the same identifier as a configuration entry hides the configuration one entirely. And the other way round, a configuration identifier that matches the name or alias of a database server captures its permissions, because identifiers are resolved before names. Both cases emit a log warning, once.

A hardened `config.yaml`

general_settings:
 # Access is not open by default, but this closes the empty-key
 # inheritance towards the team, which is the one that surprises people.
 require_key_mcp_access_defined: true
 require_end_user_mcp_access_defined: true
 # Without this, allow_all_keys overrides the key's explicit scope.
 mcp_allow_all_keys_respects_mcp_scope: true

litellm_settings:
 # Does not cover MCP arguments. See the auditing section.
 turn_off_message_logging: true
 callbacks: ["langfuse_otel"]

mcp_servers:
 inventario:
 url: https://mcp-inventario.interna.svc/mcp
 transport: http
 auth_type: bearer_token
 auth_value: os.environ/MCP_INVENTARIO_TOKEN
 # Explicit allowlist. Also enforced on tools/call.
 allowed_tools:
 - buscar_articulo
 - consultar_stock
 # Never true on a server with data.
 allow_all_keys: false
 access_groups: ["operaciones"]
 timeout: 20
 max_concurrent_requests: 8
 mcp_info:
 description: "Inventario de almacen, solo lectura"
 # Without this, MCP spend is zero and its metric does not exist.
 mcp_server_cost_info:
 default_cost_per_query: 0.0005
 tool_name_to_cost_per_query:
 consultar_stock: 0.0002

And in the process environment:

# Without this there is no MCP span, only a generic litellm_request.
LITELLM_OTEL_V2=true
# Full block hashes, not truncated, if correlating
# with engine events.
LITELLM_MCP_CLIENT_TIMEOUT=45
LITELLM_MCP_TOOL_LISTING_TIMEOUT=20

For clients, the rule is not to use the aggregated endpoint. Each agent points at /{server}/mcp or at a tool set, or at the three-tool proxy mode if the catalogue is large.

Checklist

Check the version. Below 1.83.14 there is remote execution and an unauthenticated SQL injection with an assigned CVE.
Inventory which servers have allow_all_keys and enable the setting that makes it respect the key’s scope.
Audit the access groups separately, because they are additive on top of the key and team ceiling.
Declare in writing the fail-open behaviour on tool permission resolution errors.
Measure the real cost: number of tools per server from the listing row, and tools block tokens from the stored request body.
Choose a catalogue strategy before connecting the third server: three-tool proxy mode, semantic filter, or selection by route. Not the aggregated endpoint.
Declare mcp_server_cost_info even with a token value, or the spend metric will not exist.
Enable the OTel v2 logger, and decide deliberately whether to enable content capture.
Assume the trace is cut at the gateway and plan correlation with the upstream by other means.
Treat the spend table’s metadata column as a personal data repository: encryption, its own retention and separate access control.
Set up the rug pull check, because nobody does it: hash of the description and schema of every approved tool, periodic comparison against the listing.
Restrict egress towards MCP servers at the network level, because the gateway does not validate the destination URL.
Do not enable the stdio transport unless you need it, and if you do, review what is on the command allowlist.
If you use /v1/responses with multi-turn state, do not enable disable_spend_logs.

Traps and things that are not what they seem

MCP access is not open by default in 1.102.0. I got that wrong in the previous post. What opens things up is per-server allow_all_keys.
allow_all_keys overrides the key’s explicit scope unless you enable a setting that ships disabled.
require_key_mcp_access_defined only closes key-to-team inheritance. Access group grants remain additive.
On an indeterminate failure resolving tool permissions, the code fails open for keys and JWTs.
The default cost of an MCP call is 0.0, and the spend metric is only emitted if the cost is greater than zero, so the series does not exist.
MCP calls appear in the per-team report as if they were models, named MCP: tool and with zero tokens.
The OTel logger with MCP support is off by default. Without the environment variable there is no MCP span.
Trace context is not propagated towards the upstream MCP server. There is no end-to-end trace.
mcp.tool.name does not exist in the semantic conventions. The name goes in gen_ai.tool.name.
Message redaction does not cover tool arguments. They are written in the clear into the spend table.
Registering, modifying and deleting MCP servers generates no audit log. There are three TODOs in the code.
The semantic tool filter is not applied on tools/list. Only on /chat/completions and /v1/responses.
The catalogue order is not deterministic, because it comes from iterating a set of strings. It breaks the client’s prompt cache even though nobody filters anything.
search_tools in proxy mode counts keywords by default, it uses neither regular expressions nor embeddings, and its cap of five results is hard-coded.
Description overrides really do replace what the client sees, and they are the only catalogue lever that does not break the cache. They are only configurable through the database or the API.
spec_version no longer exists in the configuration schema.
/mcp/sse does not use the SSE transport, and the messages endpoint its object advertises is dead code.
Listing pagination truncates in silence when the deadline expires, without leaving a single log line.
A database row hides a configuration entry with the same identifier, and the other way round a configuration identifier can capture the permissions of a database server.
Servers declared in config.yaml require a restart. Only the database ones reload hot.
The health check does not write its result to the database even though the columns exist, and there is no periodic job.
The gateway does not validate the upstream MCP server’s URL. The anti-SSRF utilities exist and are not used on that path.
docker and npx are on the stdio command allowlist.
The /v1/responses tool loop is a single hop, and disable_spend_logs: true breaks its multi-turn behaviour with no error.
LiteLLM advertises the June 2025 revision of the protocol, two behind the one in force.

Closing

There is a thread running through the six posts in this track that is more visible here than anywhere else. The gateway is chosen for one job and ends up doing five, and each new job arrives with its own defaults, which are almost never the ones you would have chosen.

In the case of MCP the asymmetry is twofold. On one side, the part that looks dangerous is better built than you would assume: access control is enforced on execution and not just on the listing, the hierarchy is built out of intersections, header forwarding is an allowlist, and default access is closed, contrary to what I myself wrote two days ago. On the other, the part nobody looks at is worse: tool arguments come out in the clear bypassing the privacy switch, spend is worth zero and its metric never gets to exist, the trace is cut, and a change to the description of an approved tool goes undetected by anybody in the whole chain, neither the specification nor the gateway.

The architecture decision that comes out of this is that the MCP door has to be treated as what it is, an interconnection of systems with third parties, and not as one more function of the LLM proxy. That means its own inventory, formal registration, change control over every tool’s description, network isolation towards the upstream and a log designed knowing that it is going to contain personal data.

And it means accepting that the cost you really pay for connecting fifteen MCP servers is in no column of the spend table. It is spread across the prompt_tokens of every model call, on every turn, and across the accuracy you lose when the catalogue is too large for the model to choose well. That number can be computed with what the gateway already stores. The first step is wanting to compute it.

Sources

LiteLLM, MCP Gateway: https://docs.litellm.ai/docs/mcp.
LiteLLM, código: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, auth/user_api_key_auth_mcp.py, cost_calculator.py, faults/list_outcomes.py, litellm/proxy/management_endpoints/mcp_management_endpoints.py, litellm/proxy/spend_tracking/spend_tracking_utils.py, litellm/litellm_core_utils/redact_messages.py, litellm/integrations/otel/, litellm/responses/litellm_completion_transformation/session_handler.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en llamadas MCP: https://github.com/BerriAI/litellm/issues/34534.
NVD, CVE-2026-42271, ejecución de comandos por los endpoints de prueba de MCP: https://nvd.nist.gov/vuln/detail/CVE-2026-42271.
NVD, CVE-2026-42208, inyección SQL sin autenticación en la validación de clave: https://nvd.nist.gov/vuln/detail/CVE-2026-42208.
NVD, CVE-2026-35030, colisión de clave de caché de userinfo OIDC: https://nvd.nist.gov/vuln/detail/CVE-2026-35030.
NVD, CVE-2025-49596, ejecución remota en MCP Inspector: https://nvd.nist.gov/vuln/detail/CVE-2025-49596.
NVD, CVE-2025-6514, inyección de comandos en mcp-remote: https://nvd.nist.gov/vuln/detail/CVE-2025-6514.
NVD, CVE-2025-66416, protección de DNS rebinding desactivada por defecto en el SDK de Python: https://nvd.nist.gov/vuln/detail/CVE-2025-66416.
Invariant Labs, MCP Security Notification: Tool Poisoning Attacks (tool poisoning, rug pull, shadowing): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks.
Invariant Labs, Introducing mcp-scan (fijado de herramientas por hash): https://invariantlabs.ai/blog/introducing-mcp-scan.
Model Context Protocol, changelog de la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/changelog.
Model Context Protocol, changelog de la revisión 2025-11-25: https://modelcontextprotocol.io/specification/2025-11-25/changelog.
Model Context Protocol, Security Best Practices (confused deputy, indicadores de recurso, validación de audiencia): https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.
NSA, Model Context Protocol: Security Design Considerations for AI-Driven Automation, mayo de 2026: https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDF.
CISA y agencias aliadas, Careful Adoption of Agentic AI Services, mayo de 2026: https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services.
Anthropic, Advanced tool use (coste en tokens de las definiciones, búsqueda de herramientas, precisión de selección): https://www.anthropic.com/engineering/advanced-tool-use.
Anthropic, Code execution with MCP: https://www.anthropic.com/engineering/code-execution-with-mcp.
Anthropic, Prompt caching (jerarquía tools → system → messages e invalidación): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.
Anthropic, Tool search tool (herramientas diferidas fuera del prefijo, umbral de 30 a 50 herramientas): https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, Prompt caching (el orden de las herramientas cuenta): https://developers.openai.com/api/docs/guides/prompt-caching.
OpenAI, Tool search (las herramientas descubiertas se cargan al final del contexto): https://developers.openai.com/api/docs/guides/tools-tool-search.
vLLM, Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html.
Microsoft, Use tools in chat (límite de 128 herramientas por petición en VS Code): https://code.visualstudio.com/docs/copilot/agents/agent-tools.
RAG-MCP, prueba de estrés de 1 a 11.100 servidores candidatos, mayo de 2025: https://arxiv.org/abs/2505.03275.
Benchmarking the Benchmarks, desalineación del 18,5 % entre etiqueta y realidad, junio de 2026: https://arxiv.org/html/2607.02577v1.
MCP-Atlas, taxonomía de fallos sobre 220 herramientas y veinte modelos, febrero de 2026: https://arxiv.org/html/2602.00933v3.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307.
MCP Tool Descriptions Are Smelly, auditoría de 856 herramientas de 103 servidores, febrero de 2026: https://arxiv.org/html/2602.14878v1.
ToolMenuBench, seis estrategias de filtrado y 26.460 ejecuciones, junio de 2026: https://arxiv.org/html/2606.15508.
Retrieval Models Aren’t Tool-Savvy (ToolRet), corpus de 43.000 herramientas, marzo de 2025: https://arxiv.org/abs/2503.01763.
Tools Are Not Islands, brecha entre Recall@3 y conjunto completo, julio de 2026: https://arxiv.org/html/2607.25718.
ToolFlood, saturación adversaria de la capa de recuperación, marzo de 2026: https://arxiv.org/html/2603.13950.
Scores Are Not Decisions, el K óptimo como función del coste, julio de 2026: https://arxiv.org/html/2607.27083v1.
Don’t Break the Cache, caché de prompts en tareas agénticas de horizonte largo, enero de 2026: https://arxiv.org/html/2601.06007v2.
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer, mayo de 2026: https://arxiv.org/html/2605.24660v1.
Semantic Tool Discovery for LLMs, marzo de 2026: https://arxiv.org/abs/2603.20313.
OpenTelemetry, convenciones semánticas de GenAI y MCP: https://github.com/open-telemetry/semantic-conventions-genai.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
IBM ContextForge, gateway y registro MCP: https://github.com/IBM/mcp-context-forge.
Docker MCP Gateway: https://github.com/docker/mcp-gateway.
agentgateway: https://github.com/agentgateway/agentgateway.

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

Thu, 10 Sep 2026 08:00:00 +0200

Fourth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability and virtual keys covered governance. Here comes the new tenant: agents, which speak the same protocol as humans and behave in a different way. Everything that follows is verified against LiteLLM 1.100.1 and vLLM 0.29.0, both from the first ten days of September 2026.

TL;DR

The day a team connects an agent to the corporate gateway, the platform changes regime without anyone touching a YAML. These are the seven conclusions.

LiteLLM’s priority parameter prioritises nothing under normal conditions. It is still marked as beta and its queue is only consulted when all the deployments in the group are in cooldown. With free capacity, the only thing it produces is a response header. On top of that, the success branch does not pop the item off the queue, so the list grows indefinitely; and on /chat/completions the value 0, which is the one used by all the documentation examples, is falsy in Python and disables queuing.

Separating traffic is done with two pools, not with priorities. The non-forgeable way is model_group_alias at key or team level: the client always asks for qwen-30b and the credential decides whether that resolves to qwen-30b-interactivo or to qwen-30b-agentes. The way that does not touch clients is tag_regex against the User-Agent, with a pattern like ^User-Agent: claude-code\/, which serves to classify load and not to control access, because that header is set by the client.

The useful admission control arrives in the 1.101 branch, with max_in_flight_requests_per_worker and a per-process queue that returns 503 with retry-after. It rejects before authenticating, so what it discards does not appear attributed to any key.

max_parallel_requests is only applied at virtual key level. The team, user, end customer and organisation endpoints accept the field and store it in the database, and nobody reads it. global_max_parallel_requests is a no-op unless the old limiter is reactivated, and general_settings.max_parallel_requests is read by no code path.

vLLM with --scheduling-policy priority does not push ahead anyone who is already generating. The only thing that changes is who it evicts when KV cache runs short. If the agents have filled max_num_seqs, a chat with the highest priority waits just the same. And the two knobs that in 2025 allowed short prompts to get ahead of long ones, max_num_partial_prefills and max_long_partial_prefills, disappeared from the code in 0.27.

The retry product reaches 45 calls per agent turn. Three from the client SDK times three from the router times up to five fallback groups. The proxy sets max_retries=0 on the SDK it uses itself, but it cannot touch the client’s. The timeout is per attempt, not per request, and LiteLLM’s Timeout comes out as a 408, which is retryable.

A single 429 takes a deployment out of the pool for five seconds, without going through the allowed_fails threshold. With an agentic burst against a fleet of four replicas, the whole pool enters cooldown and the gateway returns a 429 of its own to everybody, humans included.

You are here: the gateway layer with two classes of traffic

In the seven-layer stack this still lives in the gateway layer, with one difference compared to the previous three articles. There the variable was load: more requests, more teams, more budget to share out. Here the variable is the shape of the load.

A gateway sized for three hundred engineers using a chat works. The same gateway, with thirty of those engineers running coding agents, serves a fraction of the human traffic with the previous latency and the rest degrades. The GPUs stay just as busy, the utilisation metrics stay good, and the support team gets complaints from people waiting nine seconds for a paragraph.

The analogy: the registry desk and the courier with two hundred files

A registry office serves people in order of arrival. Citizens come in one at a time, present a piece of paper, and leave. The system is fair and the queue moves.

One Tuesday a courier turns up with a trolley of two hundred files. He joins the queue like anyone else, because rules are rules, and when his turn comes he occupies the desk for forty minutes. Behind him are twelve people with one piece of paper each. Nobody has broken any rule. The courier has the same right as everyone else and uses the same procedure. What fails is that the office was designed assuming a certain size of transaction and another one three orders of magnitude bigger has turned up.

The real solutions for an office are three, and all three have an exact equivalent in the platform. Open a desk for bulk deliveries, which is separating pools. Put up a capacity sign and send people back later, which is admission control with its 503. And say that files are handed in during the afternoon, which is the part that on an inference platform can almost never be applied, because the agent does not negotiate schedules.

What does not work in the office is giving the citizen a priority number when the courier is already at the desk. That number does not interrupt the transaction in progress. It is, literally, what priority does in the two layers of the platform, and it is worth keeping in mind while reading the next section.

The two kinds of traffic, measured

Before configuring anything, you have to be able to state the difference in numbers, because every later adjustment comes out of it.

Trait	Human chat	Agentic loop
Input per request	hundreds of tokens	tens of thousands, and it grows with each turn
Output per request	hundreds of tokens	from two tokens (a tool call) to several thousand
Concurrency per user	1	3 to 20 requests in flight
The metric that hurts	time to first token	total turn throughput
Tolerance for waiting	seconds	minutes
Cancelled requests	rare	routine, by the agent’s own decision
Retries	the browser’s	the SDK’s, plus the loop’s
Witness when it fails	the user	nobody

Two rows deserve comment because they contradict intuition.

The first is the metric row. An agent does not perceive time to first token, so on a shared engine it is the natural candidate to give up that metric. The operational conclusion is that the agent pool can be configured for aggressive throughput, with large batches and deep queues, while the interactive pool is configured for the opposite.

The second is the cancellations row. An agent that launches three speculative continuations and discards two generates two aborted streams per turn. At the gateway, that touches spend accounting, the release of concurrency counters and the open connection to the engine. LiteLLM 1.100 handles it reasonably well, and it did not six months ago.

What does not work: `priority` in LiteLLM

The first reaction of any operator is to look for a priority field. LiteLLM has one, documents it on its Request Prioritization page, and that page carries the beta label and the sentence saying it is for testing. The label is correct and has to be taken literally, because behind it there are four things.

The queue is only consulted when the whole group is in cooldown. The scheduler’s poll() method returns true if there is any healthy deployment, without touching the queue. With available capacity, which is the normal situation, the request goes straight through and priority orders nothing. The only thing that changes is that the response carries the x-litellm-request-prioritization-used header.

The queue does not empty. In that success branch there is no heappop and no call to the cleanup, which only happens in the expiry branch. The list under the key scheduler:queue:{model} grows without bound, is serialised whole into Redis on every insertion, and there is no maximum size and no discard policy. When the group does enter cooldown, the head of the heap is occupied by fossil entries from requests that finished hours ago, and new ones compare their identifier against that head, never match, and wait until the timeout runs out.

On /chat/completions, the highest priority disables the feature. The code reads the value with a kwargs.get("priority") or self.default_priority, and in Python 0 or None is None. Since in this scheduler the lowest number is the highest priority, priority: 0, which is the value in the documentation’s own examples, goes down the path without queuing. On /completions the check is done with is not None and it works; the two routes do not behave the same.

And default_priority in router_settings breaks the request. If the client does not send a priority but the global setting is in place, the value is truthy, the scheduling branch is entered and a function is invoked that requires the positional argument nobody has injected. The setting has no associated test and does not appear in the router_settings reference.

There is also a /queue/chat/completions, marked as experimental and hidden from the OpenAPI schema, which reads data["priority"] with no default value: a request without that field raises an exception that comes out as a 400 authentication error, with a message that has no relation to the cause.

There is a priority feature that is solid, and it is a different thing. The v3 dynamic limiter shares out token and request per minute quota between classes with weights, through litellm.priority_reservation with values like {"premium": 0.75, "standard": 0.25}. It shares out capacity, it does not order a queue: below the saturation threshold it lends free capacity and above it applies the weights. It never makes anyone wait, either it passes or it returns a 429. It requires Postgres and an enterprise licence, and it is still marked as beta.

What does work: two pools

Effective separation is done earlier, by deciding which set of replicas each class of traffic goes to. LiteLLM offers two mechanisms with very different security properties.

By credential: `model_group_alias` at key or team level

Router settings are resolved in the order key, team, global. Among those that accept that override is model_group_alias, and there lies the piece:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 model_info:
 id: chat-01
 max_input_tokens: 32768
 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 model_info:
 id: agentes-01
 max_input_tokens: 131072

And on the agent team’s key:

curl -X POST "$PROXY/key/generate" \
 -H "Authorization: Bearer $MASTER_KEY" \
 -d '{
 "team_id": "plataforma-agentes",
 "models": ["qwen-30b"],
 "router_settings": {
 "model_group_alias": {"qwen-30b": "qwen-30b-agentes"}
 }
 }'

The client asks for qwen-30b and knows nothing about any of it. The resolution is tied to the credential, so the caller cannot get around it by changing a header. The same list allows routing_strategy, fallbacks, context_window_fallbacks, retry_policy, cooldown_time and allowed_fails to be set per key or team, which gives different reliability policies for each class of traffic on the same proxy instance.

It should not be confused with access groups (model_info.access_groups), which are permissions and not routing: they decide whether a key can invoke a model name, not which replica the request goes to.

By `User-Agent`: `tag_regex`

When separate credentials cannot be issued, because the agentic client uses the same personal key as that person’s chat, LiteLLM classifies by regular expression against the request headers:

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 tag_regex: ["^User-Agent: claude-code\\/", "^User-Agent: .*codex"]
 model_info: {id: agentes-01}
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 tags: ["default"]
 model_info: {id: chat-01}

router_settings:
 enable_tag_filtering: true
 tag_filtering_match_any: true

The proxy composes strings of the form User-Agent: <value> and tests them with re.search. Exact tag matching takes precedence over the regex. The documentation itself carries a warning that has to be respected: the User-Agent is written by the client, so this is traffic classification and not a security boundary. As load sharing it is enough, because an agent that lies about its identity to sneak into the interactive pool is a governance problem, not a routing one.

The behaviour when no deployment carries the requested tag has a nuance that comes as a surprise in production. If the tag is unknown to the group, the request falls through to the pool marked with tags: ["default"]. If the tag exists in the group but no healthy deployment carries it at that moment, the request fails with the tag configuration error, even though a default pool is available. To get deterministic fallthrough to the default pool you have to set model_info.allow_fail_open: true.

A different strategy per pool with `routing_groups`

router_settings:
 routing_strategy: simple-shuffle
 routing_groups:
 - group_name: interactivo
 models: [qwen-30b-interactivo]
 routing_strategy: simple-shuffle
 - group_name: agentes
 models: [qwen-30b-agentes]
 routing_strategy: simple-shuffle

The recommendation to leave simple-shuffle on both is not laziness, and sustaining it requires looking at what the alternatives do on a homogeneous vLLM fleet.

least-busy keeps a complete dictionary under a single key and updates it with non-atomic reads and writes, without using INCR, so with several proxy replicas increments are lost. It has no TTL, so a request that never completes leaves the counter raised forever and that replica is permanently excluded. Counters can go negative. And when the minimum corresponds to an identifier that is no longer in the healthy list, it falls back to random.choice without warning. The pattern of requests that do not complete is exactly that of an agent that cancels.

latency-based-routing measures seconds per output token, not wall-clock latency. A chat turn of twenty tokens looks extremely slow by that metric and an agent generation of two thousand tokens looks extremely fast, so mixing both kinds of traffic in the same group poisons the signal. It also seeds new deployments with zero latency, and with the buffer defaulting to zero, a freshly started replica takes all the traffic until its first success callback.

usage-based-routing-v2 uses minute windows that reset all at once, and it accounts for tokens when the request completes, so long generations in flight are invisible to the router. With long outputs, it systematically underestimates the real load.

cost-based-routing sorts by the sum of the input and output unit prices and takes the first, with no random tie-break. With replicas of the same model at the same price, all the traffic always goes to the same place. And a model that is not in the price map gets a default cost of 5.0 per token, a figure designed to deprioritise it.

Admission control: the new piece in the 1.101 branch

When the agent pool saturates, what you want is to reject fast and with a code the client understands, not to accumulate requests inside the proxy until memory runs out. That arrives with per-worker admission control, added in the 1.101 branch, in release candidate as this is written.

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

It is a semaphore with a per-uvicorn-process queue, without Redis. When the queue fills or the wait expires it returns 503 with retry-after: 1 and a body with "type": "overloaded_error". It is observed through /health/backlog, which exposes requests in flight, admitted, queued and rejected, and through the Prometheus litellm_admission_* metrics, whose rejection counter distinguishes queue_full from queue_timeout.

Two properties have to go into the runbook. The middleware rejects before authenticating, so what is discarded is not attributed to any key in the spend records, and the real number of requests rejected per team cannot be reconstructed from the spend table. And the limits are read on the first request: changing them requires a restart.

Concurrency per key, and the three settings that do nothing

max_parallel_requests is the right brake for a specific agent, and its configuration surface is full of dead ends.

The only level where it is applied is the virtual key. The v3 limiter, active by default since 1.94, has descriptors for user, team, team member, end customer, organisation, model per key, tag and agent, and all of them carry only request and token counters per window. The only reference to max_parallel_requests is the one in the key descriptor. The old limiter says so in a comment: supporting it for model, user and team is pending.

At the same time, /team/new, /user/new, /customer/new and the organisation endpoints accept the field, validate it and write it to Postgres, and the documentation shows it in their examples. Nobody reads it afterwards. A team with max_parallel_requests: 20 in its row has no concurrency limit at all.

global_max_parallel_requests is injected into the request metadata and is only consulted by the old limiter, which is disabled unless LEGACY_MULTI_INSTANCE_RATE_LIMITING=true is set. The server settings documentation still describes it as a global limit coordinated through Redis. And general_settings.max_parallel_requests does not appear in any execution path: its only two appearances are the copy from the database and the type schema of the interface.

What does work, in detail:

curl -X POST "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" \
 -d '{"team_id":"plataforma-agentes","max_parallel_requests":8,"rpm_limit":600}'

Counting is done with a sorted set in Redis scored by Redis’s own clock, with Lua scripts to acquire and release. Each slot expires after 3,600 seconds, which is at once the maximum request duration the meter can track and the time a leaked slot takes to heal by itself. Release is hooked to success, to failure and to client disconnection during streaming, which was the classic leak with clients that cancel.

One live leak remains and it affects exactly the case in this article: every tool call through the MCP gateway acquires a slot and does not release it. The MCP route invokes the pre-call hook, which acquires, and none of the release functions is called from that module. With an agent that makes twenty tool calls per turn and a key with eight slots, the 429 arrives on the first turn and lasts an hour.

The order of evaluation matters for interpreting 429s. First the window counters, requests and tokens per minute, are checked, and only if they pass is the concurrency slot acquired. The code comment explains why: the other way round, every rate limit rejection would leave an orphan slot. In practice, with a low rpm you always see the requests type 429, and the max_parallel_requests one only appears when the relationship between concurrency and duration makes it binding first. The message distinguishes the two cases with the Limit type: field and the rate_limit_type header.

A minor detail with consequences on the client: the response to a concurrency rejection carries retry-after and reset_at computed from the window size, sixty seconds by default, when a concurrency meter has no window. A well-behaved client that respects that header will wait a minute when it might have had a free slot in two seconds.

Session affinity: the KV cache that gets thrown away

An agent turn resends the whole conversation context, system prompt and tools included. If that request lands on a different replica from the previous turn’s, vLLM’s prefix cache does not have the prefix and the prefill is recomputed in full. With twenty turns and four replicas balanced blindly, the long prefill is paid for fifteen times over.

LiteLLM has two mechanisms for this and they are not equally good.

The prompt_caching check computes a SHA-256 of the cacheable prefix, defined as everything up to the last block with cache_control, and pins the deployment with that hash. It has two problems. The TTL is hardcoded to 300 seconds in two places in the code, with no way to configure it, something which is reported and open; with one-hour caches, affinity is lost after five minutes of inactivity. And the key is an exact hash, so a client that moves the cut point forward on every turn, which is exactly what agents do to extend the cached prefix, generates a new key every time.

The session_affinity check is the one to use:

router_settings:
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

It pins the deployment with an atomic Lua script in Redis, with a configurable TTL that defaults to one hour and that is refreshed on every request, so it bounds the idle time between turns and not the duration of the conversation. If the pinned deployment enters cooldown, the request goes out through the normal strategy and keeps the pin so it can come back when it heals.

The session identifier comes from a precedence chain that starts with x-litellm-trace-id, continues with x-litellm-session-id and then accepts any header of the form x-<something>-session-id whose value looks like an identifier. That includes the session header Claude Code already sends, with nothing configured on the client. Whatever matches populates both the session identifier and the trace identifier, which links this to the correlation from the first article in the track.

One dangerous interaction when combining it with tags: affinity is evaluated before tag-based routing. If the pin reduces the list to a deployment that does not satisfy the request’s tag, the request fails instead of falling through to the default pool, unless allow_fail_open has been enabled. Retagging a pool with live sessions breaks them.

And a clarification that saves a pointless search: LiteLLM does not do prefix routing. There is no prefix tree and nothing that directs the request to the vLLM replica that already has that prefix warm. What there is, is affinity by session or by key. For the other thing you need a specific router underneath the gateway, and that architecture has its own article pending.

The engine: what vLLM can and cannot do

Everything above shares out requests between engines. Inside one engine, the question is what happens when a twenty-token request and a sixty-thousand-token one live side by side.

Priority exists and does less than it looks like. --scheduling-policy priority turns the waiting queue into a heap ordered by priority, arrival time and identifier, with the lowest number first. The request can carry the priority field in the body, and there is also the X-Vllm-Priority header, which takes precedence over the body and is not documented anywhere.

What priority does not do is push ahead of whoever is already generating. In the running queue loop, when KV cache runs short, the policy only changes who is chosen as the victim of eviction: with priority the one with the worst priority and arrival time is evicted, and with fcfs the last in the queue. In the waiting queue loop, if there are no blocks available the iteration is cut short, without evicting anyone. And the max_num_seqs cap is a hard cut. If the agents have filled that cap, a chat with maximum priority waits just the same. There has been an open issue since April and an unmerged pull request about this.

That nuance about the victim is not negligible, because with fcfs the victims are the most recently admitted, that is, the chats that have just come in. Switching the priority policy on already improves the situation even if it pushes nobody ahead.

Eviction is always by recomputation. The swap-to-host-memory mode and the enumeration that controlled it disappeared from the code; all that is left is freeing blocks and returning the request to the queue with the computed token counter at zero. An evicted request loses all its prefill, and the prefix cache recovers part of it, but the perceived time to first token restarts.

Two admission settings help and do not appear in the narrative documentation. watermark reserves a fraction of free KV blocks on admission, with the declared purpose of avoiding repeated eviction, and it ships disabled. And scheduler_reserve_full_isl, active by default, checks that the full input length fits in the cache before admitting, instead of looking only at the first chunk.

Chunked prefill is active by default and prioritises decode. The scheduler walks the whole running request loop before touching the waiting queue, so a long agent prefill does not freeze a chat’s generation in progress. That protects the inter-token latency of what has already been admitted, and it does not protect the time to first token of what is waiting.

Here there is a breaking change that forces a review of the manifests. max_num_partial_prefills and max_long_partial_prefills were removed in 0.27. They were precisely the mechanism that allowed short prompts to get ahead of long ones. A deployment with those flags in its args does not start. long_prefill_token_threshold still exists, and its meaning changed: today it is a per-request chunk size cap, not a classification of long requests.

The partial replacement arrives in 0.29 with max_num_queued_tokens and max_num_queued_reqs, admission valves that return 503 when the pending prefill work exceeds the target. The code itself frames it as a quality of service mechanism for time to first token, with the formula of setting it to the latency target multiplied by prefill throughput. They are blind to the class of traffic: they reject a chat just the same as an agent.

And the default value of --max-num-batched-tokens depends on the hardware and on the start-up mode. With vllm serve on H100 or H200 it is 8,192, on cards of 160 GiB or more it is 16,384, and on the rest 2,048. The 2,048 that appears in the scheduler configuration only applies to tests.

The metrics for diagnosing coexistence, with the exact names from 0.29:

Metric	What for
`vllm:num_requests_waiting`	queue depth, the autoscaling signal
`vllm:num_requests_waiting_by_reason`	separates waiting for capacity from waiting in a blocked state
`vllm:num_preemptions_total`	evictions, the signal that the two classes are treading on each other
`vllm:kv_cache_usage_perc`	cache pressure, the cause of the evictions
`vllm:request_queue_time_seconds`	what a chat waits with the agents inside
`vllm:prefix_cache_hits_total` and `_queries_total`	measured in tokens, not in requests

A warning for anyone with old alerts. The log warning about evicted sequences that appears in the optimisation documentation belongs to the previous version of the engine and no longer exists in the code. Preemptions are only visible as a counter. An alert based on searching for that string in the logs has not fired for months, and not because there are no evictions.

On isolation between classes inside one engine, the short answer is that it does not exist. There are two queues and neither is per class of traffic. The conclusion, which is the one that holds up this article’s architecture: effective isolation means separate fleets, and on top of them the gateway deciding who goes to which.

Retries: the product nobody calculates

A human who gets an error sees it and decides. An agentic loop retries, and beneath it there are two more layers that also retry.

Layer	Default
Client SDK	2 retries, 3 attempts
LiteLLM router	2 retries, 3 attempts
Fallbacks	up to 5 groups

The product is about 45 calls to the engine for one agent turn, before counting the loop’s own retry. LiteLLM does what it can: when the call comes from the router, it forces max_retries=0 on the SDK it itself uses against the engine. What it cannot touch is the client’s SDK. The only layer that breaks the product is that one, and it is broken by setting max_retries=0 on the client and delegating the retry to the gateway.

The retryable errors are 408, 409, 429 and anything 500 or above. And there lies the problem with long reasoning: litellm.Timeout is a subclass of OpenAI’s timeout exception and comes out with code 408, which is retryable, so the most expensive request of all, the one that expired after ten minutes of generating, is launched again in full. Twice by the router, and as many times again by the client.

On timeouts you have to be precise because the code comment is: the configured timeout is per attempt, not per request. A request_timeout: 600 with two retries and two fallback groups gives a worst case of almost two hours before the client receives anything. The default value is still 6,000 seconds, as was already covered in the day 2 article.

The retry storm has two specific causes and both are default settings.

The first: a single 429 takes the deployment out of the pool, without going through the allowed_fails threshold. The cooldown logic has an explicit branch which, on a 429 in a group with more than one deployment, returns immediate cooldown. With an agentic burst against four replicas, all four leave the pool at once and the gateway answers everybody with its own 429 of “no deployments available”. The default cooldown is five seconds, which in a sustained burst gets renewed.

The second: backoff is zero while the pool looks healthy. The function that computes the wait returns 0 if the healthy deployment list is not empty. Exponential backoff with jitter, capped at eight seconds, only kicks in when there is nothing healthy left. Under partial degradation, the proxy hammers away without waiting and only slows down once everything has already fallen over.

On top of that, the engine’s Retry-After header is only respected if its value is between 1 and 60 seconds. A longer limit window is ignored and replaced by the proxy’s own backoff.

The settings that cut the problem off:

router_settings:
 num_retries: 1
 retry_after: 2 # real floor; the default 0 is hammering
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3 # stops a 429 taking the deployment out
 retry_policy:
 TimeoutErrorRetries: 0 # do not relaunch the expensive request
 AuthenticationErrorRetries: 0
litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false

That last setting deserves an explanation, because it is a silent way of breaking an agent. With drop_params: true, LiteLLM removes from the body any parameter the destination provider does not support, and in that branch it writes no warning at all. The candidates for disappearing are tools, tool_choice, response_format and parallel_tool_calls. An agent that has tools taken away receives free text where it expected a function call, and interprets it as the model having decided not to use tools.

Worse still is prompt-based emulation. For a non-OpenAI-compatible provider that does not support tools, LiteLLM does not fail: it turns the tools into text inside the prompt, forces JSON format, discards tool_choice, and on the way switches on a process-global variable that makes any other tool-less provider in the same process go to emulation instead of erroring.

With that in mind, the warning about fallbacks is clear: LiteLLM does not check capabilities before falling back. The function that knows whether a model supports function calling exists and is only used in an informational endpoint. The pre-call checks filter by context window, request limit and region, and nothing else. A fallback list that goes from a model with tools to one without them degrades the agent silently. There are guards for provider-scoped resources, files and batches, so the absence of the capability guard is an omission and not a design oversight.

A month’s budget in twenty minutes

A runaway agent consumes in minutes what a team consumes in weeks, and the operational question is how long the gateway takes to find out.

The answer has changed, and the old mental model, the one where spend consolidates every minute and until then there is no brake, is no longer correct. Since 1.84 there are two new pieces. An optimistic reservation before the call, which estimates the maximum cost of the request and reserves it atomically in the counter before heading out to the engine, reconciling it with the real cost on completion. And a counter increment after the call that is explicitly awaited, with the comment that this way the counter is up to date before the next request goes through authentication. Budget checks read that counter, with Redis first and seeding from the database when cold.

Batch writing to Postgres still exists, every 10 to 15 seconds, and it affects persistence and what is shown in the interface, not the cut-off. The production documentation’s recommendation to raise that interval to 60 does not relax the control, and it does delay by a minute what /key/info shows, something which frequently generates operational panic. And the budget rescheduler’s ten minutes do not consolidate spend: they fire the job that resets expired budgets.

The overspend window then comes down to the cost of the requests in flight, provided there is Redis. Without Redis, the counters are per pod, and with N replicas you can spend up to N times the budget. That is the configuration failure that matters and it is the same Redis that the day 2 article declared non-optional for other reasons.

Two settings that do not appear in the public documentation, only in the data model descriptions, and that decide behaviour at the worst moment:

general_settings:
 fail_closed_budget_enforcement: true # false by default: if the counter fails, it lets things through
 # disable_budget_reservation: do not set it to true

With the default value, if the counter backend does not respond, the proxy fails open and carries on serving. For an agent in a loop that is exactly the opposite of what you want.

Time for the honest warning: this route is still unstable. There are open issues in both directions, one where the key and user budget limit stops being applied while spend is recorded, and another where a 429 is returned for stale spend in the reservation counters while the key query shows spend below the limit. Project budgets are not on the atomic reservation route and concurrency overshoots them. The practical conclusion is not to trust the hard cut-off to the budget alone.

On the manual cut-off, there is an asymmetry to know about before you need it. /key/block invalidates the local cache, the Redis one, and notifies all workers by publish, so with Redis the effect is almost immediate. /customer/block does the same. /team/block only writes to the database, without invalidating or notifying, so it takes however long the management object cache TTL takes, sixty seconds by default. To cut a team off dead you have to block its keys one by one, or lower user_api_key_cache_ttl. And /user/block no longer exists, even though the docstring of the function that replaced it still shows that route in its example.

On alerts, two numbers. The default thresholds warn at 85 % and 95 % of the budget. And deduplication of those alerts is 24 hours per event and identifier, so crossing 85 % generates one warning and then silence for a day. For agentic traffic, where 85 % and 100 % can be twenty minutes apart, the second warning arrives when it is no longer any use.

Attribution: who the user is when the caller is an agent

With agents, the question of who the spend is charged to has two answers and both are needed: the agent that executes and the human on whose behalf it acts.

The chain that extracts the end customer identifier tries the standard headers first, then the configurable header mapping, and only then the user field of the body, the litellm_metadata.user of the Anthropic format, the metadata.user_id and the safety_identifier. The last four are body fields and are therefore written by the client. The code itself warns about it: for untrusted callers the identifier has to be set from headers or from a middleware on the server, to avoid impersonation.

Out of that comes the correct pattern, which is the inverse of the one people put together out of inertia: the virtual key identifies the agent and is the non-forgeable part, and the header or the user field identify the human and serve for attribution, not for control. An agent that lies about the human behind it is still bounded by the budget and the limits of its key.

The spend table stores user, end_user, team_id, organization_id, request_tags, session_id and agent_id, with daily aggregation per agent, so the detail is available once it is sent. The x-litellm-agent-id header is what populates that last field.

A note about the MCP gateway, which on a platform with agents ends up being the other entry door. Tool calls do appear in the spend records, with call types of their own and a metadata block that stores tool name, arguments, result and server. What is missing is token counting: the cost is 0.0 by default and is only populated if the administrator configures a price per query, per server or per tool. It is the same pattern of silent zero cost that already came up with unpriced models in the first article in the track.

And the permission setting to review on day one: access control for MCP servers is open by default. If no level of the hierarchy defines a list, the request reaches every configured server. general_settings.require_key_mcp_access_defined: true inverts that inheritance, and it is not the default value.

A reference `config.yaml`

All of the above, together, on a fleet split into two pools:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018 # without this, silent zero cost
 output_cost_per_token: 0.00000072
 model_info:
 id: chat-01
 max_input_tokens: 32768 # without this there is no context filter

 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072
 tag_regex: ["^User-Agent: claude-code\\/"]
 model_info:
 id: agentes-01
 max_input_tokens: 131072
 allow_fail_open: true

router_settings:
 routing_strategy: simple-shuffle
 enable_pre_call_checks: true
 enable_tag_filtering: true
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600
 num_retries: 1
 retry_after: 2
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3
 retry_policy:
 TimeoutErrorRetries: 0
 AuthenticationErrorRetries: 0

general_settings:
 fail_closed_budget_enforcement: true
 require_key_mcp_access_defined: true
 user_api_key_cache_ttl: 10
 proxy_batch_write_at: 10
 # 1.101 branch onwards:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false
 cache: true # Redis, not optional

And the start-up of the two engines, with the deliberate asymmetry:

# interactive pool: protect the first token
vllm serve Qwen/Qwen3-30B \
 --scheduling-policy priority \
 --max-num-seqs 64 \
 --max-model-len 32768 \
 --max-num-batched-tokens 4096

# agent pool: throughput and deep queues
vllm serve Qwen/Qwen3-30B \
 --max-num-seqs 256 \
 --max-model-len 131072 \
 --max-num-batched-tokens 8192

Checklist

Issue separate keys for agents, with model_group_alias at key or team level. It is the only separation the client cannot evade.
Add tag_regex over the User-Agent as a safety net for agents that use personal credentials, with a default pool and allow_fail_open: true.
Leave simple-shuffle on both pools. The other four strategies have verifiable defects on homogeneous replicas.
Enable session_affinity with a one-hour TTL. Do not use prompt_caching for this while its TTL stays hardcoded.
Declare max_input_tokens on every deployment. Without that value, the context window filter is not applied and it fails silently for models that are not in the price map.
Declare per-token prices on every self-hosted model, or the agents’ spend will be zero and their budget will never run out.
max_parallel_requests on the agent’s key, never on its team. On the team it is stored and not applied.
Lower num_retries to 1, put a floor on retry_after, raise cooldown_time and disable retry on timeout. On the client, max_retries=0.
fail_closed_budget_enforcement: true and mandatory Redis. Without Redis, N replicas are N times the budget.
Start the interactive engine with --scheduling-policy priority even though it pushes nobody ahead: it changes who it evicts.
Alert on vllm:num_preemptions_total and vllm:num_requests_waiting_by_reason, not on log strings that are no longer emitted.
Prepare the cut-off: block by key, never by team, and verify the real propagation time in your own deployment.

Traps and things that are not what they seem

priority: 0 on /chat/completions disables prioritisation. It is the value in the documentation examples and it is falsy in Python.
max_parallel_requests on a team is stored and not applied. The field exists in the API, in the database and in the documentation; the limiter only reads it from the key descriptor.
global_max_parallel_requests is a no-op with the default limiter, and the server settings documentation still describes it as active.
Every MCP call leaks a concurrency slot, which only heals when its hour expires. With agents calling tools in a loop, the 429 appears early and lasts a long time.
The TTL of prompt cache affinity is 300 hardcoded seconds. For agent conversations, session affinity is the right option.
Affinity is evaluated before tag-based routing. A pin to a deployment that stops satisfying the tag makes the request fail instead of falling through to the default pool.
LiteLLM’s timeout is per attempt. The worst case is the timeout multiplied by attempts and by fallback groups.
A 429 takes the deployment out of the pool without going through allowed_fails, and backoff is zero while anything healthy remains.
The engine’s Retry-After is ignored if it asks for more than 60 seconds.
drop_params: true removes tools and response_format without leaving a warning, and the fallback does not check whether the destination model supports function calling.
/team/block does not invalidate the cache. It takes the full TTL; /key/block does propagate by publish in Redis.
Access to MCP servers is open by default if no level defines a list.
max_num_partial_prefills and max_long_partial_prefills no longer exist in vLLM. A manifest with those flags does not start from 0.27 onwards.
vLLM’s priority does not push ahead of running requests, and with max_num_seqs full it pushes nobody ahead.
The eviction log warning that appears in the vLLM documentation is no longer emitted. Only the Prometheus counter is left.

Closing

The coexistence of humans and agents on the same platform is not solved with a priority field, and that is the conclusion that takes the most effort to accept because the field exists and looks made for this. Neither the gateway nor the engine has a usable notion of class of service today: one orders a queue that is almost never consulted, the other picks who to evict. What they do have is the ability to direct each request to a different place, and that is where the win is.

The architecture that comes out of all of the above fits in one sentence. Two vLLM fleets with opposite parameters, a credential that decides which one each request goes to, session affinity so as not to throw away the prefill work, concurrency limits on the agent’s key and not on its team, retries trimmed back on the three layers where they multiply, and budgets that fail closed.

The rest is measurement. If vllm:num_preemptions_total rises in the interactive pool, the two classes are treading on each other despite the separation. If chat queue time grows while total throughput holds, the partition is badly sized. And if the agents’ spend comes out as zero, the problem is not coexistence but undeclared prices, which is where this track started.

Sources

LiteLLM, [BETA] Request Prioritization: https://docs.litellm.ai/docs/scheduler.
LiteLLM, Router - Load Balancing (estrategias, afinidad de sesión, comprobaciones previas): https://docs.litellm.ai/docs/routing.
LiteLLM, Tag Based Routing (incluye tag_regex y el ejemplo con clientes agénticos): https://docs.litellm.ai/docs/proxy/tag_routing.
LiteLLM, Router Settings for Keys and Teams (resolución clave, equipo, global): https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.
LiteLLM, Server Tuning (control de admisión por worker): https://docs.litellm.ai/docs/proxy/server_tuning.
LiteLLM, Dynamic TPM/RPM Allocation (reserva de prioridad, beta y enterprise): https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.
LiteLLM, Budgets, Rate Limits: https://docs.litellm.ai/docs/proxy/users.
LiteLLM, Production Best Practices: https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Reliability (fallbacks, ventana de contexto, enfriamientos): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, código: litellm/scheduler.py, litellm/router_strategy/tag_based_routing.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/hooks/parallel_request_limiter_v3.py, litellm/proxy/middleware/admission_control_middleware.py, litellm/router_utils/cooldown_handlers.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de la afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en cada llamada MCP: https://github.com/BerriAI/litellm/issues/34534.
LiteLLM, incidencia 26672, presupuesto de clave y usuario no aplicado: https://github.com/BerriAI/litellm/issues/26672.
LiteLLM, incidencia 872, comprobación previa de soporte de llamada a funciones, cerrada sin implementar: https://github.com/BerriAI/litellm/issues/872.
vLLM, Optimization and Tuning (prefill troceado, prioridad de decodificación): https://docs.vllm.ai/en/v0.29.0/configuration/optimization.html.
vLLM, Metrics design (nombres, convención _total, autoescalado como problema abierto): https://docs.vllm.ai/en/v0.29.0/design/metrics.html.
vLLM, Data Parallel Deployment (caché KV independiente por réplica): https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment.html.
vLLM, código: vllm/v1/core/sched/scheduler.py, vllm/v1/core/sched/request_queue.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/metrics/loggers.py: https://github.com/vllm-project/vllm.
vLLM, incidencia 40004, la prioridad no desaloja peticiones en ejecución: https://github.com/vllm-project/vllm/issues/40004.

The second cost vector of AI agents: durable execution with Temporal

Sun, 14 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

The thesis: agents have two cost vectors, not one

Almost all the cost discourse around agentic AI focuses on a single vector: the cost of inference (the tokens). And that one is solvable: FastMCP + vLLM on local hardware eliminates the cost of inference — you serve the models on your own iron, the cost per token drops to your amortised cost, with no API bill. But there is a second vector that almost nobody measures and that kills more projects: the cost of faulty execution. An agent is not a call, it is a sequence of steps (reason, call a tool, reason, call another one, and so on), and when that sequence fails halfway through with no persisted state, the cost is not linear: it doubles, it spirals with retries and it inflates on every recovery. This article is about that second vector, how durable execution attacks it (with Temporal as the reference), what the state of the art and the papers say, and why it is the real cause of so many cancellations.

The framing that backs the urgency comes from Gartner: it predicts that more than 40 % of agentic AI projects will be cancelled before the end of 2027, because of rising costs, unclear business value or inadequate risk controls (Gartner). “Rising costs” is exactly this second vector: not the cost of serving the model, but the cost of running it badly.

The three expenses of a faulty agent

The cost of faulty execution breaks down into three concrete sources, and it is worth seeing them with numbers:

Vector 1: failures charged twice

With no persisted state, a failure at step 15 of 20 costs you the tokens of the 14 previous steps (which you already paid for) plus the full retry from scratch. The wasted cost of an agent with no checkpoint, per failure, is:

$$\text{wasted cost} = \text{tokens of the completed steps} \times \text{cost per token} \times (\text{retries})$$

The longer the sequence and the later it fails, the more expensive: a failure at 75 % of the work throws away 75 % of the cost. Durable execution checkpoints every step: if the worker dies halfway through, another one picks the workflow up exactly where it stopped, not from scratch (Temporal). Temporal uses its Event History as the record of past decisions; on a failure it replays the progress to date and resumes at the exact point (CallSphere). Step 15 fails → step 15 is retried, not the previous 14. The wasted cost goes from “the 14 steps” to “the step that failed”.

Vector 2: uncontrolled retries

Agents that retry with no policy blow up the API or compute bill: a retry loop against a rate limit can generate dozens of billable calls for a single task. Durable execution defines retry policies per activity: hard limits on retries, exponential backoff and timeouts that bound the maximum possible cost of each call. Instead of “retry until it works” (unbounded cost), you define “at most N retries with backoff and a timeout of T” (bounded cost).

The state of the art backs this with data: the ReliabilityBench benchmark evaluates agent reliability under controlled failures and finds that rate limiting is the most damaging failure of all (arXiv 2601.06112). That is, exactly the failure that an unpolicied retry turns into a cost spiral. Bounding retries is not hygiene, it is the defence against the failure mode the literature singles out as the worst.

Vector 3: inflated context on recovery

With no persisted state, the agent rebuilds the full context on every recovery: it reloads the whole conversation/task history to “remember” where it was, paying the tokens of that enormous context on every recovery. With durable execution the state lives in the workflow history, so the next step receives only the context needed for that step, not the whole conversation. The saving is direct: fewer input tokens per step, and no penalty for rebuilding the context after each failure.

A worked example: the cost of one failure

Let us put numbers on the three vectors. A 20-step agent consuming ~3,000 tokens per step (reasoning plus tool call), on in-house inference at ~1.09 €/1M tokens, with a failure at step 15 and a failure rate of 10 % of executions:

Scenario	Tokens recomputed per failure	Cost per failure	Over 10,000 executions (10 % fail)
No state (full retry)	14 steps × 3,000 = 42,000	~0.046 €	~46 € wasted
With checkpoint (resumes at 15)	0 (only step 15 is retried)	~0.003 €	~3 € wasted

The absolute figure looks small because inference is in-house and cheap; but the ratio is ~15× less waste per failure, and it scales with the length of the sequence and with the cost per token (were it an external API at 10-30 €/1M, the waste without a checkpoint would be multiplied by 10-30). And this is only vector 1: add uncontrolled retries (vector 2), which can turn one failure into dozens of calls, and the context rebuilt on every recovery (vector 3). The cost of faulty execution is not the inference cost × something small; it is a multiplier that grows with the agent’s complexity, and it is exactly what does not show up in the demo and does show up in the production bill.

How Temporal works: workflows, activities and determinism

To understand why Temporal achieves this, its two primitives:

Workflow: the agent’s orchestration logic (the sequence of steps, the decisions). It has to be deterministic: given the same Event History, it produces the same decisions. That is what allows the execution to be replayed after a failure and to arrive at the same point.
Activity: every step with side effects (a call to the LLM, to a tool, to an API). Activities are not deterministic (the LLM is probabilistic), so their result is persisted in the Event History: on replay they are not executed again, they are read from the history.

The trick is that separation: the deterministic logic (workflow) is replayed cheaply from the history; the expensive, non-deterministic steps (activities, the model calls) are persisted and not repeated. That is why “completed work is never repeated”: completed activities are read, not recomputed. And retry policies are defined per activity, so each call to the model or to a tool has its own retry limit, backoff and timeout, which is the vector 2 cost control at the right level.

Durable execution: what it is and the landscape

Durable execution is a programming model that guarantees the code completes despite failures: every step is checkpointed automatically, and if a worker dies, another one picks the workflow up where it stopped (Temporal). It crossed into the “early majority” in 2025, with offerings from AWS, Cloudflare and Vercel, driven precisely by AI agent infrastructure (Inngest).

The tooling landscape (the 2026 reference architecture for durable LLM agents): Temporal, AWS Step Functions Express, Restate, DBOS and Inngest as durable execution primitives; and the checkpointer model of LangGraph (PostgresSaver, RedisSaver, DynamoDBSaver) on the agent framework side (render.com). Temporal stands out for its workflow-as-code model and its Event History as the source of truth.

Option	Model	Self-hosted	Note for sovereignty
Temporal	workflow-as-code, Event History	yes (open source)	the most mature; full on-prem
Restate	durable RPC/handlers	yes	lightweight, modern
DBOS	durable on top of Postgres	yes	state in your own Postgres
Inngest	event-driven, steps	SaaS / self-host	focus on DX
AWS Step Functions Express	state machine	no (AWS SaaS)	tied to AWS, not sovereign
LangGraph checkpointer	framework checkpointer	yes (your backend)	at framework level, not platform level

For a sovereign platform, the ones that are self-hosted (Temporal, Restate, DBOS) keep state in your cluster; the SaaS ones (Step Functions) move execution into someone else’s jurisdiction. Temporal is the choice for maturity and for its complete on-prem deployment.

The cost of durable execution itself

Honesty about the data: durable execution is not free. It has two costs to watch:

Execution overhead: persisting state and replaying history adds latency and compute. The reference architectures put it at around 5-20 % overhead on a load of 100,000 executions/day (render.com).
Price per step: in the offerings with per-step pricing, orchestrating many model calls, rate limit retries and complex interactions generates a large number of billable steps, and the cost can run away (render.com).

The conclusion for a sovereign platform: self-host the durable execution engine (Temporal is open source and deploys on-prem) and the SaaS per-step price disappears, leaving only the execution overhead, a 5-20 % that is paid back many times over by what it saves on the three vectors above. The arithmetic: if a failure halfway through a sequence used to cost you 50-75 % of the recomputed cost, a 5-20 % overhead is a bargain.

Observability: from manual post-mortem to real-time traceability

The fourth element of the equation: the observability layer. The integration with Braintrust and the complete history of every execution (Temporal’s Event History) turn cost analysis from a manual post-mortem, reconstructing by hand what happened and what it cost after a failure, into real-time traceability: every execution is recorded step by step, with its tokens, its retries and its cost, ready to analyse. This closes the FinOps loop of the cost-per-token article, but at the agent level: not just how much a token costs, but how much an execution costs, and where it is wasted.

The complete equation

Putting the pieces together, the cost architecture of a sovereign agent:

FastMCP + vLLM running locally eliminates the cost of inference. Temporal eliminates the cost of faulty execution. They are complementary, attacking different vectors of the same problem. Whoever solves only the first (serves cheaply but executes badly) keeps bleeding through the second; whoever solves only the second (executes robustly but pays for the API) keeps bleeding through the first. A sustainable platform needs both.

State of the art and papers

The reader asked for papers, and there are some: agent reliability and cost is an active research area in 2026.

Paper	Contribution
ReliabilityBench (2601.06112)	an agent reliability benchmark: consistency, robustness and tolerance to tool/API failures; rate limiting is the most damaging failure; ReAct more robust than Reflexion under stress
Cost-Efficient LLM Agents (arXiv)	the reliability-cost trade-off: routing every decision through the LLM improves accuracy but blows up cost; pre-coded graphs are cheaper but brittle against compound failures
Multi-agent Byzantine Fault Tolerance (2511.10400)	BFT consensus (CP-WBFT) to stabilise multi-agent systems under extreme failure rates (85.7 %)
ACRFence (2603.20625)	security of agent checkpoint-restore: preventing semantic rollback attacks
The Six Sigma Agent (2601.22290)	enterprise reliability via consensus-driven decomposed execution

Two readings for the proposal. First, the reliability-cost trade-off is real and is characterised: you cannot maximise accuracy ignoring cost, or the other way round; durable execution shifts the frontier by making reliability cheaper (checkpoint plus bounded retry instead of recomputing). Second, agent checkpoint-restore has its own security surface (ACRFence): persisting state introduces rollback risks that have to be controlled — durability is not only a cost decision, it is a security one too.

The sovereign angle

For a European sovereign platform the combination fits perfectly and is entirely on-prem: vLLM serves the models (no API cost and no sovereignty given away), FastMCP exposes the tools, and self-hosted Temporal orchestrates with durability (no SaaS per-step price and no execution data leaving for a foreign jurisdiction). The execution history, which describes in detail what your agent does and with which data, stays in your cluster, just like the inferences. Against a stack of agents on external APIs plus a US SaaS orchestrator, the sovereign stack removes both cost vectors and keeps the data under EU jurisdiction. It is the same argument as the rest of the series, applied to the agent layer: bounded, measured and sovereign cost.

FastMCP and the tool layer

The other component of the equation is FastMCP, the layer that exposes tools to the agent via MCP (Model Context Protocol). The agent does not call the APIs directly: it calls MCP servers that encapsulate each tool (a database, a search engine, an internal system) with a uniform contract. The connection with cost and durability: every call to an MCP tool is a Temporal activity, so it inherits its retry policy, its timeout and its persistence. That is, FastMCP defines which tools exist and Temporal governs how they are executed with durability and bounded cost. And because the MCP servers are your own, the tools and the data they touch stay on-prem, never exposed to an external provider. The triad vLLM (model) + FastMCP (tools) + Temporal (orchestration) is a fully sovereign agent stack: the model, the tools and the execution, all under your control and with the cost of both vectors bounded.

Patterns: human-in-the-loop and long sequences

Where durable execution shines most, by agent pattern:

Human-in-the-loop (HITL): human approval patterns, essential for an agent’s safety, map directly to the suspend/resume primitives of durable execution, allowing workflows that pause for hours or days waiting for an approval without losing state (Inngest). Without durability, keeping an agent “waiting” consumes resources or loses the context; with it, the workflow sleeps at no cost and wakes up with all its state.
Long sequences: the longer the chain of steps, the greater the checkpoint’s saving (a late failure throws away more work). Multi-step agents with tools are the canonical use case.
Multi-agent: when several agents collaborate, reliability gets complicated (the BFT papers study this); durable execution provides the consistent state substrate on which to build that coordination.

The pattern that does not need it: a single-step, stateless call (a classifier, a simple extraction). There the durability overhead buys nothing. The rule: durable execution for stateful sequences that fail halfway through; for everything else, a direct call.

How to measure the second vector

To manage the cost of faulty execution you have to measure it, and with the Event History plus observability (Braintrust) the KPIs are within reach:

KPI	What it indicates	Where it comes from
Tokens wasted per failure	the cost of vector 1	tokens recomputed after a failure
Retries per activity	the cost of vector 2	retry count from the Event History
Context size on recovery	the cost of vector 3	input tokens after a recovery
Failure rate per workflow	how often the problem occurs	failed executions / total
Cost per complete execution	the business figure	sum of tokens × cost/token per run

The last one closes the circle with cost per token: not just “how much a token costs”, but “how much an agent execution costs, and how much of that was useful work versus waste”. Without these KPIs the second vector is invisible, and what is invisible does not get optimised, it gets cancelled when the bill surprises you. With them, the cost of every execution is a real-time figure, and the decision to make each workflow durable (or not) is taken with numbers.

Limits and traps (data-driven)

Durability is not free. 5-20 % overhead; on SaaS with per-step pricing it can run away. Self-host Temporal to remove that price.
Not every agent needs it. A single-step, stateless task gains nothing from durable execution; the value is in the long sequences that fail halfway through.
Bounded retries can hide real failures. A retry limit avoids the cost spiral, but you have to alert when it is exhausted, not swallow the failure in silence.
Checkpoint = security surface. Persisting state opens the door to rollback attacks (ACRFence); protect it.
The first vector still exists. Temporal does not lower the inference cost; you need local vLLM for that. They are complementary, not substitutes.

The conclusion the data and the papers support: the cost of agentic AI has two vectors, and the industry has obsessed over the first (inference) while ignoring the second (faulty execution), which is precisely where Gartner locates the cause of 40 % of cancellations. FastMCP + vLLM solves the first; Temporal the second. A sovereign and sustainable agent platform needs to attack both, with observability that turns the cost of each execution into a real-time figure rather than an autopsy.

Closing

The dominant narrative about agent cost is half-lame: it treats the inference cost as if it were the only one, when it is only the one you see in the demo. The one that kills projects, the one Gartner files under “rising costs” inside its 40 % of cancellations, is the cost of faulty execution: failures charged twice, retries with no ceiling and the context rebuilt on every recovery. The engineering that attacks it is not a prompt trick or a better model, it is an execution model: durable execution, which checkpoints every step, bounds every retry and passes only the necessary context. Temporal, open source and self-hosted, does this while keeping state in your cluster, and combined with vLLM (model) and FastMCP (tools) it composes a fully sovereign agent stack where both cost vectors are bounded and the data never leaves the EU. The state of the art and the papers confirm the problem is real and characterisable: the reliability-cost trade-off, rate limiting as the worst failure, checkpoint-restore as a new security surface. For anyone who wants to be in the 60 % of agentic projects that are not cancelled, the recipe is the one in this article: solve both vectors, not one, and measure the second as seriously as the first.

Sources

Gartner · más del 40 % de proyectos de IA agéntica cancelados antes de 2027 — https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
Temporal · durable execution — https://temporal.io/
CallSphere · Temporal para workflows de agentes (Event History, replay) — https://callsphere.ai/blog/temporal-ai-agent-workflows-durable-execution-workflow-as-code
Inngest · durable execution, clave para agentes en producción — https://www.inngest.com/blog/durable-execution-key-to-harnessing-ai-agents
render.com · plataformas de workflow durables para agentes/LLM (overhead, precio por paso) — https://render.com/articles/durable-workflow-platforms-ai-agents-llm-workloads
ReliabilityBench (arXiv 2601.06112) — https://arxiv.org/abs/2601.06112
Cost-Efficient LLM Agents (arXiv 2603.01548) — https://arxiv.org/pdf/2603.01548
Byzantine Fault Tolerance multi-agente (arXiv 2511.10400) — https://arxiv.org/abs/2511.10400
ACRFence · checkpoint-restore de agentes (arXiv 2603.20625) — https://arxiv.org/pdf/2603.20625
The Six Sigma Agent (arXiv 2601.22290) — https://arxiv.org/html/2601.22290

Agents on lo0 — Tech Blog

Langfuse v4, day 2 (3 of 10): two well-known agents instrumented, Open Deep Research and GPT Researcher

TL;DR

You are here: OBSERVE, day 2

The analogy: two factories, one audit

The two agents, and why these

Open Deep Research: the shape of the graph

What one request costs, measured

Zero AGENT observations

Instrumenting it without touching the repository

GPT Researcher: two products in one repository

The myth of the fresh config

The standard mode: eight calls, eight traces

The human node that stays open

What to look at in the trace once it arrives

Checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

TL;DR

You are here: OBSERVE, day 2

The analogy: the file that gets photocopied whole at every signature

Which agentic platform is the most used

How LangGraph gets into Langfuse

The observation type is decided by the node name

What a turn costs, measured

What this means for ClickHouse

Interrupts: two traces where there should be one

Thread, session, user, and what is only read at the root

The filter that drops your own spans

The model and the cost with a gateway in the middle

How this looks in code

Agent instrumentation checklist

Traps

The series: the ten articles

See also

Sources

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

TL;DR

You are here: the resource side

The analogy: the visa and the border checkpoint

Part 1. The split of obligations

What is new in the revision and affects the setup

Part 2. What the SDKs give, and what they do not

Protected resource metadata

Audience validation

The authentication challenge

Scopes, which are not per tool

Which revision each one announces

Part 3. Building the protected resource

Part 4. The hop from the gateway to the MCP server

What the gateway does today

Part 5. The enterprise extension, which changes the picture

And here comes the problem

What is already in the gateway, undocumented

Part 6. Per-tool authorisation

Part 7. What still has no standard

Part 8. Who covers what

Reference configuration

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

TL;DR

You are here: the arithmetic that comes before buying

The analogy: the repair shop and the front desk

Part 1. The real load profile

The three sources

The figures you can actually use

What to measure before calculating

Part 2. Prefill, decode and the cache trap

Where it falls over

Co-batching, which is the other effect nobody measures

Part 3. The KV budget, with the numbers of the agentic case

Part 4. From people to concurrent requests

The amplification

A reference `config.yaml`

`allow_all_keys`

`require_key_mcp_access_defined`

A hardened `config.yaml`