Mcp on lo0 — Tech Blog

Making the model prefer our tools: the priority that exists in no layer, the two primitives that do, and the `_meta` that crosses the gateway

Tue, 15 Sep 2026 03:30:00 +0200

A follow-up to LiteLLM’s MCP gateway, which treated the tool catalogue as cost and as surface. This one treats the next question, the one that arrives in the meeting afterwards: so then, how do I get the model to use mine. Verified against LiteLLM 1.102.0 and MCP specification 2026-07-28, with a test of our own run on 14 September 2026.

TL;DR

Tool priority does not exist as a concept in any layer of the stack. Not in the specification, not in the clients, not in the providers’ APIs. What exist are two distinct primitives: removing the alternative from the catalogue, and leaving it behind a search step. Everything else is persuasion, and the model ignores it when it suits.

The specification has no ranking field. The Tool object is name, title, icons, description, schemas, five annotations that are unreliable hints by their own declaration, and _meta. There is a trap I see repeated: MCP does define a numeric priority, but in another structure, the one that applies to content blocks, resources and prompts. It does not govern tool selection.

The only filtering the specification blesses is by credential. The current revision says the set of tools must not vary by connection, and that it may vary according to the authorisation presented. That is exactly what a gateway with per-key lists does.

The client is the one that decides, and only one of them has anything resembling priority. Claude Code lets you exempt a server from deferral with alwaysLoad, so that its tools load in full at start-up while the rest require a prior search. It is not called priority and it works as such.

We proved that the per-tool mark crosses the gateway. We set up an MCP server with one marked tool, served it through LiteLLM 1.102.0 and captured the wire: the _meta arrives whole and with the correct key. The chain is clean on purpose, with a comment in the code that says so.

And it is lost on exactly three routes, which are the gateway’s three catalogue-trimming modes. Proxy mode, tool search driven by key permissions, and the REST listing. In other words: the per-tool mark and LiteLLM’s catalogue trimming are incompatible. You have to choose one.

There is a library trap that makes the field disappear silently. The SDK’s Tool model declares the alias without allowing population by field name, so building it with the field name puts the dictionary in the extras and serialises it under the wrong key. There is no error. The field simply does not arrive.

The cheap lever is the description, not the name. Swapping descriptions shifts the selection distribution substantially; changing only the name has minimal and inconsistent effects. And the gateway’s overrides really do replace what is sent to the client.

Reordering the catalogue is a weak lever. The model is already attending to the correct tool eighty per cent of the time when it fails. Interventions on prompt order repair at most 23 % of the failures.

And the asymmetry you build is attack surface. Whoever controls the order controls the selection: with injection rates of 1.2 % you can take over the head of a tool ranking between 91 % and 97 % of the time.

You are here: the question that arrives after the third server

The previous article closed with a decision ladder for the catalogue: trim, rewrite, split by route, and use progressive disclosure when the catalogue grows. That ladder answers how much catalogue to show.

The question that arrives afterwards is a different one, and it is more uncomfortable. A real agent has native tools from the client hosting it, it has the in-house gateway with the house servers, and it has two or three third-party servers that somebody connected. All of them compete. And what you want is not to show less catalogue: you want that, faced with a query three tools could serve, yours wins. The audited one, the one with access control in place, the one that leaves a trace.

The answer everybody gives is to write a rule in the agent’s instruction file politely asking it to prefer the in-house server. There is even a rule published in a template marketplace that does exactly that. It is text in the prompt and the model follows it when it feels like it.

This article is what lies underneath.

The analogy: the counter with two trays

Let us go back to the switchboard operator from the previous article, the one who also hands out keys. Now the building has three key providers: the house cabinet, the maintenance contractor’s, and the cleaning service’s. All three have a key to the store room and all three open it.

The head of security wants the house one used, because it is the one with a log. And he discovers he can do three things, not one more.

He can take the other two off the counter. It always works and it annoys whoever needed them.

He can leave his own in the front tray and the other two in a drawer, so that to take one you have to ask. It works almost always and annoys nobody.

And he can change the label on his key so that it describes better when it is the right one. It works sometimes, it is the cheapest, and it is the only one that also protects against the contractor changing his label without warning.

What he cannot do is put a number from one to ten on each key and trust the operator to respect it. That number does not exist. The rest of the article is why it does not exist and what to do instead.

What the specification says: nothing

The current revision is 2026-07-28. Read against the source schema, the Tool object has name, title, icons, description, input schema, output schema, annotations and _meta. There is no field for priority, weight, rank, cost, group or tag.

The annotations are exactly five: title, and the read-only, destructive, idempotent and open-world hints. The schema itself warns that all of them are hints and that a client should not make tool-use decisions based on them when they come from untrusted servers.

The `priority` that exists and is no use

Here is the mistake I see repeated most. MCP does define a priority field, numeric between zero and one. It is in the annotations interface that applies to content blocks, to resources and to prompts, alongside the audience and the last-modified date.

It is not in Tool. It has no relation whatsoever to which tool the model picks. Anyone who finds it by searching the word in the schema and concludes that tool priority is standardised has the wrong structure.

The only precedence the specification defines for a tool is one of presentation: to display the name, first the title, then the annotations title, then the name.

The only blessed filtering is by credential

There is a new sentence in the current revision that is genuinely useful and that goes unnoticed. The set of tools must not vary by connection nor as a side effect of other requests, and it may vary according to the authorisation presented in the request.

Translated: segmenting the catalogue by key, by team or by token is the legitimate way to do it. Segmenting it by session state stopped being legitimate when the July revision removed protocol sessions.

Discovery does not discover tools

server/discover, which the current revision adds and which servers have to implement, does not return tools. It returns supported versions, capabilities, server instructions, identity and the cache freshness fields. It accepts neither filters nor a cursor.

Listing is still asking for everything and paginating. The listing request accepts only a cursor: there is no query, no filter, no limit, no groups, no tags.

And on collisions between servers, the specification explicitly and reasonedly washes its hands: clients or proxies that aggregate tools from several servers will encounter collisions and should implement a disambiguation strategy, for example prefixing. And it adds that the server name is not guaranteed to be unique and should not be used to disambiguate. There is no precedence rule between servers. There is not going to be one soon.

What there is in proposals

Nothing accepted. The official proposals index only lists those in a final state, and none of them deals with priority, groups or search. In draft there is one asking for a free-text query in the listing and a filtering capability; another on groups and tags that ended up superseded; and one carrying the word priority but which is per-tool model routing, which is another thing.

The interest group exploring the grouping of primitives declares in its own minutes that it is not going to pick a canonical pattern soon, and the documents in its repository are empty. The year’s roadmap does not mention the problem.

The layer that decides: the client

If the specification gives you nothing, what is left is whatever each client has built on its own. And here there is a clear winner.

Claude Code

Tool search is on by default, and what it does is defer: MCP tools are not loaded in full at start-up, the model sees just enough and discovers them when it searches. It requires a model that supports tool reference blocks, that is Claude Sonnet 4.5, Haiku 4.5, Opus 4.5 and later.

On top of that, the lever:

{
 "mcpServers": {
 "gateway-casa": {
 "type": "http",
 "url": "https://litellm.interna.svc/mcp",
 "alwaysLoad": true
 }
 }
}

The documentation describes it bluntly: if a server’s tools must always be visible without a search step, you set alwaysLoad to true and all of that server’s tools are loaded into context at the start of the session, regardless of the global search setting. And it recommends using it for a small number of tools that the model needs on every turn, because each tool loaded up front consumes context.

That is visibility asymmetry, and it works as priority even if it is not called that. Ours in front, the rest behind one step.

There is an operational consequence that the documentation mentions and that matters more in a gateway than in an ordinary server: setting alwaysLoad makes start-up wait for that server to return its tools, capped by the standard five-second connection timeout. A gateway that aggregates eight upstream servers and does not cache the listing, as is the case, lists live against all eight every time. If one of them is slow, the session’s start-up feels it.

There is also the fine-grained variant. An MCP server can mark individual tools as always loaded by including "anthropic/alwaysLoad": true in the tool’s _meta object, with the same effect for that tool alone. That would let you load three tools from our gateway up front and leave the other forty from the same gateway deferred. It is the lever that matters, and it is the one that motivated the test in the next section.

The rest of this client’s arsenal is pure exclusion, and it is worth knowing one detail that decides whether it works or not. A deny rule with the bare tool name removes it from the model’s context; with parentheses, it does not. The documentation puts it in a table: plain WebFetch removes the tool and the model cannot search at all, whereas WebFetch(domain:*) keeps the tool and rejects every request. The difference is not cosmetic: in the second case the model still sees the tool, still tries it and still spends the context its definition occupies.

The --tools flag is the scalpel: it accepts the empty string to disable all the built-ins, and it does not affect MCP tools. That is, --tools "" leaves the model with only what the gateway serves. It is a command-line flag and has no equivalent as a settings-file key; the functional equivalent there is deny rules.

The other clients

Client	Disable the native ones	Filter the MCP ones	Priority
Claude Code	`--tools ""`, deny by bare name	deny by pattern, subagents	`alwaysLoad`
Codex CLI	`features.shell_tool` to false	`enabled_tools` and `disabled_tools` per server	No
Gemini CLI	`tools.core` as an allowlist	`includeTools` and `excludeTools` per server	No
Copilot in VS Code	Not documented	picker and tool sets	No
Cursor	No	per-server switches	No

Two notes. Copilot cuts off at 128 tools per request and its virtualisation threshold does not go above that; its embedding-based routing is internal and has no configuration surface. And Cursor’s rules are purely indicative: they are text prepended to the context.

The API layer

If the client is in-house, the margin is wider, though narrower than it looks.

Forcing a set is not possible. In Anthropic’s case, tool choice accepts auto, any, a specific one by name, or none, and the documentation says explicitly that pointing at a set of MCP tools or at a member is not supported. In vLLM it is the same: a specific one, all of them or none. There is no way to say “any of mine from the gateway”. The proposal that would introduce grammars restricting tool names to the request’s set has been open for months and is not merged.

Filtering the catalogue is possible, in all three of the big ones. In Anthropic, with an MCP tool-set block where the default configuration disables and specific ones are enabled. In OpenAI, with the allowed tools list inside the MCP block itself, which the documentation justifies on latency and cost grounds, avoiding the model seeing unnecessary definitions. And in Gemini, the remote MCP server configuration accepts restricting which of the server’s tools the agent may call.

It is worth not confusing two things that are named almost identically in OpenAI: the allowed tools list inside the MCP block filters the catalogue, and that is what matters here; the identically named tool-choice type is another thing, it is a selection restriction, and there is evidence that it does not combine well with hosted tools, without the current documentation recording it. If anyone depends on that, let them test it against their model: it fails as a request error, so it shows up on a cold start.

And there is a 2026 lever that almost nobody is using. Anthropic has in beta a mechanism for mid-conversation tool changes, with add and remove blocks that accept referencing an entire MCP set. The reason it exists is in its documentation: the tools array sits even earlier in the chunked prefix than the system field, so editing it invalidates the cache for the whole conversation; by declaring the full set at the start and using the blocks, the array never changes and the cached prefix stays intact.

Applied to our case: you can remove a rival server’s set hot, mid-session, without paying for the whole prefix. It is not available on every model.

The test: the `_meta` crosses the gateway

The fine-grained variant of Claude Code’s lever depends on a question no documentation answers: whether a gateway that aggregates upstream servers propagates each tool’s _meta all the way to the client, or loses it along the way.

We checked.

The setup

A minimal MCP server with the Python SDK, serving two tools over HTTP, one of them with the mark in place:

types.Tool(
 **{
 "name": "always_loaded_tool",
 "description": "tool with _meta",
 "inputSchema": {"type": "object", "properties": {}},
 "_meta": {"anthropic/alwaysLoad": True, "probe": "UPSTREAM_META_MARKER"},
 }
)

In front of it, a LiteLLM from the main branch with that server declared in config.yaml, with no database, with a master key. And an MCP client querying the listing against /mcp.

The result

This is what comes out on the wire:

{
 "_meta": {"litellm.ai/server_outcomes": {"probe": {"status": "ok", "tool_count": 2}}},
 "tools": [
 {
 "name": "probe-always_loaded_tool",
 "description": "tool with _meta",
 "inputSchema": {"type": "object", "properties": {}},
 "_meta": {"anthropic/alwaysLoad": true, "probe": "UPSTREAM_META_MARKER"}
 }
 ]
}

The mark arrives whole and with the correct key. The client parses it without trouble.

Tracing the code, the chain is clean on purpose. LiteLLM keeps the SDK objects as they are when it lists against the upstream, instead of rebuilding them. The name prefixing does a deep copy and mutates only the name, with a comment in the code declaring the intention to preserve every field including the _meta by avoiding mutation. The two permission filters are list comprehensions that reuse the same objects. And the name and description overrides mutate in place, so they do not lose anything either. Somebody thought about this.

One operational detail: the name arrives prefixed with the server alias, governed by the configurable separator and the short prefix mode.

The three routes where it is lost

And here is the finding that changes the previous article’s recommendation.

Proxy mode returns only its three fixed tools, with no _meta and with no upstream tool at all. Verified against the wire as well.

Tool search activated by key permissions does the same with four virtual tools.

And the REST listing builds its response object by hand and discards the field: the tool that did carry a mark comes out with the field null. The class inherits from the type that has the field; it simply is not filled in. The fix fits on one line.

Those three routes are, exactly, the three ways LiteLLM has of trimming the catalogue. The operational conclusion is uncomfortable and it is worth saying plainly: either you use the gateway’s catalogue trimming, or you use the client’s per-tool mark. They cannot be combined. The previous article recommended proxy mode as the fifth rung of the ladder; with this in hand, that rung and the fine-grained mark are mutually exclusive.

The library trap

This deserves its own section because it takes out anyone writing a server or a proxy, and it gives no warning.

The SDK’s tool model declares the field with an alias, and does not enable populating it by name. The consequences, both checked by running them:

Building with the field name populates nothing. The dictionary slips into the model’s extras and comes out serialised under the wrong key, without the leading underscore, which is a key no client interprets.

And dumping and revalidating without asking for aliases loses the field, for the same reason.

There is no exception, no warning in the log, nothing. The field disappears. If someone marks their tools and does not see them marked in the client, this is the first place to look, before the gateway.

What we have not tested

For honesty’s sake, and because it is the link that remains: we have shown that the mark survives the gateway. We have not shown that the client acts on it when the tool arrives with the name prefixed by the gateway. Claude Code’s documentation describes the per-tool mark in a single sentence and says nothing about aggregating servers or about prefixes.

It is a half-hour test for anyone who has the setup in front of them: two tools from the same gateway, one marked, and look at which one appears loaded at the start of the turn. If someone does it before I do, I am interested in the result.

What LiteLLM can and cannot do

With the above, the gateway’s inventory of levers comes out like this.

Lever	Serves external MCP clients	Preserves the `_meta`
`allowed_tools` and `disallowed_tools` per server	Yes	Yes
Name and description overrides	Yes	Yes
Tool sets per route	Yes	Yes
Proxy mode and virtual tools	Yes	No
Search by key permissions	Yes	No
REST listing	Yes	No
Semantic filter	No	Not applicable

Two clarifications on the previous article’s table, now that the code has been read again.

The allowed and disallowed lists are applied in the listing, not only on the call. They are the blunt, effective instrument, and they do not break the cache because they are static.

And the overrides have a limit that was not accounted for: they are applied in the MCP protocol listing, but they are skipped in proxy mode and not applied on the REST route. Anyone rewriting descriptions and also turning on proxy mode is not serving what they think.

What the gateway cannot do is inject _meta. There is no setting equivalent to the ones for name and description. The mark has to be put there by the origin server, which for in-house servers is trivial because we write them. To mark third-party tools you would have to patch, and it is small: one field in the type and three lines in the overrides function if configuring it by file is enough.

There is also an ordering lever that did not appear in the previous article. In the proxy’s search mode there is a core tools setting that puts them at the front of the ranking and that also does not count against the results cap. It is the closest thing to a declarative priority in the whole stack, and it lives inside the one mode that discards the _meta.

And a nuance about proxy mode’s results cap. The client cannot ask for more than five, that is true, but the operator can raise it by configuration. The previous article implied it was immovable.

The cheap lever: the description, not the name

If you cannot exclude, what is left is to bias. And bias comes in through the description.

The work that measures selection bias between functionally equivalent tools puts the combined bias of the evaluated models between 0.25 and 0.38, that is, you would have to redistribute between 25 % and 38 % of the probability mass for equivalent tools to be picked equally. On that basis, swapping two tools’ descriptions shifts selection substantially, whereas perturbations that touch only the name produce smaller and more inconsistent effects.

Another piece of work measures that a single pass of automatic description rewriting improves the metric over a large corpus, almost as much as iterative refinement, and with two orders of magnitude less time. And it adds a warning worth retaining: descriptions tuned against a fixed set of candidates do not generalise to the dynamically retrieved set. You have to optimise for the regime you serve in.

Applied: the gateway’s description override is at once token trimming, selection bias and mitigation of description change after approval. It is the lever with the best effort-to-effect ratio in the whole article, and it is static, so it does not touch the prompt cache.

Name prefixing, by contrast, costs tokens and probably does not change which tool the model goes to.

What does not work

Reordering the catalogue. A June paper measures that, when the model fails, it was already attending to the correct tool eighty per cent of the time, well above chance. The bottleneck is not at the input but in the late layers. Interventions on prompt order recover at most 23 % of the failures, against the 59 % to 91 % of those acting on the final readout. Reordering is cheap and that is why it gets recommended a lot; it is also weak.

Rules and instructions. They are text. They help and they do not decide.

Your own per-request top-K filters. This is covered in the previous article and it still holds: four out of five measured strategies perform worse than not filtering, and on top of that rewriting the tool block on every turn invalidates the entire cached prefix.

The ladder, updated

Exclude in the client. It is the only deterministic thing. --tools "" or deny by bare name for the competing native tools.
Mark your own as always loaded. At server level with alwaysLoad, or per tool with the mark in the _meta if you control the origin server. Counting on start-up waiting for the server.
Trim with static lists per server and per key, and split a catalogue by agent profile across routes.
Rewrite the descriptions with the gateway’s overrides.
Remove hot the rival sets, if the client is in-house and the model supports it.
Choose: the gateway’s catalogue trimming, or the per-tool mark. Not both.
Do not: reorder, rename, or trust prompt rules.

The risk that has to be declared

The asymmetry you build is also a surface. Whoever controls which tools head a ranking controls the selection, and that can be attacked: there is work measuring that by injecting adversarial tools at rates of 1.2 % you take over the head of the ranking between 91 % and 97 % of the time.

In a system under ENS (Esquema Nacional de Seguridad, Spain’s national security framework) this fits under change management, not access control: the set of tools a model sees and the order in which it sees them are security configuration, and today nobody versions them. Neither does the specification fix a hash of the description or of the schema, nor does the gateway. The specific Annex II codes are worth checking against the current text before taking them into a compliance document; the approach is developed in the MCP gateway article.

There is a precedent within the ecosystem itself that points the way. The MCP skills extension, which is in a final state, requires a manifest with a per-file hash, mandatory verification before use, and approval tied to the set of files and their digests, so that any change revokes the approval. And it obliges hosts to prevent two skills with the same name from silently replacing one another. It is exactly the pattern missing for tools. Anyone who needs it today has to implement it in their gateway.

Checklist

Decide the strategy before touching anything: catalogue trimming in the gateway, or the per-tool mark in the client. They are mutually exclusive.
If you pick the mark, put it in the origin server, because the gateway cannot inject it.
Build the tool object with the alias key, never with the field name, or the field disappears without warning.
Do not revalidate tool objects from a dump without asking for aliases.
Count on marking a server as always loaded making session start-up wait for that server, and on a gateway listing live against all of its upstreams.
Exclude in the client the native tools that compete, by bare name and not with a pattern in parentheses.
Rewrite the descriptions of your own tools, and do not waste time renaming.
Do not turn on proxy mode if you depend on the description overrides, because they are not applied there.
Version the exposed tool set and its order as security configuration, with a hash of the description and the schema.
If you use the gateway’s REST route for anything, know that it discards the _meta.

Traps and things that are not what they look like

MCP’s priority exists, and it is not about tools. It lives in the annotations for content, resources and prompts.
Tool annotations are hints and the schema itself says they are not reliable from untrusted servers.
The specification gives no precedence rule between servers, and it says besides that the server name is no use for disambiguating.
The tool set may indeed vary by authorisation, and that is the only blessed segmentation.
server/discover does not return tools.
A deny rule with parentheses does not remove the tool from the context, it only rejects the calls.
--tools is a command-line flag, with no equivalent in the settings file.
Marking a server as always loaded delays start-up by up to five seconds per server.
Tool(meta=...) does not populate the field and serialises it under the wrong key.
Proxy mode, search by permissions and the REST listing discard the _meta.
Description overrides are skipped in proxy mode and are not applied on REST.
Proxy mode’s cap of five results can be raised by the operator, even though the client cannot ask for more.
The core tools of the proxy’s ranking are the only declarative priority in the stack, and they live in the mode that loses the _meta.
You cannot force “any of mine from the gateway” in any API.
In OpenAI, the allowed list inside the MCP block and the identically named tool-choice type are different things.
Gemini does accept a per-tool allowlist on its remote MCP servers.
Reordering the catalogue repairs at most 23 % of the failures.

Closing

The starting question had a trap in it, and the trap is the word. When someone asks how you prioritise an MCP server, they are assuming a dial exists somewhere. It does not, and I have spent the whole article showing the places where it is not.

What there is instead is poorer and more manageable: you remove what competes, you leave your own in front, and you write the description better. Three things, none of them elegant, all three effective in that order.

What does seem to me worth taking away is the shape of the problem. The tool catalogue a model sees is security configuration in the full sense, because it determines what the agent can do and which system it is going to talk to. And today it is managed like a list of connections: things get added, it is not versioned, it is not signed, and nobody detects a change. The ecosystem has already solved that problem once, for skills, with hashes and tied approval. For tools, not yet.

In the meantime, the asymmetry has to be built by hand, server by server, and knowing that it rests on the _meta of an object that a library can empty without saying anything.

Sources

Model Context Protocol, herramientas en la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/server/tools.
Model Context Protocol, server/discover: https://modelcontextprotocol.io/specification/2026-07-28/server/discover.
Model Context Protocol, paginación: https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/pagination.
Model Context Protocol, caché de listados: https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/caching.
Model Context Protocol, esquema fuente de la revisión: https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/schema/2026-07-28/schema.ts.
Model Context Protocol, extensión de habilidades: https://modelcontextprotocol.io/extensions/skills/overview.
Model Context Protocol, grupo de interés sobre agrupación de primitivas: https://modelcontextprotocol.io/community/interest-groups/primitive-grouping.
Anthropic, Claude Code y MCP (alwaysLoad, búsqueda de herramientas): https://code.claude.com/docs/en/mcp.
Anthropic, Claude Code, referencia de línea de comandos (--tools): https://code.claude.com/docs/en/cli-reference.
Anthropic, Claude Code, permisos (denegación por nombre pelado): https://code.claude.com/docs/en/permissions.
Anthropic, conector MCP y conjuntos de herramientas: https://platform.claude.com/docs/en/agents-and-tools/mcp-connector.
Anthropic, cambios de herramientas a mitad de conversación: https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages.
Anthropic, búsqueda de herramientas: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, MCP y conectores (lista de herramientas permitidas): https://developers.openai.com/api/docs/guides/tools-connectors-mcp.
OpenAI, búsqueda de herramientas y caché: https://developers.openai.com/api/docs/guides/tools-tool-search.
Google, llamada a funciones en la Interactions API: https://ai.google.dev/gemini-api/docs/function-calling.
vLLM, llamada a herramientas y decodificación restringida: https://docs.vllm.ai/en/latest/features/tool_calling.html.
vLLM, propuesta de decodificación guiada por regiones y gramáticas de herramienta: https://github.com/vllm-project/vllm/issues/39848.
LiteLLM, código del gateway MCP: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, tool_search.py, litellm/experimental_mcp_client/tools.py: https://github.com/BerriAI/litellm.
SDK de Python de MCP, modelo Tool y serialización por alias: https://github.com/modelcontextprotocol/python-sdk.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307v2.
Looking Is Not Picking, atención frente a lectura final y techo de las intervenciones sobre el prompt: https://arxiv.org/html/2606.16364v1.
A Single Rewrite Suffices, reescritura de descripciones y generalización al conjunto recuperado: https://arxiv.org/html/2606.30775.
ToolFlood, saturación adversaria de la capa de recuperación: https://arxiv.org/html/2603.13950.

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

Sat, 12 Sep 2026 09:00:00 +0200

A follow-up to Keycloak in an AI platform, which flagged the gap: resource indicators are not supported and protected resource metadata belongs to the MCP server. This post tries to close it. Verified against revision 2026-07-28 of the specification, the official Python and TypeScript SDKs, Keycloak 26.7.3 and LiteLLM 1.102.0.

TL;DR

The specification splits three obligations and the authorisation server covers only one. The MCP server must publish its protected resource metadata. The client must send the resource indicator in both requests, and must do so even if the authorisation server does not support it. The MCP server must validate that the token was issued for it. Keycloak does not understand the indicator and considers the metadata someone else’s, so all three end up in the resource layer.

The Python SDK mounts the metadata on its own, but degraded. As soon as the resource server URL is configured, the route appears. What that automatic route publishes carries a single authorisation server, reuses as its scope catalogue the ones the middleware requires, and leaves the name and the documentation null. To publish a complete document you have to mount the route by hand.

Audience validation comes switched off in Python and does not exist in TypeScript. In Python, if the resource URL is configured and the corresponding flag is not enabled, the SDK emits a deprecation warning and behaves as if it were disabled; the docstring itself promises that version 3 will set it to true. In TypeScript the bearer verification does three things, and none of them is looking at the audience. Neither of the two ships a signature verifier with public keys.

Token passthrough is forbidden in writing, and the substitute has a name. The specification says the MCP server must not accept or relay tokens that were not issued for it, and that if it calls upstream APIs the token must be a different one. The mechanism is token exchange, which Keycloak supports in its standard version with a limitation you need to know: the audience is a client identifier, not a resource URL.

There is an official extension that changes the whole picture, and it is stable. Enterprise-managed authorisation replaces the redirect to the MCP authorisation server with an exchange at the corporate identity provider, which evaluates the policy and issues a grant based on an identity assertion. It has been stable since June 2026 and there are servers in production.

Keycloak implements it halfway and in experimental. It only knows how to act as receiver, not as issuer, behind a feature flag, and against draft 01 when the working group is on 04. The official documentation literally says not to use it in production.

LiteLLM already implements it on the client side and does not document it. In 1.102.0 there is an authentication mode towards MCP servers that runs the two stages of that grant. There is not a single documentation file that mentions it.

You are here: the resource side

The previous post walked through the identity provider. This one walks through what sits on the other side of the token, which is where the MCP specification puts almost all its normative obligations.

The natural reading order is When MCP grows up for the basic setup, Keycloak in an AI platform for the identity piece, and this one for the missing layer. The MCP gateway seen from the inside is in its own post.

The analogy: the visa and the border checkpoint

A consulate issues visas. A border post checks them. They are two different offices, and the classic mistake consists of assuming that because the visa is genuine it is good for entering through any door.

The visa has written on it which country it is good for. That is the token’s audience, and here the problem appears: the consulate we work with does not know how to write the destination the traveller asks for, because it does not understand that field on the form. It knows how to write a destination if asked for it under a different name, by way of a pre-agreed stamp, which is the workaround of scopes and the audience mapper.

And then there is the other half, the one almost nobody builds. The border post has to exist, it has to announce where it is and from which consulates it accepts visas, and it has to read the destination written on the visa before letting anyone through. If the border post limits itself to checking that the stamp is genuine and does not look at the destination, any valid visa from the same consulate is good for entering. That is exactly what the two official SDKs do by default.

Part 1. The split of obligations

From revision 2026-07-28, with the normative words as they are:

Obligation	Whose	Status with Keycloak
Publish protected resource metadata (RFC 9728)	MCP server, MUST	Out of its scope, by design
Send the resource indicator (RFC 8707) in authorisation and in token	Client, MUST, even if the authorisation server does not support it	It does not understand it
Validate that the token was issued for oneself	MCP server, MUST	The resource’s responsibility
Not accept or relay other tokens	MCP server, MUST NOT	The resource’s responsibility
Validate the issuer of the authorisation response (RFC 9207)	Client, MUST; authorisation server SHOULD emit it	Supported
Consent for each dynamically registered client	MCP proxy with a static identifier, MUST	Configurable

The most confusing part of that split is the second line. The specification obliges the client to send the parameter regardless of whether the authorisation server supports it. With Keycloak, that parameter is lost: the official documentation says it cannot recognise it, and the standard behaviour towards an unknown parameter is to ignore it. The client complies, the token comes out, and what does not come out is the correct audience. The failure appears later and somewhere else, which is the worst kind of failure.

It is worth knowing where that work stands. The matter is issue 14355 of the project, open, with a milestone in 26.8.0. There was a complete implementation in pull request 35711, with a mapper of its own and an extension point for resolving resources, which went back to draft in October 2025 because of a total lack of tests and because it was decided to redo it in phases, starting with a single resource per client. In March 2026 a new issue was opened for experimental support, also with a milestone in 26.8.0 and with no associated pull request. In 26.7.x there is no feature flag that enables it.

What is new in the revision and affects the setup

Four things changed with respect to the previous revision and are worth collecting:

RFC 9207. A new section on validating the authorisation response. The client must record the issuer from the validated metadata document, and the comparison is literal: you are not allowed to normalise case, elide the default port, add or remove the trailing slash or re-encode characters before comparing.
Offline tokens leave the catalogue. A new section on refresh tokens: MCP servers should not include the corresponding scope either in the challenge header or in the supported scopes of their metadata.
Dynamic registration becomes deprecated. It goes from optional to optional and deprecated, retained for compatibility with authorisation servers that do not support client identifier metadata documents. And a new requirement is added to registration: the application type is mandatory, and a client must not reuse credentials from another authorisation server, it has to register again.
Step-up elevation is rewritten. The menu of three server strategies disappears and is replaced by two rules: the challenge’s scope attribute describes what is needed for the requested resource, with no obligation to include what has already been granted, and scope accumulation becomes the client’s responsibility. With a new requirement for the server: it must take scope hierarchies into account, where a broad one implies the narrow ones.

Part 2. What the SDKs give, and what they do not

Here is the uncomfortable part, and it is the main reason for writing this post. Verified by reading the code of the Python SDK and of the TypeScript 2.0.0-alpha one.

Protected resource metadata

In Python the complete model exists and it mounts on its own. The metadata class carries the RFC fields: resource, authorisation servers with a minimum of one, key set URL, supported scopes, bearer methods with a default value of header, name, documentation, policy, terms, and the client certificate and DPoP fields. The handler serves the document with a one-hour cache directive, and the URL is built by inserting the well-known path in front of the resource path, as the RFC requires.

As soon as the resource server URL is configured, the route appears with no further work. But what that automatic route publishes is degraded in three points: it passes a single authorisation server, it takes as its scope catalogue the ones the middleware requires (which are not the same as the ones the resource supports), and it passes neither the name nor the documentation, which come out null. To publish a complete document you have to call the route creation function by hand.

In TypeScript it exists and does not mount on its own. The function that builds the document emits only five fields: resource, authorisation servers, supported scopes, name and documentation. It does not emit the supported bearer methods, nor the key set URL, nor anything about DPoP. And serving it is explicit: either you call the response function from your own handler, or you mount the metadata router. There is no point in the SDK that does it on its own.

Audience validation

This is the one to fix on day one.

In Python it comes disabled. The token verifier is a single-method protocol, that is to say a hole the implementer fills. The access token’s resource field is the indicator you set yourself. And the comparison only happens if you ask for it: there is a function that normalises as a URL and ignores the trailing slash, but the resource server URL reaches it empty unless the validation flag is enabled.

The detail to read twice is in the configuration: if the resource URL is set and the flag is not, the SDK raises a deprecation warning and behaves as if the flag were false. The docstring says version 3 will set the default to true. Until then, a Python server configured with authorisation accepts tokens issued for another resource, unless the implementer enables the flag or validates the audience inside their own verifier.

In TypeScript it is not validated at all. Bearer token verification does exactly three things: strip the prefix, check the required scopes, and require that the expiry is present and not past. A search for audience or key set across the server and middleware packages returns no validation. It is all left in the hands of whichever verifier you plug in.

Neither of the two ships a signature verifier with public keys. In Python the only example is introspection against the authorisation server, and its audience check sits behind a flag that also comes out false. In TypeScript the interface is empty. That is: the component that validates the token, which is the one holding up the most important normative requirement of the specification, is your own code in both cases.

The authentication challenge

Python builds the challenge with the error and its description, and adds the metadata URL only if it is configured. It returns 401 with invalid token and 403 with insufficient scope, both from the same function, so both carry the metadata URL. What it never emits is the scope parameter, which revision 2026-07-28 asks to be included, and which is exactly what the client needs in order to know what to ask for.

TypeScript does emit the scope when there are required scopes, in addition to the metadata URL, and it correctly maps invalid token to 401 and insufficient scope to 403.

That is, each SDK gets a different half right. The Python one validates better and warns worse; the TypeScript one warns better and does not validate.

Scopes, which are not per tool

In both SDKs scopes are a static list at the transport mounting level, not per tool. There is no per-tool scope declaration and no step-up elevation help on the server side. Emitting a 403 with the specific scope that call requires, which is what the specification describes, is your own code.

That is the origin of almost all the fine-grained authorisation you have to build, and it is why there is a Part 6.

Which revision each one announces

Python is already on 2026-07-28. The revision appears in the known versions and in the list of modern versions, described as the ones that use the stateless envelope per request. The server discovery method is registered with a default handler, and the method and name headers are validated against the body.

TypeScript has it, but in a separate list. The latest protocol version constant is still 2025-11-25, because that list is only the one for the initial handshake. The modern era lives in a separate module, with a comment explaining the reason: keeping them deliberately separate so that adding a revision there never leaks a modern version string into a 2025-era handshake.

And it is worth recalling the mismatch that already came up in the MCP gateway post: LiteLLM 1.102.0 still announces 2025-06-18.

Part 3. Building the protected resource

With the above, the list of what you have to write is short and concrete.

1. A token verifier that validates the signature and the audience. Against the realm’s key set, checking issuer, expiry and that the audience contains the MCP server’s canonical URL and only things that concern it. Here the side effect of the Keycloak workaround comes in: if a client asks for two scopes from two different resources, out comes a token with two audiences, because each mapper contributes its own to the array. There is no client policy condition or executor in 26.7 that limits the number of audiences per token. The practical defence is the resource’s: reject tokens whose audience includes other resources, instead of merely checking that its own is present.

2. The complete metadata document, with all the authorisation servers you accept, the real catalogue of scopes the resource understands, and the name and documentation filled in. In Python, mounting the route by hand instead of leaving the automatic one. In TypeScript, mounting it, full stop.

3. The authentication challenge with the scope. In Python you have to add it, because the SDK does not emit it. And you have to take scope hierarchies into account, which is a new requirement of the revision.

4. The decision between local validation and introspection. Validating the signature locally costs no network, but revocation does not take effect until the token expires: there are no revocation lists for signed tokens. Introspection costs a round trip per request and only confidential clients can invoke it. The reasonable combination is a short token life and introspection on the operations that change state.

5. Binding state to the user. The new revision is explicit about this, because with protocol sessions gone, state is carried in handlers that travel as an ordinary tool argument. Servers must verify every incoming request and must not treat possession of a handler as authentication; and they should bind the handler to the user on the server side, for example storing the state under a key that combines the user identifier derived from the verified token with the handler, and reject the handler if someone else presents it. It is a change in the way of working compared with the sessions of before.

Part 4. The hop from the gateway to the MCP server

The real shape in an inference platform is not client against MCP server: it is client against gateway, and gateway against MCP server. That second hop is where it is decided whether the architecture is correct.

The specification leaves no room. The MCP server must not accept or relay tokens that were not issued for it. And if it makes requests to upstream APIs, it may act as an OAuth client of theirs, but the token it uses there is a different token, issued by the upstream authorisation server, and it must not forward the one it received.

The standard mechanism for that is token exchange, and here you need to know three Keycloak details.

First, the permission model changed. The old version required fine-grained administration permissions and an explicit exchange authorisation on the target client. The standard version, supported since 26.2, does not require them: it is enough for the requesting client to be confidential and to have the corresponding switch enabled. In exchange there is a surprising condition: the subject token has to carry the requesting client in its audience, unless it exchanges its own token. That is, the gateway needs to appear in the audience of the user token, which is achieved with an audience mapper in a default scope of the gateway. Only then can it narrow the audience to that of the specific MCP server.

Second, the audience is a client identifier, not a resource URL. The parameter filters audiences, that is to say it narrows them, which is what you want. But it takes the identifier of a client registered in the realm. The practical consequence is that the MCP server has to be registered as a client, and its identifier should be its canonical URL, so that the resulting audience matches what the resource validates. It is a naming trick, and it has to be documented for whoever comes after. The documentation itself acknowledges it: token exchange does not yet support the resource parameter.

Third, the subject token type is limited. The standard version only accepts access tokens as subject.

There is also a new experimental feature in 26.7 that adds a parameterised scope type for validating whether the requesting user is authorised to act on behalf of another. Interesting for delegation, but experimental.

What the gateway does today

In LiteLLM 1.102.0 the authentication modes towards MCP servers number twelve, with a default value of none. The ones that matter for this discussion are four.

oauth2_token_exchange implements the standard exchange. It sends the correct grant type, the subject token and its type, and the audience, never the resource. The omission is deliberate according to the code itself: fabricating a target risks an invalid target error. It caches the resulting token under a key that combines the subject token with the whole configuration, and with a lifetime equal to the token’s minus one minute. And it blocks, never degrades: with no incoming token it returns 401, which the edge turns into a challenge with the resource metadata URL; a rejection from the identity provider returns 401; a gateway configuration error returns 500; a transport failure, 503.

true_passthrough and oauth_delegate forward the client’s authorisation header as is. That is token passthrough, by the name the specification gives it, and its legitimate use is narrow: when the token the client presents was already issued for the target MCP server and the gateway is a mere transport. Outside that case, it is non-compliant.

oauth2_id_jag is the surprise, and it deserves a part of its own.

Part 5. The enterprise extension, which changes the picture

In June 2026 the MCP project published an authorisation extension that solves a different problem from the one the standard flow solves, and which in an organisation with an identity provider of its own is the real problem.

The normal flow is per user: each employee authorises each client against each MCP server. It works for consumer applications and it does not work in a company, because onboarding a person requires authorising dozens of services one by one and offboarding requires revoking them one by one.

The io.modelcontextprotocol/enterprise-managed-authorization extension inverts that. It is in the stable specifications directory of the extensions repository, it comes from SEP-990, and the flow is this:

The MCP client authenticates the user against the corporate identity provider through the normal flow, and stores the identity assertion, which may be an OpenID identity token or a SAML assertion.
When the server indicates that enterprise-managed authorisation is needed, the client exchanges that assertion at the corporate provider for an authorisation grant based on an identity assertion. The provider evaluates the organisation’s policy there: group membership, roles, conditional access.
The client presents that grant to the MCP authorisation server and obtains an access token.
The sentence that defines the extension, verbatim: the user is not redirected to the authorisation endpoint of the MCP authorisation server.

The MCP authorisation server validates the signature against the corporate provider’s key set, plus audience, issuer and expiry, and uses the subject as the user’s stable identifier, with the email as a fallback for linking earlier accounts.

What that buys is exactly what a compliance file asks for: policy in a single place, an auditable decision at the identity provider, and centralised revocation that takes effect across all clients at once. The employee who loses access stops receiving grants, without touching any server.

The underlying standard is a draft from the IETF’s OAuth working group, under the name of JWT identity assertion authorisation grant, revision 04 of May 2026, signed by people from Okta, Ping Identity and an independent author. It profiles identity chaining across trust domains by combining token exchange with the JWT profile.

And here comes the problem

Keycloak implements it only halfway. It has its own documentation page, behind a feature flag. And three limitations you need in front of you before designing anything:

It only acts as receiver. It accepts assertions issued by an external provider and issues local tokens. Native support for acting as issuer, the documentation says, is not yet fully implemented. In the extension’s flow, the issuer is the corporate provider: if that provider is Keycloak, the missing piece is exactly the one you need.
It is based on draft 01 while the working group is on 04.
The official documentation says not to use it in production.

The asymmetry is striking and worth pointing out: Keycloak does not support a standard published eight years ago such as resource indicators, and does partially support a draft from a few months back.

What is already in the gateway, undocumented

In LiteLLM 1.102.0, the oauth2_id_jag mode runs the two stages of the flow. In the first it asks the corporate provider for a token exchange with the requested token type specific to this grant, sending the user’s identity token as subject and, optionally, audience, resource and scope. In the second it presents the resulting assertion to the resource’s authorisation server with the JWT bearer grant type. In both the gateway authenticates as a client with a JWT signed by private key.

Two operational warnings that come out of the code and are nowhere else:

The subject comes from the caller’s identity token, or from the one captured at login. And only the generic OIDC provider captures assertions: with Google, Microsoft or SAML there is none, and all users fail. The code raises a warning when loading the configuration.
A badly configured server refuses with a 500 instead of falling back to the static credential. It is the right decision, and it is worth knowing before it happens.

And the main warning: there is no documentation. Not a single file in the repository that mentions this mode. Anyone using it is reading code, like this post.

Part 6. Per-tool authorisation

None of the previous pieces answers the question the business ends up asking: whether this person may call this tool with these arguments. The identity provider gives roles and scopes, and with that you do not model a graph of relations, as was already argued in the previous post.

What is published today, with sources:

OpenFGA is the only one with a dedicated product guide, updated on 9 September 2026. The pattern is a tool type with an invocation relation, checked on each call, and a listing query so the client only sees the tools it may invoke. It is a pattern and a modelling language, not a reference implementation.
Pomerium ships a per-tool policy criterion, with exact, prefix, suffix and list matching.
Kong offers it as a product, with per-tool control lists.
Traefik Hub goes as far as parameter-level restrictions, which is the finest published grain.
Envoy has an MCP filter that extracts protocol attributes for fine-grained access control, declared to be in active development and with no OAuth of its own: the hook is external authorisation towards a policy engine.

The lowest-friction option in a platform that already has a service mesh is external authorisation against a policy engine. The most expressive one, the relations model.

Part 7. What still has no standard

Two gaps the specification does not cover and that have to be closed from outside.

Tool definition change. Neither revision 2026-07-28 nor the SDKs pin a hash of the description or of the input schema. The best practices page covers confused deputy, token passthrough, server-side request forgery in discovery, state handler hijacking, server confusion and validation of the authorisation URL scheme, but there is no integrity or pinning requirement. The tool that does it is the Invariant Labs scanner, under the Apache 2.0 licence, which pins hashes to detect definition change after approval and also ships a proxy mode with guardrails. It already came up in the MCP gateway post and it remains the only answer.

The public guidance that does get into this is the NSA’s on MCP security design considerations, from May 2026, done with Carnegie Mellon’s software engineering institute. Two of its observations are directly actionable: that authorisation in MCP is optional and that the specification imposes no requirement on token lifecycle management, so expiry and rotation are left in the organisation’s hands; and the warning against dynamic tool discovery with no origin verification and no authorisation checks, which is the most direct acknowledgement of the risk in a state guidance document.

Part 8. Who covers what

With all of the above, the decision table for the layer you put in front of the MCP server:

Piece	Publishes resource metadata	Validates audience	Token exchange	Per-tool authorisation
Python SDK, automatic route	Yes, degraded	Not by default	No	No
TypeScript SDK	Does not mount on its own	No	No	No
Traefik Hub (commercial)	Yes, automatic	Not documented	Not documented	Yes, down to parameter
Kong, OAuth plugin for MCP	Not documented	Yes	Not documented	Yes, per-tool lists
Pomerium	Partial	Not documented	Not documented	Yes, per-tool criterion
agentgateway (Solo.io)	Yes	Yes, by JWT policy	Yes, including the assertion grant	Yes, with expressions
mcp-context-forge (IBM)	Not in release notes	Not documented	Yes, since 1.0.6	Yes, per-role control
Envoy, MCP filter	No	No	No	Via external authorisation
LiteLLM 1.102.0	Yes, and authorisation server metadata too	Not on the incoming JWT	Yes, and the assertion grant	Yes, per-key and per-team permissions

Two warnings about that table. The Kong plugin requires a minimum version of 3.12, is in technical preview and needs a licence for its AI edition; it also introduces a breaking change in 3.13, which starts treating all traffic as MCP in order to close a possible authentication bypass. And on oauth2-proxy I found no primary documentation either for or against protected resource mode: I am not claiming it does not have it, I am claiming it is not documented where I could look.

Reference configuration

Keycloak: the per-resource scope, optional, with its audience. That it is optional and not default is the key piece, because it is what makes the client have to ask for it explicitly and makes it work as a substitute for the resource indicator:

kcadm.sh create client-scopes -r plataforma \
 -s name=mcp:inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

And the MCP server registered as a client with its canonical URL as identifier, so that token exchange produces the right audience:

kcadm.sh create clients -r plataforma \
 -s clientId=https://mcp.ejemplo.es/inventario \
 -s enabled=true -s publicClient=false -s consentRequired=true

The gateway, with token exchange instead of token passthrough:

mcp_servers:
 inventario:
 url: "https://mcp.ejemplo.es/inventario"
 transport: "http"
 auth_type: "oauth2_token_exchange"
 client_id: "litellm-gateway"
 client_secret: os.environ/GW_SECRET
 token_exchange_endpoint: "https://sso.ejemplo.es/realms/plataforma/protocol/openid-connect/token"
 audience: "https://mcp.ejemplo.es/inventario"
 subject_token_type: "urn:ietf:params:oauth:token-type:access_token"
 token_exchange_profile: "rfc8693"

The resource verifier, which is what nobody hands you ready-made. The essential bit is the last check, the one that rejects other resources’ audiences instead of settling for finding its own:

RECURSO = "https://mcp.ejemplo.es/inventario"

async def verificar(token: str) -> AccessToken | None:
 claims = jwt.decode(
 token, await jwks(),
 algorithms=["RS256"],
 audience=RECURSO, # validates that I am there
 issuer=ISSUER,
 )
 aud = claims["aud"]
 aud = [aud] if isinstance(aud, str) else aud
 # and that nobody else is: the scopes workaround allows
 # tokens with two audiences, and that reopens the confused deputy
 if set(aud) - {RECURSO, CLIENTE_GATEWAY}:
 return None
 return AccessToken(
 token=token,
 client_id=claims["azp"],
 scopes=claims.get("scope", "").split(),
 expires_at=claims["exp"],
 resource=RECURSO,
 )

And when mounting the server, the flag that does not come set:

auth = AuthSettings(
 issuer_url=ISSUER,
 resource_server_url=RECURSO,
 validate_token_resource=True, # without this there is only a warning
 required_scopes=["mcp:inventario"],
)

Checklist

Enable audience validation in the Python SDK, or write it into the verifier if it is TypeScript. Without that, the central normative requirement is not met.
Reject tokens with other resources’ audiences, not just check that your own is present.
Mount the metadata document by hand, with all the authorisation servers and the real catalogue of scopes.
Add the scope parameter to the challenge, in Python, and account for scope hierarchies.
Register each MCP server as a Keycloak client with its canonical URL as identifier, so that token exchange produces the right audience.
Replace any token passthrough mode with token exchange, except the narrow case where the token was already issued for the target.
Check that the gateway appears in the audience of the user token, or the exchange will fail.
Bind state handlers to the user on the server side, and never treat them as authentication.
Decide between local validation and introspection knowing that the first delays revocation until expiry, and shorten the token life accordingly.
Pin a hash of the description and the input schema from outside, because the specification does not cover it.
Put a policy engine in for per-tool authorisation, and do not try to model it with roles.
If you are considering the enterprise extension, check first who issues the grant: Keycloak today only knows how to receive it, in experimental and against an earlier draft.

Traps and things that are not what they look like

The Python SDK mounts the metadata document on its own, and that is why it looks solved. What it publishes carries a single authorisation server and no name and no documentation.

The audience validation deprecation warning does not enable anything. It warns and carries on behaving as disabled.

The TypeScript SDK does not validate the audience at all. It is not configuration, it is that there is no code for it.

Python’s challenge does not carry the scope, so a client that receives a 403 does not know what to ask for.

The scopes workaround allows two audiences in the same token. Each mapper contributes its own, and there is no client policy that limits it. The defence is in the resource.

The token exchange audience is a client identifier, not a URL. If the MCP server is not registered with its URL as identifier, the resulting audience will not match what the resource validates.

Token exchange requires the gateway to be in the audience of the user token. It is the condition that wastes the most time the first time round.

Dynamic client registration is deprecated as of this revision, and the substitute, client identifier metadata documents, is experimental in Keycloak and has an open bug with documents carrying unknown fields, which blocks login with real clients.

LiteLLM’s identity assertion grant depends on the generic OIDC provider. With Google, Microsoft or SAML no assertion is captured and all users fail.

Step-up elevation is no longer the server’s responsibility. Scope accumulation moved to the client in this revision.

The authentication elevation signal goes in a 401, not a 403. Insufficient scope is 403; asking for a higher authentication level is a different thing and a different code.

Closing

The summary of this post fits in one sentence: the identity provider is the cheap half of the problem, and the expensive half is the protected resource, which no SDK hands you ready-made and which in the two official ones comes with audience validation disabled or absent.

From that comes an order of work. First the token verifier, with the audience validated in both directions, because without it everything else is decoration. Second, the complete metadata document, mounted by hand. Third, replacing any token passthrough with exchange, with the MCP server registered with its URL as identifier. Fourth, a policy engine for what roles do not model.

And a scheduling decision worth taking with the information in front of you. The enterprise-managed authorisation extension is the right route for an organisation with a provider of its own, it is stable and it has servers in production behind it. What is not ready is Keycloak as issuer. Anyone who wants that flow today has two honest options: delegate the mechanics to a gateway that implements it, or wait. Putting it into production with a feature the official documentation marks as experimental and expressly advises against is not one of them.

Sources

Especificación MCP, revisión 2026-07-28: autorización, registro de clientes y buenas prácticas de seguridad.
Extensión de autorización gestionada por la empresa y su especificación estable, del SEP-990. Anuncio de 18 de junio de 2026.
draft-ietf-oauth-identity-assertion-authz-grant, revisión 04 de 21 de mayo de 2026, grupo de trabajo OAuth del IETF.
Código de los SDK oficiales de MCP: python-sdk y typescript-sdk 2.0.0-alfa, clonados el 12 de septiembre de 2026.
Integración de Keycloak con Model Context Protocol, intercambio de tokens y concesión de autorización por aserción de identidad, consultados el 12 de septiembre de 2026.
Incidencias de Keycloak 14355, 47117 y 51039, y petición de cambios 35711.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
NSA y SEI, Model Context Protocol: Security Design Considerations, mayo de 2026.
OpenFGA, autorización de servidores MCP, actualizado el 9 de septiembre de 2026.
mcp-scan de Invariant Labs, Apache 2.0.
Documentación de agentgateway, Traefik Hub, Kong y Pomerium, consultadas el 12 de septiembre de 2026.

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

Sat, 12 Sep 2026 07:30:00 +0200

This post opens the identity block of the operations track. The previous one covered the gateway side: the JWT that does not validate audience, the user who never reaches the trace. This one covers the piece on the other side. Verified against Keycloak 26.7.3, revision 2026-07-28 of the MCP specification and the text of Royal Decree 311/2022 in the BOE.

TL;DR

Keycloak 26.7.3 was released on 31 August 2026 and is still incubating at the CNCF, which it entered in April 2023. The cadence is four minor releases a year and support in practice covers a single branch: the twenty security bugs fixed in 26.7.3 were not backported to the 26.4, 26.5 or 26.6 branches at the time of release.

The Quarkus-based distribution separates build and runtime configuration. What is set in the build phase is frozen into the image, and changing it at startup forces a rebuild. In containers, the pattern is to build into the image and start with the optimised flag.

Storage is relational and only relational. The alternative engine that was in development was withdrawn in version 25.0. Since 26.0 sessions are persisted to the database by default, which changes the load profile: the dominant cost in the official benchmarks is database CPU.

Organisations solves multi-tenancy inside a realm, fully supported since 26.0. It groups members, associates email domains and links identity providers per organisation. It is the alternative to replicating realms per customer.

The measured bottleneck is password hashing. Since 25.0 the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. The official sizing guide gives one vCPU for every 15 password logins per second, against 120 per second for client credentials and for refreshes.

Keycloak does not implement two standards the MCP specification marks as mandatory. Protected resource metadata belongs to the MCP server, not the authorisation server, so there is nothing to hold against it there. Resource indicators do belong to it and are not supported. Keycloak’s own documentation declares partial conformance with the last three MCP revisions for that reason. The practical substitute is scopes plus an audience mapper.

There is a recent critical bug worth looking at today. CVE-2026-18963, scored 9.1, allowed resetting any user’s password without authenticating, administrators included. Fixed in 26.7.2, released on 19 August 2026.

You are here: the cross-cutting layer

Identity is not a stage of the pipeline, it is a layer that crosses all of them. It shows up in stack hardening as one of the seven layers of defence, in the model’s chain of trust as workload identity, in the gateway as token validation, and in the post about authenticating MCP servers as authorisation server.

What was missing was the post about the piece itself, and above all the decision of where to put it and what to expect from it.

The analogy: the civil registry and building permits

A small town hall has a population register, which says who lives there, and it has permits, which say who can do what. They are two different things and it is worth not confusing them, because the register is good for the whole town while a permit is good for one specific job on one specific street.

Keycloak is the register. It knows who each person is, identifies them reliably, knows which groups they belong to and issues a credential document that expires. What it does not do is decide whether this person can modify this particular file: that is the permit, and it lives in each service.

Confusing the two produces the most expensive design mistake in platforms of this kind, which consists of putting into the token everything that will need to be decided later. The register document grows, gets handed out everywhere, and when a permission changes you have to wait for everyone’s document to expire.

There is a third element the analogy captures well. A credential document issued for presenting at the tax office should not be good for entering the municipal depot. That is the token’s audience, and it is the central subject of the second half of this post.

Part 1. What Keycloak is today

Project status

Item	Value
Latest stable version	26.7.3, released on 31 August 2026
Previous ones	26.7.2 (19 August), 26.7.0 (9 July), 26.6.0 (8 April)
Cadence	four minor releases a year
CNCF	incubating since 10 April 2023, not graduated
Support	the current branch; earlier ones do not receive fixes reliably

That last point deserves underlining because it conditions the upgrade policy. The analysis of 26.7.3 shows that its twenty security fixes were not backported to earlier branches at release time. Operating Keycloak means keeping up to date with the current branch, not staying on a “stable” version for a year.

Since 26.0 there is a backward compatibility guarantee across minor versions for features marked as fully supported, and breaking changes are opt-in.

The internal architecture in five pieces

Two startup modes. The development one enables HTTP, does not resolve the hostname strictly and disables the theme cache. The production one requires a hostname and TLS, or it will not start. Confusing them in production is the first antipattern on the final list.

The build phase. The build command materialises the optimisations into the image. Build options, such as the database engine, the enabled features or metrics, are frozen. Runtime ones, such as the database password or the hostname, are applied at startup. Changing a build option at startup forces a rebuild with the corresponding latency cost. The correct pattern in containers is to build into the image and start with the optimised flag.

Extensions via SPI. Authenticators, protocol mappers, user storage, event listeners, hashing algorithms and secret vaults. They are packaged as JARs and resolved in the build phase.

Infinispan cache. Embedded by default, with database-based discovery as the recommended mechanism. The discovery alternatives via Kubernetes DNS, TCP and UDP are deprecated. There are local caches, with default capacities of 10,000 entries in several of them, and replicated caches for sessions, client sessions, offline sessions, authentication sessions, login failures and action tokens. Remote mode, with an external Infinispan, is needed for first-generation multi-site deployment.

Relational storage. JPA over a database. The alternative engine was removed in version 25.0. The databases supported today include PostgreSQL from 14 to 18, MariaDB and MySQL in their extended support versions, SQL Server 2019 and 2022, Oracle 19c and 23.x, and the managed variants Aurora PostgreSQL and Azure SQL.

The minimum conceptual model

Before touching anything you need to be clear on eight concepts, and the order in which they relate:

Realm: the isolation boundary. Its own users, its own signing keys, its own policies. Two realms share nothing.
Client: an application that asks for tokens. Public if it cannot keep a secret, such as a browser application; confidential if it can, such as a service; and bearer-only if it merely validates tokens without requesting them.
Client scope: a reusable bundle of scopes and mappers, assigned to clients as default or as optional. It is the unit with which you bound what a token carries.
Protocol mapper: what projects an attribute, a role or an audience into the token. The audience mapper is the one that solves the central problem of this post.
Role, realm or client, and composite role, which aggregates others.
Group, hierarchical, with inherited roles and attributes.
Identity provider: federation towards another external OIDC or SAML. User federation: LDAP, Active Directory and Kerberos, with synchronisation and attribute mapping.
Authentication flow: a sequence of executions with required, alternative, conditional or disabled requirements, and subflows. It is where you build a second factor or a check of your own.

And one feature that changes architectures: organisations, fully supported since 26.0. It solves multi-tenancy inside a single realm, grouping members, associating email domains and linking identity providers per organisation, so that provider discovery is done by the user’s email domain. The classic alternative, one realm per customer, multiplies administration and shares no configuration.

Part 2. Who can authenticate against it

This is the table you need before drawing anything. The right-hand column is usually the one that decides.

Piece	Protocol	Cost
Kubernetes API	native OIDC, or the structured authentication configuration file	Free
LiteLLM	OIDC and JWT, plus UI SSO	JWT and SSO above five users: Enterprise
vLLM	static key only	No OIDC
Langfuse	generic OIDC and Keycloak provider	SSO free; per-project access control: Enterprise
Grafana	generic OAuth yes in the open edition	Team synchronisation: Enterprise
Backstage	generic OIDC and community-maintained Keycloak provider	Free
Open WebUI	OIDC, with role management by claim	Free
LibreChat	generic OIDC, with Keycloak documented	Free
Argo CD	native OIDC or Dex, with groups into its RBAC file	Free
MinIO	OIDC with role assumption via web identity	See the note below

Two clarifications that change decisions.

Kubernetes. The modern mechanism is structured authentication configuration, which allows several JWT issuers plus validations and mappings with CEL expressions. The corresponding proposal, KEP-3331, went alpha in 1.29, beta in 1.30 and is stable since 1.34. It is the route to use today, and it replaces the loose API server flags.

vLLM has no identity. All there is is a static key via environment variable. This is not a defect to fix, it is an architectural fact: the engine sits behind the gateway and its access control is network-based, not identity-based. Any design that exposes vLLM directly to users is giving up per-person traceability.

About MinIO a note of honesty is in order: the public documentation now redirects to the commercial edition, so the detail of how it maps policies from a claim is verified there and not in the open edition. Anyone depending on that mechanism should confirm it against their own version before designing on top of it.

Part 3. Humans and machines

They are two different problems and it is worth solving them separately.

People

Authorization code with PKCE, with the S256 method. The MCP specification requires it when the client is technically capable, and it goes further: it mandates rejecting the flow if the authorisation server metadata does not publish the supported challenge methods.

Workloads with no person behind them

Here the obvious option is client credentials with a Keycloak service account, authenticated with a secret or, better, with a JWT signed by private key.

But there is a less known and better option for workloads inside the cluster: federated client authentication, introduced in January 2026 under the banner of doing away with secrets. It lets a client authenticate with a token issued by an external provider, and one of those providers is Kubernetes. The announcement placed full support for the Kubernetes service account variant and for OIDC in version 26.6; the SPIFFE variant, by contrast, expressly says it will stay in preview until the SPIFFE client authentication standard is finalised at the IETF.

That means a pod can authenticate against Keycloak with the token the kubelet already projects into it, with no additional secret to rotate. It is worth confirming the exact status and the flag name against the release notes of whichever version you have deployed, because the feature is recent and has changed status between 26.5 and 26.6. It is preferable to token exchange, whose standard version is supported but is internal-to-internal only, while the old one, which covered external-to-internal and impersonation, is deprecated and in preview.

Keycloak against SPIFFE

The question always comes up and the answer is that they solve different things with a small overlap.

SPIRE attests the node and the process, and issues short-lived identity documents with no prior secret. Keycloak does not attest workloads: it trusts a credential someone had to place there, except in the federated authentication case just mentioned, where the one attesting is Kubernetes. In exchange, Keycloak federates people, issues tokens with business data and governs consent, things SPIFFE does not do.

The practical rule: SPIFFE for service-to-service identity inside the perimeter, Keycloak for anything that has a person behind it or crosses the perimeter. The SPIFFE detail is in the close of the chain of trust series.

Part 4. MCP, and the two missing standards

This part has the most consequences and is the worst documented outside the primary sources.

What the specification requires

From revision 2026-07-28, with the normative words as they appear:

The MCP server must implement RFC 9728 protected resource metadata, and the client must use it to discover the authorisation server.
The client must implement RFC 8707 resource indicators, including the resource parameter in both the authorisation request and the token request, and must send it regardless of whether the authorisation server supports it.
The MCP server must validate that the access token was issued specifically for it as the audience, and must not accept or relay any other token.
Authorisation servers and clients should support client identifier metadata documents. RFC 7591 dynamic registration becomes optional and is declared deprecated, retained for compatibility.

The revision also adds the issuer in the authorisation response, from RFC 9207, and a stateless model with state handlers where it is expressly stated that holding a state handler is not equivalent to being authenticated.

What Keycloak supports

From its own MCP integration guide:

Standard	Keycloak
OAuth 2.1 and authorisation server metadata	Supported
RFC 9207, issuer in the response	Supported
RFC 7591, dynamic registration	Supported
Client identifier metadata document	Supported, experimental, after enabling the feature
RFC 8707, resource indicators	Not supported
RFC 9728, protected resource metadata	Out of scope, it belongs to the MCP server

The conformance declared by the project itself is “partially supported, without resource indicators” for the last three MCP revisions.

Two practical conclusions:

The protected resource metadata document is published by the MCP server or a proxy in front of it, never by Keycloak. That is correct and there is nothing to fix, but you have to remember to implement it, because the client needs it to discover where to go.
Without resource indicators, the audience is bounded with scopes. The official route is to use the scope parameter instead of resource, and to hang from that client scope an audience mapper with the MCP server URL as the included custom audience.

It is a workaround, it works, and it has to be documented in the design because anyone coming after will look for the standard parameter and will not find it.

Part 5. The confused deputy

With the above on the table, the classic mistake becomes obvious on its own. A token with a generic audience turns any compromised service into a master key: whoever obtains the token a user presented to the dashboard can present it to the gateway, and from there to the MCP server.

The MCP specification is explicit in three sentences worth quoting verbatim in any architecture document:

MCP servers must not accept tokens that were not explicitly issued for them.
If the MCP server calls upstream APIs, it must not forward the token it received from the client.
MCP proxies that use static client identifiers must obtain user consent for each dynamically registered client before forwarding to third-party authorisation servers.

Translated into the specific figure this track has been dealing with: the gateway must not forward the user’s JWT to third-party MCP servers. It has to obtain a different token with the right audience, or act as an OAuth client in its own right. By default, as already documented in the MCP gateway post, the behaviour is to use the server’s own credential, which is the correct one; there are two modes that forward the caller’s token and both have a name in the specification.

Part 6. Two hours of task against five minutes of token

The specification pushes towards short-lived tokens and mandatory refresh token rotation in public clients. A two-hour agentic task comfortably exceeds the life of an access token and may also exceed session idle time if nobody refreshes.

Three real options, with their cost:

Offline tokens. They survive session expiry, and expire through their own idle timeout or through a maximum if that limit is enabled. The risk is obvious: a near-permanent credential, hard to inventory. And the MCP specification adds that MCP servers should not include that scope either in the authentication header or in the supported scopes of their metadata.

Automatic refresh in the client. It is the right answer when there is a client that can implement it, with refresh token rotation and secure storage. For an agent with no person behind it, it is better to renew via client credentials.

Long-lived virtual keys in the gateway. It works, and it is what most people end up doing. The price has to be written down in the risk analysis: identity ends up decoupled from the provider, so deactivating someone in Keycloak does not invalidate their key, and traceability back to the real person depends on the discipline with which the keys were created. The levers that compensate are expiry, automatic rotation and a mandatory alias, covered in the virtual keys post.

A note of honesty about the default values. The numbers going around (five minutes of access token, thirty of session idle time, ten hours of maximum) do not appear published in the administration guide and I did not locate them in the code. Treat them as a console observation and confirm them in your own realm before quoting them in a document.

Part 7. Sizing and operating

On Kubernetes

The operator exposes three custom resources: the server, the realm import and the OIDC client. Probes and metrics live on management port 9000, with the health and readiness paths, and metrics in Prometheus format provided the corresponding option was enabled in the build phase. That port is not exposed publicly.

For high availability there are three shapes. A single cluster spread across zones, with no additional infrastructure. First-generation multi-cluster, with an external load balancer and an external Infinispan at each site. And second-generation multi-cluster, introduced in 26.7.0 and still in preview, which connects clusters without external Infinispan, using the synchronously replicated database as the source of truth, with less cache and more database load.

The official sizing figures

From the CPU and memory sizing concepts guide, which is still published:

1 vCPU for every 15 password logins per second, tested up to 300 per second.
1 vCPU for every 120 client credential grants per second, tested up to 2,000.
1 vCPU for every 120 token refresh requests per second, tested up to 435.
Leave a 150 % CPU headroom for spikes, startup and failover.
1,250 MB of memory per pod with realm data and 10,000 cached sessions. Keycloak allocates 70 % of the limit to the heap and around 300 MB outside it, so the limit comes from subtracting those 300 MB from the expected usage and dividing by 0.7.

The stated assumptions behind those figures include Argon2 with five iterations, sessions in the database and the default cache of 10,000 entries.

The difference between 15 and 120 is the whole story of Keycloak’s performance: the cost is in password hashing. Since 25.0, the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. Before it was PBKDF2, whose iterations were multiplied by ten in version 24. In FIPS mode PBKDF2 is still used.

From that comes an architectural consequence for an AI platform: machine workloads, which go through client credentials, cost eight times less than a human login. Sizing by number of services is cheap; sizing by morning login peaks is the expensive part.

The benchmark

From the 26.4 performance report, published on 1 October 2025: up to 2,000 logins per second and 10,000 refreshes per second with three pods, 74 vCPU in total and 8 GB per pod, over an Aurora PostgreSQL database, with almost linear vertical scaling.

Two useful findings from that report. With 20 ms of round-trip time between sites, the 99th percentile rises from 51 to 130 ms, and the project advises against deploying across different regions. And raising the cache from 10,000 to 200,000 entries brought the database CPU peak down from 77.77 % to 63.77 %, in exchange for somewhat longer garbage collection pauses.

Security

The urgent matter, if the deployment is below 26.7.2:

CVE-2026-18963, score 9.1, critical. A bypass of the email action token in the credential reset flow allowed resetting any user’s password with no prior authentication, administrators included. Fixed in 26.7.2, released on 19 August 2026. The temporary mitigation, if you cannot upgrade, is to disable password recovery in all realms. No known exploitation was recorded as of the end of August.

In 26.7.3 there are another nineteen, among them the missing hostname verification in the TLS certificate towards LDAP with a score of 8.8, two tenant restriction bypasses in token exchange with Microsoft and Google brokers, and two access control bypasses in organisations.

Official hardening comes down to four decisions:

Strict hostname, which is the default value since the second generation of that configuration. The server does not infer the URL from headers, which prevents poisoning email links and redirects.
Administration console on a different hostname, and network-restricted.
Proxy headers declared explicitly, trusting only a controlled reverse proxy.
Separate management port, not exposed.

Part 8. The mapping to the ENS

The codes and titles below are verified against the official text of Royal Decree 311/2022 published in the BOE, Annex II.

Code	Literal title	What a central IdP contributes
op.acc.1	Identificación	A unique identifier per entity, user or process that gains access
op.acc.2	Requisitos de acceso	Permissions are granted by role and group, not by loose credential
op.acc.3	Segregación de funciones y tareas	It supports it with roles, but the control is organisational
op.acc.4	Proceso de gestión de derechos de acceso	Least privilege and a specific remote access policy
op.acc.5	Mecanismo de autenticación (usuarios externos)	Flows and factors configurable per flow
op.acc.6	Mecanismo de autenticación (usuarios de la organización)	Reinforcements required already at the low level
op.exp.8	Registro de la actividad	It contributes part of the events, not all of them
op.exp.10	Protección de claves criptográficas	Management and rotation of the realm signing keys

Two nuances an auditor is going to look for.

The first, about op.acc.1: the text requires every entity that accesses the system, whether user or process, to have a unique identifier. That includes agents. A virtual key shared between several agents does not comply, and that is exactly what usually happens when JWT authentication falls outside the budget.

The second, about op.exp.8: the control requires recording the identifier of the user or entity associated with the event, the date and time, what information it is performed on, the type and the result. Keycloak contributes the authentication events. The usage events are contributed by the gateway and the MCP servers, and there the two already documented gaps reappear: adding and removing MCP servers generates no audit log and the trace user is set by the client.

The correspondence between control and product in this table is my own interpretation and must be validated by the certification body. The codes and titles are not: those are literal from the BOE.

Part 9. The configuration to leave written down

Three concrete pieces, so as not to stay at the level of concepts.

Hardened startup, as container arguments:

kc.sh start --optimized \
--hostname=https://sso.ejemplo.es \
--hostname-admin=https://sso-admin.interna.ejemplo.es \
--proxy-headers=xforwarded \
--health-enabled=true \
--metrics-enabled=true \
--http-management-port=9000 \
--cache=ispn --cache-stack=jdbc-ping \
--db=postgres

The metrics, health, cache and database options are build options: they go into the image, and passing them here forces a rebuild if they do not match.

The client scope with audience, which is the workaround forced by the lack of resource indicators. Created with the administration tool:

# one scope per protected resource
kcadm.sh create client-scopes -r plataforma \
 -s name=mcp-inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

# the mapper that sets the token audience
kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocol=openid-connect \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

The client asks for that scope and receives a token whose audience is only that MCP server. The MCP server validates the audience and rejects any other token, as the specification requires.

Realm import via GitOps, with the operator’s custom resource, so that the configuration does not live only in the console:

apiVersion: k8s.keycloak.org/v2beta1
kind: KeycloakRealmImport
metadata:
 name: plataforma
 namespace: identidad
spec:
 keycloakCRName: sso
 realm:
 realm: plataforma
 enabled: true
 # without this, the recovery email is attack surface
 resetPasswordAllowed: false
 bruteForceProtected: true
 sslRequired: all

Part 10. What Keycloak does not solve

Fine-grained authorisation over resources. The question “can this person see this particular trace” is a relation between objects, not a role. That problem is solved by Zanzibar-style engines, with OpenFGA and its object, relation and user tuples, or a policy language such as Cedar. Keycloak gives roles and scopes, and with those you do not model a graph of relations.

Machine secrets. Storage credentials, provider keys, database passwords. That is a secrets manager, covered in stack hardening.

Cryptographic workload identity with attestation. That is SPIFFE and SPIRE. In Keycloak it exists as an identity provider in preview, which is a different thing.

Authorisation inside the model. Which MCP tool an agent may invoke, and with what arguments. The specification offers scopes, the insufficient scope error and step-up elevation, but the per-tool and per-argument decision is taken by the MCP server or a policy engine, not by the identity provider.

Antipatterns

Forwarding the user’s token from the gateway to third-party MCP servers. Explicitly forbidden by the specification.
Static client identifier with dynamic registration and no per-client consent, which is the exact recipe for the confused deputy.
General scopes, of the full-access kind, and publishing the entire catalogue of supported scopes in the metadata.
Treating a state handler as authentication, which the new revision expressly forbids.
Following OAuth discovery towards any URL, which opens the door to requests towards internal addresses. It also applies to the authorisation server when it downloads a client metadata document.
Trust headers badly exposed in chat front ends. The Open WebUI documentation warns that an incorrect configuration allows authenticating as any user.
Starting in development mode in production, with the hostname not resolved strictly.
A single shared static key as the only control between the gateway and the engine.

Startup checklist

Check the version. Below 26.7.2 there is a critical password reset bug.
Build the image with the build phase done and start with the optimised flag.
Strict hostname, administration console on another hostname, separate management port and not exposed.
Decide between organisations and one realm per tenant before creating the second tenant.
One client scope per resource, with an audience mapper, because resource indicators are not supported.
In the gateway, define the two audience and issuer variables. Without them none of that is validated.
For workloads inside the cluster, look at the Kubernetes identity provider before handing out client secrets.
Size by login peaks, not by number of services: the difference is eightfold.
Export the Keycloak events to the same place the gateway’s go, because the activity logging control needs them together.
Write into the design what the system does with a two-hour task, before someone solves the problem with an offline token.

Closing

Keycloak is a mature piece that solves one bounded problem well: knowing who the caller is, and saying so verifiably in a document that expires. In a sovereign inference platform that is worth a lot, because the alternative is an inventory of static keys spread across ten services that no audit can walk through.

What is not advisable is asking more of it. Fine-grained authorisation lives in each service, workload identity with attestation lives in SPIFFE, and the decision of which tool an agent may call lives in the MCP server.

And there is one concrete gap worth noting down, because it does not close on its own: the MCP specification marks two standards as mandatory, and of the two that would fall to an authorisation server, resource indicators are not implemented. The workaround with scopes and an audience mapper works, but it is a workaround, and whoever comes after will look for the standard parameter. Leaving it written in the design saves an afternoon.

Sources

Keycloak 26.7.3, 31 de agosto de 2026, y descargas.
Configuración del servidor, caché, bases de datos y nombre de host.
Integración con Model Context Protocol, intercambio de tokens y autenticación federada de clientes, 27 de enero de 2026.
Dimensionado de CPU y memoria e introducción a alta disponibilidad.
Bancos de pruebas de rendimiento de Keycloak 26.4, 1 de octubre de 2025.
Especificación de MCP, revisión 2026-07-28, autorización y buenas prácticas de seguridad.
KEP-3331, configuración estructurada de autenticación.
Real Decreto 311/2022, Anexo II.
Registro de CVE-2026-18963 en la base de datos de Red Hat y notas de la versión 26.7.3 en GitHub.
CNCF, proyecto Keycloak.

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

Fri, 11 Sep 2026 01:45:00 +0200

Sixth post in the operational track on the control layer. The previous five treat the gateway as the piece the tokens pass through: the pair with Langfuse, day 2, virtual keys, humans and agents and prefix routing. This one is about the other door. Verified against LiteLLM 1.102.0, commit of 10 September 2026.

TL;DR

The MCP gateway is not a module, it is a complete second surface. A native MCP endpoint with four route spellings, a proxy mode with three fixed tools, four REST routes, some thirty-five administration routes and a long set of OAuth discovery endpoints. It still lives under _experimental and it is nearly 20,000 lines.

I correct what I said in the previous post: access is NOT open by default. In 1.102.0, a key with no declared permissions and no team resolves zero servers. What does open things up is a per-server flag, allow_all_keys, and that flag overrides the key’s explicit scope unless you enable a setting that is off by default.

Permissions are enforced on execution, not just on the listing. Calling a tool that did not appear in tools/list returns 403. There are three chained checks before the call, including one on the allowed arguments. The same predicate governs listing and calling, by declared design.

The hierarchy is built out of intersections, with two escapes and a fail-open. The key’s access groups are additive on top of the key and team ceiling. The organisation ceiling replaces rather than intersects when there is nothing below it. And on an indeterminate error while resolving tool permissions, the code returns “unrestricted”. It is a deliberate decision and it has to be declared in the risk analysis.

Per-tool cost is worth zero and that is why its metric disappears. There are exactly two Prometheus metrics for MCP, and the spend one is only incremented if the cost is greater than zero. With the default configuration, that series never exists. There is no listing metric, no latency metric, no error metric, no server health metric.

The cost that does matter is not attributed to MCP anywhere. It is the tokens of the tool definitions, resent on every turn inside the prompt_tokens of the model call. Five typical MCP servers add up to 58 tools and some 55,000 tokens. The tool’s spend row, with its zero cost, has nothing to do with that.

And the gateway’s semantic filter does not act on the MCP door. It only applies when it is LiteLLM calling the model. A client connected to /mcp receives the whole catalogue, filter enabled or not. And even if it did act, filtering per request rewrites the cached prefix of the prompt, which is the opposite of what Anthropic and OpenAI do with their own tool searches.

The good observability ships switched off. Without LITELLM_OTEL_V2, a tool call comes out as a generic span with the model set to MCP: name. With it enabled there is a dedicated span and correct attributes, and the arguments and the result stay hidden unless you opt in. And in neither case is trace context propagated towards the upstream MCP server: the trace is cut at the gateway.

Tool arguments are written, and they are written in the clear. They go in full into the metadata column of the spend table, without passing through the switch that governs prompt storage nor through the redaction function, which only knows about messages and response. Whoever enabled redaction for GDPR probably believes they are covered.

The 1.83 branch concentrated four serious vulnerabilities, and one of them belongs to the MCP gateway itself. The endpoint for testing a connection before saving accepted the stdio transport’s command, args and environment in the request body. A low-privilege key got command execution on the proxy host.

And the specification moved underneath. The revision in force since July 2026 removed protocol sessions, the initialisation handshake and the session header. LiteLLM 1.102.0 advertises the June 2025 revision and its version enumeration does not even contemplate the two later ones.

You are here: the door nobody chose

The gateway was chosen for the reasons in the switchboard post: licence, fit inside the cluster, maturity. What was evaluated was the inference traffic.

Later, somebody connects an agent, the agent needs tools, and the same process that is already authenticated and deployed turns out to speak MCP too. The decision to turn it into the tool door is almost never taken: it is inherited. And that door has neither the same controls, nor the same observability, nor the same cost model.

When MCP grows covered the identity of your own MCP servers. MCP from the inside covered the protocol and its instrumentation. This post covers the piece in between.

The analogy: the switchboard that also hands out keys

An office building’s telephone switchboard. Its original job is to route calls: it decides who can call outside, how much each department spends, and it leaves a record of every conference call.

Over time, the switchboard operator is also given the key cabinet. They are already at reception, they already know who everybody is, it looks like the natural place. And from that moment there are two services at the same desk with very different rules.

Calls are priced by the minute and show up on the bill. Keys have no tariff, so on the department’s bill they come out at zero, and somebody might conclude that the key cabinet costs nothing. It does cost: it costs the operator’s time, it costs having to read the whole key catalogue every time somebody asks what is available, and it costs that nobody checks that the label on a key still says what it said the day its loan was authorised.

The analogy ends at the interesting spot. When somebody takes a key, the record says which key it was and what they said they wanted it for. That record is simultaneously the evidence of the action and a piece of data that perhaps should not be stored in the clear.

The real routes

The first thing that surprises you when you look at the code is how many spellings get you to the same place.

The native endpoint

The MCP sub-application is mounted at /mcp and inside it has four mounts: the root as a wildcard, /mcp, /{name}/mcp and /sse.

And there is also a hand-declared route for bare /mcp on the main application, with a comment that explains why: the mount cannot match its own bare prefix, and the resulting 307 redirect breaks MCP clients sitting behind a proxy that terminates TLS. It is the kind of detail you discover in production on a Friday.

/mcp/sse holds a surprise. It does not use the classic SSE transport: it uses the same HTTP session manager in stateless mode. The SseServerTransport object is constructed and is not referenced on any other line of the file. It is dead code, and the /mcp/sse/messages endpoint that this object advertises does not exist as a working route.

The /{name}/mcp route resolves in a documented order: server alias, comma-separated list with a maximum of 16 elements, tool set, access group. If nothing matches, 404.

And /mcp/proxy exposes a fixed surface of three tools. It is the progressive disclosure pattern, and it will come up again further down because it is the most important mitigation in this post.

The REST routes

Four, under the /mcp-rest prefix:

Method	Route	Authentication
GET	`/mcp-rest/tools/list`	Virtual key
POST	`/mcp-rest/tools/call`	Virtual key
POST	`/mcp-rest/test/connection`	Key plus administrator role
POST	`/mcp-rest/test/tools/list`	Key plus administrator role

The variants without -rest that appear in the internal types have no router registering them. The /mcp/* wildcard absorbs them as JSON-RPC.

The last two are the stars of the security section.

Administration and discovery

Some thirty-five routes under /v1/mcp, almost all of them with virtual key authentication. Three details worth knowing:

GET /v1/mcp/registry.json carries no authentication dependency. It returns 404 if the public registry is not enabled, which is the real protection.

POST /v1/mcp/server requires the proxy administrator role. POST /v1/mcp/server/register does not: it is the proposal path, which requires a key with a team, explicitly forbids the stdio transport and leaves the entry pending review.

And on top of that there is a long set of OAuth .well-known endpoints, with dynamic client registration, authorisation, token, revocation and introspection, plus their per-server variants.

Configuration and transport

The mcp_servers schema in config.yaml has fifty-odd fields, all of them optional. A warning for anyone coming from earlier versions: spec_version no longer exists in 1.102.0.

The default transport when omitted is http, that is, streamable HTTP. The other two values are sse and stdio. The string streamable_http is not valid in the configuration file, although it is accepted as a synonym when importing connector files.

Tool prefixing uses the format {server}{separator}{tool} with - as the default separator, configurable through an environment variable read at import time. The prefix chosen is the alias, or the name, or the identifier, in that order. There is an optional short mode that derives three characters from the hash of the server identifier and resolves collisions by rehashing.

Access control, and the correction to the previous post

In the humans and agents post I closed the list of traps with this one: “access to MCP servers is open by default if no level defines a list”. Verified in 1.102.0, that is false.

The function that resolves the allowed servers does the opposite. If the team declares nothing, it inherits from the key; if the key declares nothing either, the base set is empty and the scope is marked as unbounded with zero servers. There is no branch that expands to “the whole registry” other than the administration role one.

What does open things up, and this is the important part, is something else.

`allow_all_keys`

It is a per-server flag, in configuration, in the API and in the database, defaulting to false. When set to true, that server joins the set of open servers and any key can see it.

The detail to know: by default it overrides the key’s explicit scope. A key that declares access only to server A will also see server B if B has the flag. The key’s scope is only respected if you enable general_settings.mcp_allow_all_keys_respects_mcp_scope, which is off by default.

There is a second opening case: anonymous subjects, with no user identifier, role or key, additionally receive the servers configured with authentication delegation towards the upstream and the pass-through ones. It is consistent with the design of those two modes, and it is a surface you have to know about.

`require_key_mcp_access_defined`

It lives in general_settings, defaults to false, and needs no licence. What it inverts is only the empty-key inheritance towards the team: with the flag on, a key with no list of its own gets an empty set instead of inheriting the team’s.

It does not affect keys that do declare servers, which still intersect with the team. And it does not close off the access group grants, which are additive and are the escape route you have to audit separately.

There is a sibling, require_end_user_mcp_access_defined, with the same shape and applied to the end user.

The hierarchy

The function’s docstring is normative and says that all the rules are intersections. In detail:

Key and team: intersection when both declare. Inheritance in one direction when one is empty.
Key access groups: additive union on top of the previous result.
End user: intersection if it declares anything.
Agent: intersection.
Internal user: ceiling that only narrows.
Organisation: ceiling that intersects if there are restrictions below, and that replaces by becoming the ceiling when there are none.

For subjects with no key, admitted by SSO or by the gateway, the model changes to a per-source union instead of an intersection.

The fail-open

This is the point you have to take to the risk analysis and not leave in a technical note.

On an indeterminate failure while resolving the tool permissions of a key or a JWT, the code returns “unrestricted”, that is, all tools allowed. The organisation ceiling is skipped in the same situation. For subjects with no key the behaviour is the opposite, fail-closed.

There is a well-resolved intermediate case: when a permission is named but cannot be read, a specific error is raised and it is denied in both cases.

The doctrine is documented in the class docstring, which is appreciated. And it remains a behaviour that, in a system under the ENS, you have to declare in writing instead of discovering it during an audit.

What is done well

Permissions are enforced on the call, not just on the listing. The execution path chains three checks before contacting the server: allowed or forbidden tools, per key and team tool permission, and validation of the arguments against the allowed ones. All three return 403.

Listing and calling share the same predicate, and the docstring declares it as an explicit invariant. A client that calls a tool directly when it did not appear in its listing receives 403, it does not execute. The same over REST and over the responses API route.

The x-mcp-servers and x-mcp-access-groups headers only narrow, and they fail closed if they do not resolve.

Header forwarding towards the upstream is an explicit per-server allowlist, with a centralised decision about when to strip the caller’s Authorization, and a protection against cross-forwarding: in a multi-server listing, the global header is withheld if more than one server would consume it. That is RFC 9700 applied.

The bill that never shows up

Here is, in my opinion, the most important thing in this post, and it has nothing to do with security.

Every tool the gateway aggregates takes up space in the model’s context. Name, description and parameter schema travel in the request’s tools block, on every turn, and they are paid for as input tokens of the real model.

The figures published by Anthropic in November 2025 are the best public reference there is. Five connected MCP servers add up to 58 tools and some 55,000 tokens. The breakdown: GitHub, 35 tools and some 26,000 tokens. Slack, 11 tools and some 21,000. Sentry, five and some 3,000. Grafana, five and some 3,000. Splunk, two and some 2,000.

With tool search enabled, that set drops from some 72,000 to some 8,700 tokens, and the usable context goes from 122,800 to 191,300.

The second cost of a large catalogue is not economic and is not billed: it is that the model chooses worse. It has its own section further down, because of the two it is the one that decides whether the agent is any good.

How to measure it with what is already there

The gateway stores enough data, even if it does not cross-reference it for you.

In the listing’s spend row, with call type list_mcp_tools, it enriches the metadata with allowed_server_count, tool_count_total, per_server_tool_counts and per_server_list_outcomes.

In the model call rows, the proxy_server_request column keeps the request body with the tools block inside it.

The practical method: pair the listing rows with the LLM rows of the same session by session_id, read the total number of tools from the first, and pass the tools array of the second through litellm.token_counter(model=..., tools=...), which accepts that argument. Multiply by the number of turns, because the block is resent on all of them.

For the relative cost, a quicker comparison: mean prompt_tokens of keys with MCP enabled against those without, same model and same period.

Tool selection

The previous section treats the catalogue as a bill. This one treats it as what determines whether the agent gets it right. This is ground where the talk runs well ahead of the measurement, so it is worth separating what has been measured from what gets repeated.

How many tools a model can take

Anthropic’s official figure is that Claude’s ability to pick the right tool degrades when going from 30 to 50 available tools. The same documentation gives the inverse criterion, which is the more useful one: the full catalogue with nothing else is the right option with fewer than ten tools, when all of them are used on every request, or when the definitions add up to fewer than a hundred tokens.

There is also an administrative ceiling before the semantic one. VS Code with Copilot cuts off at 128 tools per request and returns an error when you exceed it.

The full curve gets cited far more than it has been measured. The soundest primary data I found is from May 2025: varying the number of candidates from 1 to 11,100 over a registry of more than 4,400 servers, below some 30 tools the success rate is above 90 %, between 31 and 70 intermittent failures appear, and past the hundred mark the degradation becomes dominated by retrieval. It is a qualitative heat map, not a numeric table.

The figure that circulates most (78 % with ten tools, 40 % with a hundred, 13.6 % with seven hundred) comes from a conference talk with no paper, no repository and no published methodology. Do not take it to a client presentation.

And there is a warning that affects everything else. An audit of 496 tasks from four tool calling benchmarks found 18.5 % misalignment between the label and reality. In the BFCL v4 subset, 80 % of the failures assigned to the agent came from fragile state comparisons. In one of the benchmarks with an automatic judge, 23 identical re-runs gave a range from 57.9 % to 76.8 %, almost 19 points of spread, enough to reorder the ranking. Any absolute figure in this section deserves a generous margin of error.

How it fails

The work with the cleanest taxonomy evaluates 36 servers, 220 tools and 1,000 tasks against twenty frontier models, with an average of 15.2 tools exposed per task of which 4.1 are relevant.

Failure mode	Share
Not using a tool when one was needed	10.5 %
Choosing the wrong tool	9.0 %
Malformed parameters	6.9 %
Not recovering from an error	10.3 %
Cognitive failures, not tool failures	63.3 %

The interesting part is in the per-model breakdown: the dominant failure mode changes with the model, not with the size of the catalogue. In one of them, 40.1 % of the failures are simply not calling any tool. In others the malformed parameter dominates. In another, the wrong tool. There is no canonical failure to attack.

On tools that do the same thing, which is the real case of a gateway with fifteen servers, there is a good measurement. A study with ten groups of five functionally equivalent tools and a thousand query pairs puts the selection bias between 0.3 and 0.4, that is, you would have to redistribute between 30 and 40 % of the probability mass for equivalent tools to be chosen equally. The same work shows that a biased continued training takes the selection of a specific endpoint from 0.6 % to 12.8 %.

That study contributes another figure that reorders priorities: shuffling a tool’s description moves the selection distribution substantially, whereas changing only the name has minimal and inconsistent effects. Applied to the gateway’s server-tool prefixing: it costs tokens and probably does not change which tool the model goes to. There is no study isolating the effect of name spacing, and it is a real gap in the literature.

If the description is what carries weight, the state of the estate is the problem. An audit of 856 tools from 103 MCP servers found undeclared limitations in 89.8 %, missing usage guidance in 89.3 % and opaque parameters in 84.3 %. Only 2.9 % were free of problems across all their components.

The optimum is not a number

The May 2026 work I cited above proposes a chance-corrected metric and uses it to size the list adaptively. Over 370 functions it reaches 90.3 % coverage while seeing an average of 7.4 tools, against 90.8 % seeing a fixed 50.

But the same paper publishes its own counterexample, and it is honest to cite it: over another benchmark of 3,251 tools, the fixed K of 5 wins in aggregate, 64.7 % against 61.9 %. The adaptive advantage concentrates in the hard queries, where the right tool sits between sixth and twentieth by similarity: there it finds 16.7 % of the cases where the fixed K finds 0 %.

The March 2026 work over 121 tools puts the plateau at K=3, with 97.1 % that does not improve going up to K=5. With the caveat that it measures retrieval against labels, not model accuracy.

And a July 2026 work gives the formulation that seems to me the right one: ranking by score is inconsistent with optimal acquisition when costs are heterogeneous, so the optimal K is a function of cost, not a constant. Trimming from 7 to 4.4 exposed tools while keeping success is possible; fixing a universal number is not.

Filtering can make the result worse

This is the part that appears in no product blog. A June 2026 benchmark with three registry sizes (25, 100 and 250 tools), six filtering methods, seven models and 26,460 runs:

Strategy	Success	Visible tools	Tokens
Expose the whole catalogue	32.1 %	125.00	56,062
Keyword, top-5	22.1 %	4.80	3,200
Keyword, top-10	22.4 %	9.54	5,356
State-aware	24.0 %	25.95	13,522
Full causal path	24.0 %	26.24	13,697

Four of the five filtering strategies perform worse than not filtering. The fifth, which does win comfortably, exposes 0.99 tools per step and requires hand-written precondition and effect contracts for every tool, plus a search of the causal path to the goal state. It is a planner with the goal given, not a retrieval filter, and it is not comparable with anything a gateway offers.

There is more evidence in the same direction. A work over a corpus of 43,000 tools measured that the agent’s completion rate drops when using retrieved sets against hand-annotated sets: the filter introduces its own recall ceiling. And top-K breaks the tasks that need several tools: on a benchmark of 16,464 endpoints, a fine-tuned retriever reaches a Recall@3 of 68.6 % but only 39.7 % of the queries have their complete tool set within the first three. Almost 29 points of gap between “the one I needed is there” and “all the ones I needed are there”.

Filtering is not free on security either. A March 2026 work attacks the retrieval layer by injecting adversarial tools that saturate the top-k: with injection rates of 1.2 % it manages to dominate the top-k between 91 % and 97 % of the time. It is an attack surface that exposing the whole catalogue does not have.

The most elegant counterpoint comes from Anthropic’s own numbers. With the same catalogue of more than fifty tools, the gain from tool search drops from 25 points on one model to 8.6 on the next, because the unfiltered baseline rises from 49 % to 79.5 %. The margin left by filtering shrank to a third in one generation of models. Any architecture decision taken here has a short shelf life.

The prompt cache trap

Here is what makes a homemade top-K filter a bad idea even if it worked.

Tool definitions do not go just anywhere in the prompt: they go at the beginning. Anthropic documents the order of the cacheable prefix as tools, then system, then messages, with a hierarchy where each level rests on the previous one. And the invalidation rule is explicit: modifying tool definitions, be it names, descriptions or parameters, invalidates the entire cache, and a change at one level invalidates that level and all subsequent ones.

OpenAI says the same and adds a detail: the prefix must remain unchanged, and order counts. The same set of tools served in a different order is a cache miss.

With vLLM serving an open model there is no tool block of its own. The tools parameter is serialised by the chat template inside the first system message, behind the system prompt text. For automatic prefix caching the effect is identical, because it is still near the absolute start of the prompt.

The consequence is the one that matters. A filter that returns a different set of tools on the next turn does not just invalidate the tool block: it invalidates everything behind it, which is the system prompt and the whole conversation, which is exactly the part that grows with the turns. At list prices, a prefix token goes from costing 0.1 times when read from cache to 1.25 times when written. An agent with sixty thousand tokens of stable prefix goes from some six thousand equivalent tokens per turn to some seventy-five thousand, plus the prefill that gets paid again in latency.

The two big providers have separately arrived at the same solution, and it is the opposite of filtering in the prefix. Anthropic excludes deferred tools from the prefix and, when the model discovers one through search, injects it inline inside the conversation: “the prefix stays intact, so prompt caching is preserved”. OpenAI: “all tools are loaded at the end of the model’s context window (…) this allows the cache to be preserved from one request to the next”. Anthropic’s operational recommendation is the stable core pattern: leave the three to five most used tools undeferred.

The closest academic work, from January 2026, states the problem without measuring it and recommends exactly that, keeping a fixed set of reusable general-purpose functions and implementing dynamic capability through code generation.

The July 2026 MCP specification picks up both halves of the matter. Servers “should return the tools from tools/list in a deterministic order to allow caching in the client and improve the hit rate of the model’s prompt cache”. And it adds mandatory ttlMs and cacheScope in listing results, so that the client can freeze the catalogue for a known window instead of re-querying it and propagating any variation to the prompt.

Nobody has publicly measured the cost in euros of this effect. The only attempt I found is a self-published audit of ten thousand turns that puts it at 2.4 % to 3.5 % of cache misses, with non-reproducible methodology. It is a gap in the literature, and the order of magnitude in the previous paragraph comes from applying list prices, not from a measurement.

And now, LiteLLM

With all of the above in mind, the gateway’s levers read differently.

The semantic filter does not act on the MCP endpoint. It is registered as a generic LiteLLM callback, and its only dispatcher is the proxy’s precall hook, which additionally discards anything that is not completion, acompletion or aresponses. The tools/list path does not invoke that hook at any point; there is a comment in the code that admits it in passing, when explaining why the listing’s JWT signature is done separately. An external MCP client connected to /mcp receives the whole catalogue, filter enabled or not.

Lever	Serves external MCP clients	Breaks the prompt cache
Per-server `allowed_tools`	Yes	No, it is static
Name and description overrides	Yes	No, it is static
Selection by route or header	Yes	No, if it is stable per agent
`/mcp/proxy` and virtual tools	Yes	No, the surface is fixed
Semantic filter	No	Yes, it rewrites the block per request

When it does act, the filter has details you need to know before enabling it. It embeds only each tool’s description, with the name as a fallback if it is missing; the parameter schema does not go in. Of the query it embeds only the last user message: on turn twelve of an agent conversation, that might be “yes, go ahead”. The tool embeddings are cached and indexed at start-up; the query’s is computed on every request, with a synchronous call inside a coroutine.

Its failure modes are almost all open, which is the reasonable thing: if the embedding model does not respond, if nothing passes the threshold, if the names do not match the catalogue, it returns all the tools. The exception is context overflow, which fails closed with a 400, and which if it happens while building the index during start-up is memoised and makes all subsequent requests fail until you restart. The defaults are text-embedding-3-small, top_k of 10 and a threshold of 0.3, under litellm_settings.mcp_semantic_tool_filter.

/mcp/proxy does serve external clients, and that is where its real value lies. It replaces the catalogue with three tools with opaque identifiers. How search_tools searches deserves a line, because it is not what you would assume: by default it is neither a regular expression nor a semantic match, it is a keyword counter, how many tokens of the query appear as a substring in the concatenated name and description. It only moves to embeddings if you configure a model in mcp_tool_search. The result cap is hard-coded at five in this mode, and the client cannot ask for more. In exchange, call_tool goes through the same permission checks as a normal call plus a validation of the argument schema that the normal route does not do at that point.

There is a related variant: with mcp_tool_search_enabled in the key’s permissions, the catalogue is replaced by four virtual tools, and there the search one does expose top_k to the client.

The catalogue order is not deterministic, and that breaks the client’s cache without anybody filtering anything. The list of allowed servers comes from iterating a Python set of strings, so with the random hash seed the order differs between processes, between uvicorn workers and between restarts. There is no sorting anywhere in the listing path, and there is no tool cache: every tools/list queries the upstreams live, so the order within each server also depends on what the upstream returns that time. The gateway does not offer today the guarantee that the July specification recommends. The fix, sorting the catalogue before serving it, fits on one line.

Nor is there any pagination towards the client. The catalogue is returned whole, always.

And the lever almost nobody uses is the static one. Every MCP server accepts tool_name_to_display_name and tool_name_to_description, and those overrides really do replace what is sent to the client in tools/list, not just what the administrator sees in the interface. The tools/call routing translates from the new name to the real one before any permission check. With that you can trim a four-hundred-token description written by a third party, rename an ambiguous tool so it does not compete with a similar one, and do it without touching the prompt cache, because it is static and holds for all turns.

It is also the rug pull mitigation left pending from the security section: if the description the model sees is the one the operator wrote, whatever the upstream server changes stops reaching the context. Two limits. They are configured through the database or the management API, not through config.yaml. And the reverse name translation is by exact equality and is not disambiguated between servers, so two servers with the same display name route unstably, made worse by the non-deterministic order of the previous paragraph.

Alongside that, per-server allowed_tools remains the blunt and effective instrument: it filters the listing the client sees, reduces tokens, and is static.

The ladder, in order

Trim. Per-server allowed_tools, plus a per-key allowlist. Cheap, static, does not break the cache and reduces the permission surface at the same time.
Rewrite the descriptions with the overrides. That is where the evidence is, because the description carries weight and the name almost none, and along the way it pins the contract against the rug pull.
Split by route. Each agent to /{server}/mcp or to a tool set, never to the aggregated endpoint. Stable per agent, so the cache survives.
Sort the catalogue before serving it if you control the deployment, while the gateway does not do it.
/mcp/proxy when the catalogue goes past thirty or fifty tools. It is progressive disclosure and it works with external clients, which is more than you can say for the semantic filter.
The semantic filter, only when the one calling the model is LiteLLM. With a generous top_k, knowing that it embeds only the last user message and that context overflow is a closed and sticky failure.
What not to do: a home-grown top-K filter that rewrites the tools array on every turn. It costs the whole prefix and, according to four of the five measured strategies, probably makes selection worse.

The declared cost, which is worth zero

The calculation has four precedence steps: cost set by a post-call hook, per-tool cost in tool_name_to_cost_per_query, server default cost, and 0.0.

Without configuring mcp_server_cost_info, all MCP traffic is accounted at zero and consumes neither key nor team budget.

There are exactly two Prometheus metrics for MCP:

litellm_mcp_tool_calls_total
litellm_mcp_tool_call_spend_metric

Both with the same eight labels: tool name, server name, key hash, key alias, team, team alias, user and end user.

And the second is only incremented if the cost is greater than zero. With the default configuration, that time series never appears in Prometheus. It is not that it is worth zero: it is that it does not exist.

Nor is there a tools/list metric, or MCP latency, or MCP errors, or upstream server health. For that you are left with the generic request metrics.

In the reports, /spend/calculate has no MCP branch: it accepts a model and messages, or a completion response, and it always calls the LLM cost calculation. The per-team report does include MCP calls, grouped by the model column, which for these rows is worth MCP: tool_name. That is: every tool appears as if it were a model, with zero input and output tokens. Only the sessions endpoint separates the two call types explicitly.

Observability, and the trace that gets cut

There are two OpenTelemetry stacks in LiteLLM and which one runs depends on an environment variable.

Without LITELLM_OTEL_V2=true, the old logger runs, and it has no MCP handling: the only appearance of MCP in the whole file is the metadata key in a list. The tool call comes out as a generic litellm_request span with the model set to MCP: name.

With v2 enabled there is a dedicated span, with a dedicated role, client kind and a parent in the proxy’s request span. The span name is tools/call tool-name. The attributes are correct and follow the conventions:

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`mcp.method.name`	`tools/call`
`mcp.session.id`	MCP session identifier
`gen_ai.tool.name`	Tool name
`gen_ai.tool.call.arguments`	Only with content capture enabled
`gen_ai.tool.call.result`	Only with content capture enabled
`server.address`, `server.port`	From the upstream URL, redacted
`litellm.cost.total`	The cost that is probably worth zero

Content capture is off by default. Without it you see neither the arguments nor the result in the trace.

A note for anyone working with semantic conventions: mcp.tool.name does not exist. The MCP registry has exactly four attributes (mcp.method.name, mcp.session.id, mcp.protocol.version and mcp.resource.uri), all four in development, and the tool name goes in gen_ai.tool.name. LiteLLM gets it right. And the GenAI conventions were extracted into their own repository which as of today has no release at all, so there is no version number to cite.

In Langfuse, the vendor mapper returns an empty dictionary for any span that is not an LLM call. An MCP call appears as a raw span with its attributes, with no native input, output or model fields.

The trace is cut at the gateway

Incoming context is read. The traceparent the client puts in params._meta, following the specification’s propagation proposal, is extracted and turned into a link on the span, never into its parent. The reason is documented in the code: parenting to the remote trace would leave the span hanging off a trace whose root never reaches the backend. The client’s baggage is discarded on purpose to avoid forging identity attributes.

Outgoing context is not propagated. There is no propagator injection call anywhere in the tree. The headers that go out towards the upstream MCP server are built with authentication and the authorised extra headers, and nothing else.

The operational consequence: there is no end-to-end trace. If the upstream MCP server is instrumented, its spans live in another trace and there is no way to correlate them automatically.

Auditing and the conflict with privacy

A tool call leaves a normal spend row, with call type call_mcp_tool, spend equal to the cost per query, model MCP: name, zero tokens in the three token columns, a dedicated mcp_namespaced_tool_name column, and the usual key, team, user and session fields.

And in the metadata column, in JSON, the complete structure of the call: name, arguments, result, server, namespaced name, MCP session identifier, authentication mode and server resource with the URL redacted.

The problem

That structure is written into the metadata column without passing through the privacy switch. The assignment is direct and is not conditioned by the function that governs whether prompts and responses are stored, which does control the messages and response columns.

And the redaction function does not cover it either: it only touches messages and response. A search for MCP in the redaction file returns nothing.

The practical result: an operator who enables turn_off_message_logging for GDPR keeps sending the full arguments of every tool to every logging destination they have configured. To the SIEM, to Langfuse, to S3.

If an argument carries a national ID number, an address, a case number or a medical record, it has been exported.

There is an aggravating factor that does not depend on LiteLLM. The revision of the specification in force introduces an extension that allows mirroring tool arguments into HTTP headers visible to load balancers, proxies and web application firewalls. The specification recommends not marking sensitive parameters that way, and it is a recommendation, and the upstream server decides.

The fit with the ENS

The conflict is real and has a familiar shape: activity logging (op.exp.8) requires the call to be traced and non-repudiable; protection of personal data (mp.info.1) requires that log not to become a secondary repository of personal data with a different purpose. The tool argument is simultaneously the evidence of the action and the data. Redacting it destroys non-repudiation; storing it in the clear creates the repository.

The defensible way out you can build today: a hash of the argument in the audit log, the argument in the clear with encryption (mp.info.3), separate retention and its own access control. The hash sustains non-repudiation and the clear text is only opened under procedure.

There are two other measures this material touches directly. Change management (op.exp.5) because registering an MCP server and, above all, changing the description of an already approved tool are changes to the security configuration. And maintenance and updates (op.exp.4) because, as you can see in the next section, an old version of the gateway is a non-compliance with a name and a number.

Two honesty warnings. The codes follow the numbering of Annex II of RD 311/2022 and you have to check them against the text in force before citing them in a compliance document. And on NIS2: as of the date of this post the directive is still not transposed in Spain, with the draft bill on cybersecurity coordination and governance going through the process and a reasoned opinion from the Commission over the delay. What is enforceable today in the public sector and its suppliers is the ENS.

What is not logged

An administrative gap: registering, modifying and deleting MCP servers generates no audit log. There are three TODO comments in the management code that say so, one of them with an if is_audit_logging_enabled(): pass. In the same deletion block there are two more TODOs about not cleaning up orphaned permissions on keys and teams.

Security

The vulnerability you need to know about

CVE-2026-42271, published on 8 May 2026. Affects LiteLLM from 1.74.2 up to versions prior to 1.83.7. CVSS 3.1 of 8.8.

The POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list endpoints, which exist to preview an MCP server before saving it, accepted the stdio transport’s command, args and env in the request body. A low-privilege key executed arbitrary commands on the proxy host. It also affects Red Hat OpenShift AI on several branches.

It is the story of this post condensed into one line: the gateway you put in to contain MCP was the hole.

It does not come alone. The 1.83 branch concentrates three more serious ones:

CVE	Component	CVSS	Fixed in
`CVE-2026-42208`	SQL injection in key validation, with no prior authentication	9.8	1.83.7
`CVE-2026-35030`	OIDC userinfo cache keyed on `token[:20]`, collision and impersonation	9.1	1.83.0
`CVE-2026-40217`	Remote execution through bytecode rewriting in the guardrails test endpoint	8.8	no version recorded

Plus two privilege escalations fixed in 1.83.10 and 1.83.14.

The operational conclusion is short: a deployment below 1.83.14 is exposed to remote execution or to an unauthenticated SQL injection. The current branch is 1.102.0.

And a note for context: the whole MCP ecosystem has the same pattern. The unauthenticated remote execution in MCP Inspector, the command injection in mcp-remote when connecting to an untrusted server, DNS rebinding disabled by default in the Python and TypeScript SDKs until late 2025, the directory escapes in the reference servers. Connecting to an untrusted MCP server compromises the client, not just the other way round.

The rug pull nobody covers

Documented since April 2025: a server can change a tool’s description after the client has approved it. The description is what the model reads to decide when and how to use it, so changing it is changing the behaviour without asking for permission again.

The specification in force does not address it. It defines list change notification and cache freshness fields, which are recency mechanisms, not integrity ones. The closest normative warning is that clients must treat tool annotations as untrusted unless they come from trusted servers.

LiteLLM does not cover it either. The only “pinning” in the code is of the configuration’s server identifier, so that it does not change when you edit the YAML. There is no hash or comparison of the description or the input schema between successive listings. A search for related terms in the MCP module: nothing functional.

The control exists outside the gateway. mcp-scan, from the same researchers who documented the attack, implements description hash pinning precisely to detect this.

In ENS terms it is an unauthorised change to the security configuration that today goes undetected. And the home-made fix fits in a cron job: store the hash of the description and schema of every approved tool, compare against tools/list periodically, alert on the difference.

Three more holes in the code

No URL validation towards the upstream. A new server’s url field has no validator. The SSRF protection utilities exist in the tree and are used in OAuth discovery and in downloading OpenAPI specifications, but not in the egress of tools/call. A server pointing at a private range or at the cloud metadata IP is accepted as is.

The stdio transport is local execution on the proxy host. There is a command allowlist: npx, uvx, python, python3, node, docker, deno, extendable through an environment variable. The code’s own comment admits the residual risk: the permitted runtimes can execute code through arguments. With docker and npx in there, it is de facto arbitrary execution for any proxy administrator.

Two server modes skip the gateway’s authentication. Pass-through and authentication delegation to the upstream completely omit virtual key validation. They are well hardened (fail-closed if the target is mixed or unresolvable, strict comparison against truthy values, explicit blocking of the client credentials flow so as not to lend out the proxy’s service account), and the result is still that those servers have no gateway access control.

The limit that truncates in silence

The tools/list pagination stops and returns what it has accumulated in three cases: repeated cursor, one thousand pages, or expiry of a global deadline. The docstring declares it: a slow or faulty upstream produces a partial catalogue instead of an error.

The first two log a warning. The third, the deadline, exits without logging anything at all. A slow server makes part of its tools disappear from the catalogue with no trace left behind.

Correction on the slot leak

In the previous post I took issue 34534 at face value, according to which every MCP tool call acquires a max_parallel_requests slot and does not release it. Tracing the 1.102.0 code, that diagnosis no longer holds: the release path exists, on both success and failure, and the limiter is registered as a litellm callback and not just as a proxy hook.

With one honest caveat: I found no regression test covering the full cycle for MCP, and the definitive confirmation is a real proxy with max_parallel_requests: 2 and five MCP calls in a row. If anybody reproduces it today, I would like to know.

The specification moved underneath

The MCP revision in force is 2026-07-28, published on 28 July 2026, and in between there was another one, 2025-11-25. The changes in the latest one are not cosmetic:

Protocol sessions and the session header are removed. Cross-request state becomes an explicit identifier in the tool arguments.
The initialisation handshake is removed. Every request carries version and capabilities in _meta. MCP becomes stateless.
server/discover appears as a mandatory call.
Elicitation changes shape: out go server-initiated requests, in comes a multi-round pattern where the server returns an “input required” result and the client retries with the answers.
ping, logging/setLevel and the roots change notification are removed. SSE stream resumption is removed.
Standard HTTP headers are mandatory on POST, which lets a gateway or a web application firewall decide without parsing the JSON.
On authorisation: mandatory issuer validation in the client, and dynamic client registration is deprecated in favour of client identifier metadata documents.
Roots, sampling and logging are deprecated, with a minimum window of twelve months.

LiteLLM 1.102.0 advertises 2025-06-18. Its version enumeration has three values and none of them is later than that date. The literal is sent in the initialize against the upstream.

One nuance that is not entirely bad: the version the gateway advertises to its clients is not hard-coded in LiteLLM, the SDK negotiates it. And there is partial implementation of the intermediate revision, because the elicitation and sampling handlers reference it in their docstrings. That is: partial implementation of one revision while advertising an earlier one.

In practical interoperability it does not hurt yet, because the client ecosystem is still mostly on the 2025 revisions. It hurts in planning: July’s stateless redesign invalidates the gateway’s session model, and that is work ahead.

Two ecosystem notes for anyone choosing. There are clients that speak remote streamable HTTP natively (Claude and Claude Code, ChatGPT with OAuth and dynamic registration, VS Code with Copilot, Cursor, Zed, Kiro) and others that still demand stdio or a local bridge with mcp-remote, among them Windsurf, Cline and Continue.

Tool execution from the proxy itself

Besides serving MCP outwards, the proxy can consume it itself. It is enabled with an mcp block in the body, with server_url equal to the litellm_proxy sentinel, and it works both on /v1/responses and on /chat/completions.

The flow: it separates the gateway’s tools, lists them, filters by allowed, deduplicates, translates them into OpenAI function format and concatenates them. It calls the model. If there are tool calls and auto-execution is permitted, it runs them against the MCP servers. It builds the follow-up and makes one second call.

Two limits you need to know before designing on top of this.

The loop is a single hop. There is no multi-step iteration. The calls from the first response are executed and the follow-up is returned. A plan that chains tools requires the client to call again.

Auto-execution fails closed and as a block. It requires all MCP references to have approval set to “never”; a single one requiring approval disables auto-execution for the whole request.

And a third one, which is the one that bites: the multi-turn state of /v1/responses is rehydrated from the spend table. The query is literal, it looks up the session of the previous request_id and retrieves all the rows of that session ordered by time.

With disable_spend_logs: true nothing is written, the query comes out empty and the history is lost in silence, with no 4xx error. The code itself acknowledges it: it does not expect retries because deployments that do not write spend logs have nothing to wait for. It is a coupling between a performance option and a conversation feature that nobody expects to find.

Health and day-to-day operation

Health checks exist, they are on demand and they are not periodic. GET /v1/mcp/server/health accepts repeated identifiers and returns per-server status. The implementation opens a client and runs an empty operation inside the session, with a ten-second timeout.

It skips the check entirely if the server requires per-user authentication, and those always return unknown status.

The surprising part: the status, last check and error columns have existed in the database since a May 2025 migration, and the check function returns an object without writing to the database. Status is recalculated on every request. There is no periodic MCP health job.

When an upstream server is down, the behaviour is to degrade and not block. The listing fans out in parallel and each failing server contributes an empty list plus a result classified into six categories. Those results reach the client in the response’s _meta under a vendor key, without filtering the upstream’s text or URL. The reason for the design is in the code: an empty contribution with no signal makes a broken upstream indistinguishable from a healthy server with no tools.

There is no cache of the previous list. A server that is down disappears from that turn’s tools.

On registering and deregistering without restarting, there are two worlds. Database servers can be created, edited and deleted through the API, and every operation reloads the registry with an atomic swap. Propagation between replicas goes through Redis publication on the configuration change channel, if there is Redis, and through polling every 30 seconds if there is not.

Servers declared in config.yaml are loaded once, at start-up. There is no re-read. Changing them requires a restart.

And there is a precedence trap between the two worlds, with two different warnings in the code. A database row with the same identifier as a configuration entry hides the configuration one entirely. And the other way round, a configuration identifier that matches the name or alias of a database server captures its permissions, because identifiers are resolved before names. Both cases emit a log warning, once.

A hardened `config.yaml`

general_settings:
 # Access is not open by default, but this closes the empty-key
 # inheritance towards the team, which is the one that surprises people.
 require_key_mcp_access_defined: true
 require_end_user_mcp_access_defined: true
 # Without this, allow_all_keys overrides the key's explicit scope.
 mcp_allow_all_keys_respects_mcp_scope: true

litellm_settings:
 # Does not cover MCP arguments. See the auditing section.
 turn_off_message_logging: true
 callbacks: ["langfuse_otel"]

mcp_servers:
 inventario:
 url: https://mcp-inventario.interna.svc/mcp
 transport: http
 auth_type: bearer_token
 auth_value: os.environ/MCP_INVENTARIO_TOKEN
 # Explicit allowlist. Also enforced on tools/call.
 allowed_tools:
 - buscar_articulo
 - consultar_stock
 # Never true on a server with data.
 allow_all_keys: false
 access_groups: ["operaciones"]
 timeout: 20
 max_concurrent_requests: 8
 mcp_info:
 description: "Inventario de almacen, solo lectura"
 # Without this, MCP spend is zero and its metric does not exist.
 mcp_server_cost_info:
 default_cost_per_query: 0.0005
 tool_name_to_cost_per_query:
 consultar_stock: 0.0002

And in the process environment:

# Without this there is no MCP span, only a generic litellm_request.
LITELLM_OTEL_V2=true
# Full block hashes, not truncated, if correlating
# with engine events.
LITELLM_MCP_CLIENT_TIMEOUT=45
LITELLM_MCP_TOOL_LISTING_TIMEOUT=20

For clients, the rule is not to use the aggregated endpoint. Each agent points at /{server}/mcp or at a tool set, or at the three-tool proxy mode if the catalogue is large.

Checklist

Check the version. Below 1.83.14 there is remote execution and an unauthenticated SQL injection with an assigned CVE.
Inventory which servers have allow_all_keys and enable the setting that makes it respect the key’s scope.
Audit the access groups separately, because they are additive on top of the key and team ceiling.
Declare in writing the fail-open behaviour on tool permission resolution errors.
Measure the real cost: number of tools per server from the listing row, and tools block tokens from the stored request body.
Choose a catalogue strategy before connecting the third server: three-tool proxy mode, semantic filter, or selection by route. Not the aggregated endpoint.
Declare mcp_server_cost_info even with a token value, or the spend metric will not exist.
Enable the OTel v2 logger, and decide deliberately whether to enable content capture.
Assume the trace is cut at the gateway and plan correlation with the upstream by other means.
Treat the spend table’s metadata column as a personal data repository: encryption, its own retention and separate access control.
Set up the rug pull check, because nobody does it: hash of the description and schema of every approved tool, periodic comparison against the listing.
Restrict egress towards MCP servers at the network level, because the gateway does not validate the destination URL.
Do not enable the stdio transport unless you need it, and if you do, review what is on the command allowlist.
If you use /v1/responses with multi-turn state, do not enable disable_spend_logs.

Traps and things that are not what they seem

MCP access is not open by default in 1.102.0. I got that wrong in the previous post. What opens things up is per-server allow_all_keys.
allow_all_keys overrides the key’s explicit scope unless you enable a setting that ships disabled.
require_key_mcp_access_defined only closes key-to-team inheritance. Access group grants remain additive.
On an indeterminate failure resolving tool permissions, the code fails open for keys and JWTs.
The default cost of an MCP call is 0.0, and the spend metric is only emitted if the cost is greater than zero, so the series does not exist.
MCP calls appear in the per-team report as if they were models, named MCP: tool and with zero tokens.
The OTel logger with MCP support is off by default. Without the environment variable there is no MCP span.
Trace context is not propagated towards the upstream MCP server. There is no end-to-end trace.
mcp.tool.name does not exist in the semantic conventions. The name goes in gen_ai.tool.name.
Message redaction does not cover tool arguments. They are written in the clear into the spend table.
Registering, modifying and deleting MCP servers generates no audit log. There are three TODOs in the code.
The semantic tool filter is not applied on tools/list. Only on /chat/completions and /v1/responses.
The catalogue order is not deterministic, because it comes from iterating a set of strings. It breaks the client’s prompt cache even though nobody filters anything.
search_tools in proxy mode counts keywords by default, it uses neither regular expressions nor embeddings, and its cap of five results is hard-coded.
Description overrides really do replace what the client sees, and they are the only catalogue lever that does not break the cache. They are only configurable through the database or the API.
spec_version no longer exists in the configuration schema.
/mcp/sse does not use the SSE transport, and the messages endpoint its object advertises is dead code.
Listing pagination truncates in silence when the deadline expires, without leaving a single log line.
A database row hides a configuration entry with the same identifier, and the other way round a configuration identifier can capture the permissions of a database server.
Servers declared in config.yaml require a restart. Only the database ones reload hot.
The health check does not write its result to the database even though the columns exist, and there is no periodic job.
The gateway does not validate the upstream MCP server’s URL. The anti-SSRF utilities exist and are not used on that path.
docker and npx are on the stdio command allowlist.
The /v1/responses tool loop is a single hop, and disable_spend_logs: true breaks its multi-turn behaviour with no error.
LiteLLM advertises the June 2025 revision of the protocol, two behind the one in force.

Closing

There is a thread running through the six posts in this track that is more visible here than anywhere else. The gateway is chosen for one job and ends up doing five, and each new job arrives with its own defaults, which are almost never the ones you would have chosen.

In the case of MCP the asymmetry is twofold. On one side, the part that looks dangerous is better built than you would assume: access control is enforced on execution and not just on the listing, the hierarchy is built out of intersections, header forwarding is an allowlist, and default access is closed, contrary to what I myself wrote two days ago. On the other, the part nobody looks at is worse: tool arguments come out in the clear bypassing the privacy switch, spend is worth zero and its metric never gets to exist, the trace is cut, and a change to the description of an approved tool goes undetected by anybody in the whole chain, neither the specification nor the gateway.

The architecture decision that comes out of this is that the MCP door has to be treated as what it is, an interconnection of systems with third parties, and not as one more function of the LLM proxy. That means its own inventory, formal registration, change control over every tool’s description, network isolation towards the upstream and a log designed knowing that it is going to contain personal data.

And it means accepting that the cost you really pay for connecting fifteen MCP servers is in no column of the spend table. It is spread across the prompt_tokens of every model call, on every turn, and across the accuracy you lose when the catalogue is too large for the model to choose well. That number can be computed with what the gateway already stores. The first step is wanting to compute it.

Sources

LiteLLM, MCP Gateway: https://docs.litellm.ai/docs/mcp.
LiteLLM, código: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, auth/user_api_key_auth_mcp.py, cost_calculator.py, faults/list_outcomes.py, litellm/proxy/management_endpoints/mcp_management_endpoints.py, litellm/proxy/spend_tracking/spend_tracking_utils.py, litellm/litellm_core_utils/redact_messages.py, litellm/integrations/otel/, litellm/responses/litellm_completion_transformation/session_handler.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en llamadas MCP: https://github.com/BerriAI/litellm/issues/34534.
NVD, CVE-2026-42271, ejecución de comandos por los endpoints de prueba de MCP: https://nvd.nist.gov/vuln/detail/CVE-2026-42271.
NVD, CVE-2026-42208, inyección SQL sin autenticación en la validación de clave: https://nvd.nist.gov/vuln/detail/CVE-2026-42208.
NVD, CVE-2026-35030, colisión de clave de caché de userinfo OIDC: https://nvd.nist.gov/vuln/detail/CVE-2026-35030.
NVD, CVE-2025-49596, ejecución remota en MCP Inspector: https://nvd.nist.gov/vuln/detail/CVE-2025-49596.
NVD, CVE-2025-6514, inyección de comandos en mcp-remote: https://nvd.nist.gov/vuln/detail/CVE-2025-6514.
NVD, CVE-2025-66416, protección de DNS rebinding desactivada por defecto en el SDK de Python: https://nvd.nist.gov/vuln/detail/CVE-2025-66416.
Invariant Labs, MCP Security Notification: Tool Poisoning Attacks (tool poisoning, rug pull, shadowing): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks.
Invariant Labs, Introducing mcp-scan (fijado de herramientas por hash): https://invariantlabs.ai/blog/introducing-mcp-scan.
Model Context Protocol, changelog de la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/changelog.
Model Context Protocol, changelog de la revisión 2025-11-25: https://modelcontextprotocol.io/specification/2025-11-25/changelog.
Model Context Protocol, Security Best Practices (confused deputy, indicadores de recurso, validación de audiencia): https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.
NSA, Model Context Protocol: Security Design Considerations for AI-Driven Automation, mayo de 2026: https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDF.
CISA y agencias aliadas, Careful Adoption of Agentic AI Services, mayo de 2026: https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services.
Anthropic, Advanced tool use (coste en tokens de las definiciones, búsqueda de herramientas, precisión de selección): https://www.anthropic.com/engineering/advanced-tool-use.
Anthropic, Code execution with MCP: https://www.anthropic.com/engineering/code-execution-with-mcp.
Anthropic, Prompt caching (jerarquía tools → system → messages e invalidación): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.
Anthropic, Tool search tool (herramientas diferidas fuera del prefijo, umbral de 30 a 50 herramientas): https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, Prompt caching (el orden de las herramientas cuenta): https://developers.openai.com/api/docs/guides/prompt-caching.
OpenAI, Tool search (las herramientas descubiertas se cargan al final del contexto): https://developers.openai.com/api/docs/guides/tools-tool-search.
vLLM, Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html.
Microsoft, Use tools in chat (límite de 128 herramientas por petición en VS Code): https://code.visualstudio.com/docs/copilot/agents/agent-tools.
RAG-MCP, prueba de estrés de 1 a 11.100 servidores candidatos, mayo de 2025: https://arxiv.org/abs/2505.03275.
Benchmarking the Benchmarks, desalineación del 18,5 % entre etiqueta y realidad, junio de 2026: https://arxiv.org/html/2607.02577v1.
MCP-Atlas, taxonomía de fallos sobre 220 herramientas y veinte modelos, febrero de 2026: https://arxiv.org/html/2602.00933v3.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307.
MCP Tool Descriptions Are Smelly, auditoría de 856 herramientas de 103 servidores, febrero de 2026: https://arxiv.org/html/2602.14878v1.
ToolMenuBench, seis estrategias de filtrado y 26.460 ejecuciones, junio de 2026: https://arxiv.org/html/2606.15508.
Retrieval Models Aren’t Tool-Savvy (ToolRet), corpus de 43.000 herramientas, marzo de 2025: https://arxiv.org/abs/2503.01763.
Tools Are Not Islands, brecha entre Recall@3 y conjunto completo, julio de 2026: https://arxiv.org/html/2607.25718.
ToolFlood, saturación adversaria de la capa de recuperación, marzo de 2026: https://arxiv.org/html/2603.13950.
Scores Are Not Decisions, el K óptimo como función del coste, julio de 2026: https://arxiv.org/html/2607.27083v1.
Don’t Break the Cache, caché de prompts en tareas agénticas de horizonte largo, enero de 2026: https://arxiv.org/html/2601.06007v2.
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer, mayo de 2026: https://arxiv.org/html/2605.24660v1.
Semantic Tool Discovery for LLMs, marzo de 2026: https://arxiv.org/abs/2603.20313.
OpenTelemetry, convenciones semánticas de GenAI y MCP: https://github.com/open-telemetry/semantic-conventions-genai.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
IBM ContextForge, gateway y registro MCP: https://github.com/IBM/mcp-context-forge.
Docker MCP Gateway: https://github.com/docker/mcp-gateway.
agentgateway: https://github.com/agentgateway/agentgateway.

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

Thu, 10 Sep 2026 08:00:00 +0200

Fourth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability and virtual keys covered governance. Here comes the new tenant: agents, which speak the same protocol as humans and behave in a different way. Everything that follows is verified against LiteLLM 1.100.1 and vLLM 0.29.0, both from the first ten days of September 2026.

TL;DR

The day a team connects an agent to the corporate gateway, the platform changes regime without anyone touching a YAML. These are the seven conclusions.

LiteLLM’s priority parameter prioritises nothing under normal conditions. It is still marked as beta and its queue is only consulted when all the deployments in the group are in cooldown. With free capacity, the only thing it produces is a response header. On top of that, the success branch does not pop the item off the queue, so the list grows indefinitely; and on /chat/completions the value 0, which is the one used by all the documentation examples, is falsy in Python and disables queuing.

Separating traffic is done with two pools, not with priorities. The non-forgeable way is model_group_alias at key or team level: the client always asks for qwen-30b and the credential decides whether that resolves to qwen-30b-interactivo or to qwen-30b-agentes. The way that does not touch clients is tag_regex against the User-Agent, with a pattern like ^User-Agent: claude-code\/, which serves to classify load and not to control access, because that header is set by the client.

The useful admission control arrives in the 1.101 branch, with max_in_flight_requests_per_worker and a per-process queue that returns 503 with retry-after. It rejects before authenticating, so what it discards does not appear attributed to any key.

max_parallel_requests is only applied at virtual key level. The team, user, end customer and organisation endpoints accept the field and store it in the database, and nobody reads it. global_max_parallel_requests is a no-op unless the old limiter is reactivated, and general_settings.max_parallel_requests is read by no code path.

vLLM with --scheduling-policy priority does not push ahead anyone who is already generating. The only thing that changes is who it evicts when KV cache runs short. If the agents have filled max_num_seqs, a chat with the highest priority waits just the same. And the two knobs that in 2025 allowed short prompts to get ahead of long ones, max_num_partial_prefills and max_long_partial_prefills, disappeared from the code in 0.27.

The retry product reaches 45 calls per agent turn. Three from the client SDK times three from the router times up to five fallback groups. The proxy sets max_retries=0 on the SDK it uses itself, but it cannot touch the client’s. The timeout is per attempt, not per request, and LiteLLM’s Timeout comes out as a 408, which is retryable.

A single 429 takes a deployment out of the pool for five seconds, without going through the allowed_fails threshold. With an agentic burst against a fleet of four replicas, the whole pool enters cooldown and the gateway returns a 429 of its own to everybody, humans included.

You are here: the gateway layer with two classes of traffic

In the seven-layer stack this still lives in the gateway layer, with one difference compared to the previous three articles. There the variable was load: more requests, more teams, more budget to share out. Here the variable is the shape of the load.

A gateway sized for three hundred engineers using a chat works. The same gateway, with thirty of those engineers running coding agents, serves a fraction of the human traffic with the previous latency and the rest degrades. The GPUs stay just as busy, the utilisation metrics stay good, and the support team gets complaints from people waiting nine seconds for a paragraph.

The analogy: the registry desk and the courier with two hundred files

A registry office serves people in order of arrival. Citizens come in one at a time, present a piece of paper, and leave. The system is fair and the queue moves.

One Tuesday a courier turns up with a trolley of two hundred files. He joins the queue like anyone else, because rules are rules, and when his turn comes he occupies the desk for forty minutes. Behind him are twelve people with one piece of paper each. Nobody has broken any rule. The courier has the same right as everyone else and uses the same procedure. What fails is that the office was designed assuming a certain size of transaction and another one three orders of magnitude bigger has turned up.

The real solutions for an office are three, and all three have an exact equivalent in the platform. Open a desk for bulk deliveries, which is separating pools. Put up a capacity sign and send people back later, which is admission control with its 503. And say that files are handed in during the afternoon, which is the part that on an inference platform can almost never be applied, because the agent does not negotiate schedules.

What does not work in the office is giving the citizen a priority number when the courier is already at the desk. That number does not interrupt the transaction in progress. It is, literally, what priority does in the two layers of the platform, and it is worth keeping in mind while reading the next section.

The two kinds of traffic, measured

Before configuring anything, you have to be able to state the difference in numbers, because every later adjustment comes out of it.

Trait	Human chat	Agentic loop
Input per request	hundreds of tokens	tens of thousands, and it grows with each turn
Output per request	hundreds of tokens	from two tokens (a tool call) to several thousand
Concurrency per user	1	3 to 20 requests in flight
The metric that hurts	time to first token	total turn throughput
Tolerance for waiting	seconds	minutes
Cancelled requests	rare	routine, by the agent’s own decision
Retries	the browser’s	the SDK’s, plus the loop’s
Witness when it fails	the user	nobody

Two rows deserve comment because they contradict intuition.

The first is the metric row. An agent does not perceive time to first token, so on a shared engine it is the natural candidate to give up that metric. The operational conclusion is that the agent pool can be configured for aggressive throughput, with large batches and deep queues, while the interactive pool is configured for the opposite.

The second is the cancellations row. An agent that launches three speculative continuations and discards two generates two aborted streams per turn. At the gateway, that touches spend accounting, the release of concurrency counters and the open connection to the engine. LiteLLM 1.100 handles it reasonably well, and it did not six months ago.

What does not work: `priority` in LiteLLM

The first reaction of any operator is to look for a priority field. LiteLLM has one, documents it on its Request Prioritization page, and that page carries the beta label and the sentence saying it is for testing. The label is correct and has to be taken literally, because behind it there are four things.

The queue is only consulted when the whole group is in cooldown. The scheduler’s poll() method returns true if there is any healthy deployment, without touching the queue. With available capacity, which is the normal situation, the request goes straight through and priority orders nothing. The only thing that changes is that the response carries the x-litellm-request-prioritization-used header.

The queue does not empty. In that success branch there is no heappop and no call to the cleanup, which only happens in the expiry branch. The list under the key scheduler:queue:{model} grows without bound, is serialised whole into Redis on every insertion, and there is no maximum size and no discard policy. When the group does enter cooldown, the head of the heap is occupied by fossil entries from requests that finished hours ago, and new ones compare their identifier against that head, never match, and wait until the timeout runs out.

On /chat/completions, the highest priority disables the feature. The code reads the value with a kwargs.get("priority") or self.default_priority, and in Python 0 or None is None. Since in this scheduler the lowest number is the highest priority, priority: 0, which is the value in the documentation’s own examples, goes down the path without queuing. On /completions the check is done with is not None and it works; the two routes do not behave the same.

And default_priority in router_settings breaks the request. If the client does not send a priority but the global setting is in place, the value is truthy, the scheduling branch is entered and a function is invoked that requires the positional argument nobody has injected. The setting has no associated test and does not appear in the router_settings reference.

There is also a /queue/chat/completions, marked as experimental and hidden from the OpenAPI schema, which reads data["priority"] with no default value: a request without that field raises an exception that comes out as a 400 authentication error, with a message that has no relation to the cause.

There is a priority feature that is solid, and it is a different thing. The v3 dynamic limiter shares out token and request per minute quota between classes with weights, through litellm.priority_reservation with values like {"premium": 0.75, "standard": 0.25}. It shares out capacity, it does not order a queue: below the saturation threshold it lends free capacity and above it applies the weights. It never makes anyone wait, either it passes or it returns a 429. It requires Postgres and an enterprise licence, and it is still marked as beta.

What does work: two pools

Effective separation is done earlier, by deciding which set of replicas each class of traffic goes to. LiteLLM offers two mechanisms with very different security properties.

By credential: `model_group_alias` at key or team level

Router settings are resolved in the order key, team, global. Among those that accept that override is model_group_alias, and there lies the piece:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 model_info:
 id: chat-01
 max_input_tokens: 32768
 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 model_info:
 id: agentes-01
 max_input_tokens: 131072

And on the agent team’s key:

curl -X POST "$PROXY/key/generate" \
 -H "Authorization: Bearer $MASTER_KEY" \
 -d '{
 "team_id": "plataforma-agentes",
 "models": ["qwen-30b"],
 "router_settings": {
 "model_group_alias": {"qwen-30b": "qwen-30b-agentes"}
 }
 }'

The client asks for qwen-30b and knows nothing about any of it. The resolution is tied to the credential, so the caller cannot get around it by changing a header. The same list allows routing_strategy, fallbacks, context_window_fallbacks, retry_policy, cooldown_time and allowed_fails to be set per key or team, which gives different reliability policies for each class of traffic on the same proxy instance.

It should not be confused with access groups (model_info.access_groups), which are permissions and not routing: they decide whether a key can invoke a model name, not which replica the request goes to.

By `User-Agent`: `tag_regex`

When separate credentials cannot be issued, because the agentic client uses the same personal key as that person’s chat, LiteLLM classifies by regular expression against the request headers:

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 tag_regex: ["^User-Agent: claude-code\\/", "^User-Agent: .*codex"]
 model_info: {id: agentes-01}
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 tags: ["default"]
 model_info: {id: chat-01}

router_settings:
 enable_tag_filtering: true
 tag_filtering_match_any: true

The proxy composes strings of the form User-Agent: <value> and tests them with re.search. Exact tag matching takes precedence over the regex. The documentation itself carries a warning that has to be respected: the User-Agent is written by the client, so this is traffic classification and not a security boundary. As load sharing it is enough, because an agent that lies about its identity to sneak into the interactive pool is a governance problem, not a routing one.

The behaviour when no deployment carries the requested tag has a nuance that comes as a surprise in production. If the tag is unknown to the group, the request falls through to the pool marked with tags: ["default"]. If the tag exists in the group but no healthy deployment carries it at that moment, the request fails with the tag configuration error, even though a default pool is available. To get deterministic fallthrough to the default pool you have to set model_info.allow_fail_open: true.

A different strategy per pool with `routing_groups`

router_settings:
 routing_strategy: simple-shuffle
 routing_groups:
 - group_name: interactivo
 models: [qwen-30b-interactivo]
 routing_strategy: simple-shuffle
 - group_name: agentes
 models: [qwen-30b-agentes]
 routing_strategy: simple-shuffle

The recommendation to leave simple-shuffle on both is not laziness, and sustaining it requires looking at what the alternatives do on a homogeneous vLLM fleet.

least-busy keeps a complete dictionary under a single key and updates it with non-atomic reads and writes, without using INCR, so with several proxy replicas increments are lost. It has no TTL, so a request that never completes leaves the counter raised forever and that replica is permanently excluded. Counters can go negative. And when the minimum corresponds to an identifier that is no longer in the healthy list, it falls back to random.choice without warning. The pattern of requests that do not complete is exactly that of an agent that cancels.

latency-based-routing measures seconds per output token, not wall-clock latency. A chat turn of twenty tokens looks extremely slow by that metric and an agent generation of two thousand tokens looks extremely fast, so mixing both kinds of traffic in the same group poisons the signal. It also seeds new deployments with zero latency, and with the buffer defaulting to zero, a freshly started replica takes all the traffic until its first success callback.

usage-based-routing-v2 uses minute windows that reset all at once, and it accounts for tokens when the request completes, so long generations in flight are invisible to the router. With long outputs, it systematically underestimates the real load.

cost-based-routing sorts by the sum of the input and output unit prices and takes the first, with no random tie-break. With replicas of the same model at the same price, all the traffic always goes to the same place. And a model that is not in the price map gets a default cost of 5.0 per token, a figure designed to deprioritise it.

Admission control: the new piece in the 1.101 branch

When the agent pool saturates, what you want is to reject fast and with a code the client understands, not to accumulate requests inside the proxy until memory runs out. That arrives with per-worker admission control, added in the 1.101 branch, in release candidate as this is written.

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

It is a semaphore with a per-uvicorn-process queue, without Redis. When the queue fills or the wait expires it returns 503 with retry-after: 1 and a body with "type": "overloaded_error". It is observed through /health/backlog, which exposes requests in flight, admitted, queued and rejected, and through the Prometheus litellm_admission_* metrics, whose rejection counter distinguishes queue_full from queue_timeout.

Two properties have to go into the runbook. The middleware rejects before authenticating, so what is discarded is not attributed to any key in the spend records, and the real number of requests rejected per team cannot be reconstructed from the spend table. And the limits are read on the first request: changing them requires a restart.

Concurrency per key, and the three settings that do nothing

max_parallel_requests is the right brake for a specific agent, and its configuration surface is full of dead ends.

The only level where it is applied is the virtual key. The v3 limiter, active by default since 1.94, has descriptors for user, team, team member, end customer, organisation, model per key, tag and agent, and all of them carry only request and token counters per window. The only reference to max_parallel_requests is the one in the key descriptor. The old limiter says so in a comment: supporting it for model, user and team is pending.

At the same time, /team/new, /user/new, /customer/new and the organisation endpoints accept the field, validate it and write it to Postgres, and the documentation shows it in their examples. Nobody reads it afterwards. A team with max_parallel_requests: 20 in its row has no concurrency limit at all.

global_max_parallel_requests is injected into the request metadata and is only consulted by the old limiter, which is disabled unless LEGACY_MULTI_INSTANCE_RATE_LIMITING=true is set. The server settings documentation still describes it as a global limit coordinated through Redis. And general_settings.max_parallel_requests does not appear in any execution path: its only two appearances are the copy from the database and the type schema of the interface.

What does work, in detail:

curl -X POST "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" \
 -d '{"team_id":"plataforma-agentes","max_parallel_requests":8,"rpm_limit":600}'

Counting is done with a sorted set in Redis scored by Redis’s own clock, with Lua scripts to acquire and release. Each slot expires after 3,600 seconds, which is at once the maximum request duration the meter can track and the time a leaked slot takes to heal by itself. Release is hooked to success, to failure and to client disconnection during streaming, which was the classic leak with clients that cancel.

One live leak remains and it affects exactly the case in this article: every tool call through the MCP gateway acquires a slot and does not release it. The MCP route invokes the pre-call hook, which acquires, and none of the release functions is called from that module. With an agent that makes twenty tool calls per turn and a key with eight slots, the 429 arrives on the first turn and lasts an hour.

The order of evaluation matters for interpreting 429s. First the window counters, requests and tokens per minute, are checked, and only if they pass is the concurrency slot acquired. The code comment explains why: the other way round, every rate limit rejection would leave an orphan slot. In practice, with a low rpm you always see the requests type 429, and the max_parallel_requests one only appears when the relationship between concurrency and duration makes it binding first. The message distinguishes the two cases with the Limit type: field and the rate_limit_type header.

A minor detail with consequences on the client: the response to a concurrency rejection carries retry-after and reset_at computed from the window size, sixty seconds by default, when a concurrency meter has no window. A well-behaved client that respects that header will wait a minute when it might have had a free slot in two seconds.

Session affinity: the KV cache that gets thrown away

An agent turn resends the whole conversation context, system prompt and tools included. If that request lands on a different replica from the previous turn’s, vLLM’s prefix cache does not have the prefix and the prefill is recomputed in full. With twenty turns and four replicas balanced blindly, the long prefill is paid for fifteen times over.

LiteLLM has two mechanisms for this and they are not equally good.

The prompt_caching check computes a SHA-256 of the cacheable prefix, defined as everything up to the last block with cache_control, and pins the deployment with that hash. It has two problems. The TTL is hardcoded to 300 seconds in two places in the code, with no way to configure it, something which is reported and open; with one-hour caches, affinity is lost after five minutes of inactivity. And the key is an exact hash, so a client that moves the cut point forward on every turn, which is exactly what agents do to extend the cached prefix, generates a new key every time.

The session_affinity check is the one to use:

router_settings:
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

It pins the deployment with an atomic Lua script in Redis, with a configurable TTL that defaults to one hour and that is refreshed on every request, so it bounds the idle time between turns and not the duration of the conversation. If the pinned deployment enters cooldown, the request goes out through the normal strategy and keeps the pin so it can come back when it heals.

The session identifier comes from a precedence chain that starts with x-litellm-trace-id, continues with x-litellm-session-id and then accepts any header of the form x-<something>-session-id whose value looks like an identifier. That includes the session header Claude Code already sends, with nothing configured on the client. Whatever matches populates both the session identifier and the trace identifier, which links this to the correlation from the first article in the track.

One dangerous interaction when combining it with tags: affinity is evaluated before tag-based routing. If the pin reduces the list to a deployment that does not satisfy the request’s tag, the request fails instead of falling through to the default pool, unless allow_fail_open has been enabled. Retagging a pool with live sessions breaks them.

And a clarification that saves a pointless search: LiteLLM does not do prefix routing. There is no prefix tree and nothing that directs the request to the vLLM replica that already has that prefix warm. What there is, is affinity by session or by key. For the other thing you need a specific router underneath the gateway, and that architecture has its own article pending.

The engine: what vLLM can and cannot do

Everything above shares out requests between engines. Inside one engine, the question is what happens when a twenty-token request and a sixty-thousand-token one live side by side.

Priority exists and does less than it looks like. --scheduling-policy priority turns the waiting queue into a heap ordered by priority, arrival time and identifier, with the lowest number first. The request can carry the priority field in the body, and there is also the X-Vllm-Priority header, which takes precedence over the body and is not documented anywhere.

What priority does not do is push ahead of whoever is already generating. In the running queue loop, when KV cache runs short, the policy only changes who is chosen as the victim of eviction: with priority the one with the worst priority and arrival time is evicted, and with fcfs the last in the queue. In the waiting queue loop, if there are no blocks available the iteration is cut short, without evicting anyone. And the max_num_seqs cap is a hard cut. If the agents have filled that cap, a chat with maximum priority waits just the same. There has been an open issue since April and an unmerged pull request about this.

That nuance about the victim is not negligible, because with fcfs the victims are the most recently admitted, that is, the chats that have just come in. Switching the priority policy on already improves the situation even if it pushes nobody ahead.

Eviction is always by recomputation. The swap-to-host-memory mode and the enumeration that controlled it disappeared from the code; all that is left is freeing blocks and returning the request to the queue with the computed token counter at zero. An evicted request loses all its prefill, and the prefix cache recovers part of it, but the perceived time to first token restarts.

Two admission settings help and do not appear in the narrative documentation. watermark reserves a fraction of free KV blocks on admission, with the declared purpose of avoiding repeated eviction, and it ships disabled. And scheduler_reserve_full_isl, active by default, checks that the full input length fits in the cache before admitting, instead of looking only at the first chunk.

Chunked prefill is active by default and prioritises decode. The scheduler walks the whole running request loop before touching the waiting queue, so a long agent prefill does not freeze a chat’s generation in progress. That protects the inter-token latency of what has already been admitted, and it does not protect the time to first token of what is waiting.

Here there is a breaking change that forces a review of the manifests. max_num_partial_prefills and max_long_partial_prefills were removed in 0.27. They were precisely the mechanism that allowed short prompts to get ahead of long ones. A deployment with those flags in its args does not start. long_prefill_token_threshold still exists, and its meaning changed: today it is a per-request chunk size cap, not a classification of long requests.

The partial replacement arrives in 0.29 with max_num_queued_tokens and max_num_queued_reqs, admission valves that return 503 when the pending prefill work exceeds the target. The code itself frames it as a quality of service mechanism for time to first token, with the formula of setting it to the latency target multiplied by prefill throughput. They are blind to the class of traffic: they reject a chat just the same as an agent.

And the default value of --max-num-batched-tokens depends on the hardware and on the start-up mode. With vllm serve on H100 or H200 it is 8,192, on cards of 160 GiB or more it is 16,384, and on the rest 2,048. The 2,048 that appears in the scheduler configuration only applies to tests.

The metrics for diagnosing coexistence, with the exact names from 0.29:

Metric	What for
`vllm:num_requests_waiting`	queue depth, the autoscaling signal
`vllm:num_requests_waiting_by_reason`	separates waiting for capacity from waiting in a blocked state
`vllm:num_preemptions_total`	evictions, the signal that the two classes are treading on each other
`vllm:kv_cache_usage_perc`	cache pressure, the cause of the evictions
`vllm:request_queue_time_seconds`	what a chat waits with the agents inside
`vllm:prefix_cache_hits_total` and `_queries_total`	measured in tokens, not in requests

A warning for anyone with old alerts. The log warning about evicted sequences that appears in the optimisation documentation belongs to the previous version of the engine and no longer exists in the code. Preemptions are only visible as a counter. An alert based on searching for that string in the logs has not fired for months, and not because there are no evictions.

On isolation between classes inside one engine, the short answer is that it does not exist. There are two queues and neither is per class of traffic. The conclusion, which is the one that holds up this article’s architecture: effective isolation means separate fleets, and on top of them the gateway deciding who goes to which.

Retries: the product nobody calculates

A human who gets an error sees it and decides. An agentic loop retries, and beneath it there are two more layers that also retry.

Layer	Default
Client SDK	2 retries, 3 attempts
LiteLLM router	2 retries, 3 attempts
Fallbacks	up to 5 groups

The product is about 45 calls to the engine for one agent turn, before counting the loop’s own retry. LiteLLM does what it can: when the call comes from the router, it forces max_retries=0 on the SDK it itself uses against the engine. What it cannot touch is the client’s SDK. The only layer that breaks the product is that one, and it is broken by setting max_retries=0 on the client and delegating the retry to the gateway.

The retryable errors are 408, 409, 429 and anything 500 or above. And there lies the problem with long reasoning: litellm.Timeout is a subclass of OpenAI’s timeout exception and comes out with code 408, which is retryable, so the most expensive request of all, the one that expired after ten minutes of generating, is launched again in full. Twice by the router, and as many times again by the client.

On timeouts you have to be precise because the code comment is: the configured timeout is per attempt, not per request. A request_timeout: 600 with two retries and two fallback groups gives a worst case of almost two hours before the client receives anything. The default value is still 6,000 seconds, as was already covered in the day 2 article.

The retry storm has two specific causes and both are default settings.

The first: a single 429 takes the deployment out of the pool, without going through the allowed_fails threshold. The cooldown logic has an explicit branch which, on a 429 in a group with more than one deployment, returns immediate cooldown. With an agentic burst against four replicas, all four leave the pool at once and the gateway answers everybody with its own 429 of “no deployments available”. The default cooldown is five seconds, which in a sustained burst gets renewed.

The second: backoff is zero while the pool looks healthy. The function that computes the wait returns 0 if the healthy deployment list is not empty. Exponential backoff with jitter, capped at eight seconds, only kicks in when there is nothing healthy left. Under partial degradation, the proxy hammers away without waiting and only slows down once everything has already fallen over.

On top of that, the engine’s Retry-After header is only respected if its value is between 1 and 60 seconds. A longer limit window is ignored and replaced by the proxy’s own backoff.

The settings that cut the problem off:

router_settings:
 num_retries: 1
 retry_after: 2 # real floor; the default 0 is hammering
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3 # stops a 429 taking the deployment out
 retry_policy:
 TimeoutErrorRetries: 0 # do not relaunch the expensive request
 AuthenticationErrorRetries: 0
litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false

That last setting deserves an explanation, because it is a silent way of breaking an agent. With drop_params: true, LiteLLM removes from the body any parameter the destination provider does not support, and in that branch it writes no warning at all. The candidates for disappearing are tools, tool_choice, response_format and parallel_tool_calls. An agent that has tools taken away receives free text where it expected a function call, and interprets it as the model having decided not to use tools.

Worse still is prompt-based emulation. For a non-OpenAI-compatible provider that does not support tools, LiteLLM does not fail: it turns the tools into text inside the prompt, forces JSON format, discards tool_choice, and on the way switches on a process-global variable that makes any other tool-less provider in the same process go to emulation instead of erroring.

With that in mind, the warning about fallbacks is clear: LiteLLM does not check capabilities before falling back. The function that knows whether a model supports function calling exists and is only used in an informational endpoint. The pre-call checks filter by context window, request limit and region, and nothing else. A fallback list that goes from a model with tools to one without them degrades the agent silently. There are guards for provider-scoped resources, files and batches, so the absence of the capability guard is an omission and not a design oversight.

A month’s budget in twenty minutes

A runaway agent consumes in minutes what a team consumes in weeks, and the operational question is how long the gateway takes to find out.

The answer has changed, and the old mental model, the one where spend consolidates every minute and until then there is no brake, is no longer correct. Since 1.84 there are two new pieces. An optimistic reservation before the call, which estimates the maximum cost of the request and reserves it atomically in the counter before heading out to the engine, reconciling it with the real cost on completion. And a counter increment after the call that is explicitly awaited, with the comment that this way the counter is up to date before the next request goes through authentication. Budget checks read that counter, with Redis first and seeding from the database when cold.

Batch writing to Postgres still exists, every 10 to 15 seconds, and it affects persistence and what is shown in the interface, not the cut-off. The production documentation’s recommendation to raise that interval to 60 does not relax the control, and it does delay by a minute what /key/info shows, something which frequently generates operational panic. And the budget rescheduler’s ten minutes do not consolidate spend: they fire the job that resets expired budgets.

The overspend window then comes down to the cost of the requests in flight, provided there is Redis. Without Redis, the counters are per pod, and with N replicas you can spend up to N times the budget. That is the configuration failure that matters and it is the same Redis that the day 2 article declared non-optional for other reasons.

Two settings that do not appear in the public documentation, only in the data model descriptions, and that decide behaviour at the worst moment:

general_settings:
 fail_closed_budget_enforcement: true # false by default: if the counter fails, it lets things through
 # disable_budget_reservation: do not set it to true

With the default value, if the counter backend does not respond, the proxy fails open and carries on serving. For an agent in a loop that is exactly the opposite of what you want.

Time for the honest warning: this route is still unstable. There are open issues in both directions, one where the key and user budget limit stops being applied while spend is recorded, and another where a 429 is returned for stale spend in the reservation counters while the key query shows spend below the limit. Project budgets are not on the atomic reservation route and concurrency overshoots them. The practical conclusion is not to trust the hard cut-off to the budget alone.

On the manual cut-off, there is an asymmetry to know about before you need it. /key/block invalidates the local cache, the Redis one, and notifies all workers by publish, so with Redis the effect is almost immediate. /customer/block does the same. /team/block only writes to the database, without invalidating or notifying, so it takes however long the management object cache TTL takes, sixty seconds by default. To cut a team off dead you have to block its keys one by one, or lower user_api_key_cache_ttl. And /user/block no longer exists, even though the docstring of the function that replaced it still shows that route in its example.

On alerts, two numbers. The default thresholds warn at 85 % and 95 % of the budget. And deduplication of those alerts is 24 hours per event and identifier, so crossing 85 % generates one warning and then silence for a day. For agentic traffic, where 85 % and 100 % can be twenty minutes apart, the second warning arrives when it is no longer any use.

Attribution: who the user is when the caller is an agent

With agents, the question of who the spend is charged to has two answers and both are needed: the agent that executes and the human on whose behalf it acts.

The chain that extracts the end customer identifier tries the standard headers first, then the configurable header mapping, and only then the user field of the body, the litellm_metadata.user of the Anthropic format, the metadata.user_id and the safety_identifier. The last four are body fields and are therefore written by the client. The code itself warns about it: for untrusted callers the identifier has to be set from headers or from a middleware on the server, to avoid impersonation.

Out of that comes the correct pattern, which is the inverse of the one people put together out of inertia: the virtual key identifies the agent and is the non-forgeable part, and the header or the user field identify the human and serve for attribution, not for control. An agent that lies about the human behind it is still bounded by the budget and the limits of its key.

The spend table stores user, end_user, team_id, organization_id, request_tags, session_id and agent_id, with daily aggregation per agent, so the detail is available once it is sent. The x-litellm-agent-id header is what populates that last field.

A note about the MCP gateway, which on a platform with agents ends up being the other entry door. Tool calls do appear in the spend records, with call types of their own and a metadata block that stores tool name, arguments, result and server. What is missing is token counting: the cost is 0.0 by default and is only populated if the administrator configures a price per query, per server or per tool. It is the same pattern of silent zero cost that already came up with unpriced models in the first article in the track.

And the permission setting to review on day one: access control for MCP servers is open by default. If no level of the hierarchy defines a list, the request reaches every configured server. general_settings.require_key_mcp_access_defined: true inverts that inheritance, and it is not the default value.

A reference `config.yaml`

All of the above, together, on a fleet split into two pools:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018 # without this, silent zero cost
 output_cost_per_token: 0.00000072
 model_info:
 id: chat-01
 max_input_tokens: 32768 # without this there is no context filter

 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072
 tag_regex: ["^User-Agent: claude-code\\/"]
 model_info:
 id: agentes-01
 max_input_tokens: 131072
 allow_fail_open: true

router_settings:
 routing_strategy: simple-shuffle
 enable_pre_call_checks: true
 enable_tag_filtering: true
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600
 num_retries: 1
 retry_after: 2
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3
 retry_policy:
 TimeoutErrorRetries: 0
 AuthenticationErrorRetries: 0

general_settings:
 fail_closed_budget_enforcement: true
 require_key_mcp_access_defined: true
 user_api_key_cache_ttl: 10
 proxy_batch_write_at: 10
 # 1.101 branch onwards:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false
 cache: true # Redis, not optional

And the start-up of the two engines, with the deliberate asymmetry:

# interactive pool: protect the first token
vllm serve Qwen/Qwen3-30B \
 --scheduling-policy priority \
 --max-num-seqs 64 \
 --max-model-len 32768 \
 --max-num-batched-tokens 4096

# agent pool: throughput and deep queues
vllm serve Qwen/Qwen3-30B \
 --max-num-seqs 256 \
 --max-model-len 131072 \
 --max-num-batched-tokens 8192

Checklist

Issue separate keys for agents, with model_group_alias at key or team level. It is the only separation the client cannot evade.
Add tag_regex over the User-Agent as a safety net for agents that use personal credentials, with a default pool and allow_fail_open: true.
Leave simple-shuffle on both pools. The other four strategies have verifiable defects on homogeneous replicas.
Enable session_affinity with a one-hour TTL. Do not use prompt_caching for this while its TTL stays hardcoded.
Declare max_input_tokens on every deployment. Without that value, the context window filter is not applied and it fails silently for models that are not in the price map.
Declare per-token prices on every self-hosted model, or the agents’ spend will be zero and their budget will never run out.
max_parallel_requests on the agent’s key, never on its team. On the team it is stored and not applied.
Lower num_retries to 1, put a floor on retry_after, raise cooldown_time and disable retry on timeout. On the client, max_retries=0.
fail_closed_budget_enforcement: true and mandatory Redis. Without Redis, N replicas are N times the budget.
Start the interactive engine with --scheduling-policy priority even though it pushes nobody ahead: it changes who it evicts.
Alert on vllm:num_preemptions_total and vllm:num_requests_waiting_by_reason, not on log strings that are no longer emitted.
Prepare the cut-off: block by key, never by team, and verify the real propagation time in your own deployment.

Traps and things that are not what they seem

priority: 0 on /chat/completions disables prioritisation. It is the value in the documentation examples and it is falsy in Python.
max_parallel_requests on a team is stored and not applied. The field exists in the API, in the database and in the documentation; the limiter only reads it from the key descriptor.
global_max_parallel_requests is a no-op with the default limiter, and the server settings documentation still describes it as active.
Every MCP call leaks a concurrency slot, which only heals when its hour expires. With agents calling tools in a loop, the 429 appears early and lasts a long time.
The TTL of prompt cache affinity is 300 hardcoded seconds. For agent conversations, session affinity is the right option.
Affinity is evaluated before tag-based routing. A pin to a deployment that stops satisfying the tag makes the request fail instead of falling through to the default pool.
LiteLLM’s timeout is per attempt. The worst case is the timeout multiplied by attempts and by fallback groups.
A 429 takes the deployment out of the pool without going through allowed_fails, and backoff is zero while anything healthy remains.
The engine’s Retry-After is ignored if it asks for more than 60 seconds.
drop_params: true removes tools and response_format without leaving a warning, and the fallback does not check whether the destination model supports function calling.
/team/block does not invalidate the cache. It takes the full TTL; /key/block does propagate by publish in Redis.
Access to MCP servers is open by default if no level defines a list.
max_num_partial_prefills and max_long_partial_prefills no longer exist in vLLM. A manifest with those flags does not start from 0.27 onwards.
vLLM’s priority does not push ahead of running requests, and with max_num_seqs full it pushes nobody ahead.
The eviction log warning that appears in the vLLM documentation is no longer emitted. Only the Prometheus counter is left.

Closing

The coexistence of humans and agents on the same platform is not solved with a priority field, and that is the conclusion that takes the most effort to accept because the field exists and looks made for this. Neither the gateway nor the engine has a usable notion of class of service today: one orders a queue that is almost never consulted, the other picks who to evict. What they do have is the ability to direct each request to a different place, and that is where the win is.

The architecture that comes out of all of the above fits in one sentence. Two vLLM fleets with opposite parameters, a credential that decides which one each request goes to, session affinity so as not to throw away the prefill work, concurrency limits on the agent’s key and not on its team, retries trimmed back on the three layers where they multiply, and budgets that fail closed.

The rest is measurement. If vllm:num_preemptions_total rises in the interactive pool, the two classes are treading on each other despite the separation. If chat queue time grows while total throughput holds, the partition is badly sized. And if the agents’ spend comes out as zero, the problem is not coexistence but undeclared prices, which is where this track started.

Sources

LiteLLM, [BETA] Request Prioritization: https://docs.litellm.ai/docs/scheduler.
LiteLLM, Router - Load Balancing (estrategias, afinidad de sesión, comprobaciones previas): https://docs.litellm.ai/docs/routing.
LiteLLM, Tag Based Routing (incluye tag_regex y el ejemplo con clientes agénticos): https://docs.litellm.ai/docs/proxy/tag_routing.
LiteLLM, Router Settings for Keys and Teams (resolución clave, equipo, global): https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.
LiteLLM, Server Tuning (control de admisión por worker): https://docs.litellm.ai/docs/proxy/server_tuning.
LiteLLM, Dynamic TPM/RPM Allocation (reserva de prioridad, beta y enterprise): https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.
LiteLLM, Budgets, Rate Limits: https://docs.litellm.ai/docs/proxy/users.
LiteLLM, Production Best Practices: https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Reliability (fallbacks, ventana de contexto, enfriamientos): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, código: litellm/scheduler.py, litellm/router_strategy/tag_based_routing.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/hooks/parallel_request_limiter_v3.py, litellm/proxy/middleware/admission_control_middleware.py, litellm/router_utils/cooldown_handlers.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de la afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en cada llamada MCP: https://github.com/BerriAI/litellm/issues/34534.
LiteLLM, incidencia 26672, presupuesto de clave y usuario no aplicado: https://github.com/BerriAI/litellm/issues/26672.
LiteLLM, incidencia 872, comprobación previa de soporte de llamada a funciones, cerrada sin implementar: https://github.com/BerriAI/litellm/issues/872.
vLLM, Optimization and Tuning (prefill troceado, prioridad de decodificación): https://docs.vllm.ai/en/v0.29.0/configuration/optimization.html.
vLLM, Metrics design (nombres, convención _total, autoescalado como problema abierto): https://docs.vllm.ai/en/v0.29.0/design/metrics.html.
vLLM, Data Parallel Deployment (caché KV independiente por réplica): https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment.html.
vLLM, código: vllm/v1/core/sched/scheduler.py, vllm/v1/core/sched/request_queue.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/metrics/loggers.py: https://github.com/vllm-project/vllm.
vLLM, incidencia 40004, la prioridad no desaloja peticiones en ejecución: https://github.com/vllm-project/vllm/issues/40004.

When MCP grows up: giving it authentication with Keycloak

Thu, 18 Jun 2026 08:30:00 +0200

When a child is small, it is enough to make sure they do not hurt themselves. You put a gate on the stairs, you cover the sockets and little else. But the child grows. They start going out alone, handling money, making decisions that have consequences. And then raising them stops being “don’t let them fall” and turns into something far more demanding: teaching them to identify themselves, to ask permission, to know how far they can go and to account for what they do.

That is exactly what has happened with our MCP servers.

MCP is no longer a toy

Not long ago, a Model Context Protocol server was an experiment running on someone’s laptop so a model could read a couple of files or call an internal API. It did not matter who used it: it was us, locally, with nothing at stake.

That is over. MCP servers have become infrastructure pieces that AI uses to touch real systems. And, like any child who grows up, they have developed new needs they did not have before:

Multitenancy: there is no longer a single user but many, and one user’s data cannot be visible to another.
Cost control: every model call consumes tokens, compute and, sometimes, paid APIs. You have to measure, attribute and set limits.
Authentication and authorisation: knowing who is on the other side, what permissions they hold and which specific server their credential is good for.

These are three separate fronts. Today we focus on the third, which is the one that opens the door to the other two: authentication. Without knowing who is calling, there is no tenant to separate and no cost to attribute.

Why MCP authentication is a problem with a name of its own

The good news is that nothing has to be invented here. The MCP specification did not pull a security mechanism of its own out of a hat: it builds on OAuth, the same standard that has been behind half the internet’s “Sign in with…” for years. MCP defines how an MCP server acts as a protected resource and delegates credential issuance to an authorisation server.

And that authorisation server can be Keycloak, the open source identity manager from the Cloud Native Computing Foundation that you probably already run for the rest of your applications. The idea is not to add one more identity silo just because we are now talking to models: reuse the one that already governs your users.

The important nuance is that MCP is not a single frozen standard but a living specification with several versions, and each one asks for different things. As of today four coexist:

2025-11-25 (the latest)
2025-06-18
2025-03-26
2024-11-05 (the initial one, which did not even contemplate authorisation)

Just as with raising a child what they need at 6 is not what they need at 16, what Keycloak has to satisfy depends on the MCP version you work with. Let us look at it.

What MCP demands from an authorisation server (and what Keycloak provides)

The specification lists a series of OAuth standards the authorisation server must support, with different levels of requirement (MUST, SHOULD, MAY). This is the picture, cross-referenced with what Keycloak supports:

Standard	2025-11-25	2025-06-18	2025-03-26	Keycloak
OAuth 2.1 Authorization Framework (draft)	MUST	MUST	MUST	Supported
OAuth 2.0 Authorization Server Metadata (RFC 8414)	MUST	MUST	MUST	Supported
Resource Indicators for OAuth 2.0 (RFC 8707)	MUST	MUST	—	Not supported
Dynamic Client Registration (RFC 7591)	MAY	SHOULD	SHOULD	Supported
OAuth Client ID Metadata Document (draft)	SHOULD	—	—	Supported (experimental)

One clarification: MCP also adopts OAuth 2.0 Protected Resource Metadata (RFC 9728), but that standard is the MCP server’s business, not the authorisation server’s, so it does not appear in the Keycloak table.

If we take as our conformance criterion that Keycloak “supports” a version when it meets all its MUST and SHOULD requirements, the result is this:

MCP version	Conformance
`2025-03-26`	Supported
`2025-06-18`	Partial, without Resource Indicators (RFC 8707)
`2025-11-25`	Partial, without Resource Indicators (RFC 8707)

The only piece Keycloak lacks today is Resource Indicators for OAuth 2.0 (RFC 8707), the resource parameter that ties a token to a specific server. The Keycloak community already has support for it on the roadmap; in the meantime there is a perfectly valid workaround we look at below. Following the analogy, it is that subject the child still struggles with but can pass with a study trick until they fully mature.

Down to work: setting up Keycloak for your MCP version

For MCP 2025-03-26

Nothing special has to be configured. This version does not require the resource parameter, so Keycloak complies out of the box. The child is still small and the gate on the stairs is enough.

For MCP 2025-06-18 and 2025-11-25: tying the token to its audience

Here is where it gets interesting. For security, these versions require an access token to be bound to the audience it was issued for. In plain terms:

The MCP client must include the resource parameter (from RFC 8707) in the authorisation and token requests, with the value that identifies the MCP server it is going to use.
The MCP server must validate that the tokens it receives were issued specifically for it.

This prevents a stolen or reused token from getting into a server other than the intended one. The problem, as we said, is that Keycloak does not yet understand the resource parameter.

The solution until native RFC 8707 support arrives is to use OAuth’s scope parameter to achieve the same effect. Picture this situation:

The MCP server is at https://example.com/mcp.
It supports three scopes: mcp:tools, mcp:prompts and mcp:resources.
The client asks for a token with resource = https://example.com/mcp and a combination of those scopes.
We want Keycloak to issue a token whose aud (audience) is precisely https://example.com/mcp.

To achieve it, we configure Keycloak like this, repeating the pattern for each scope:

Create a client scope mcp:tools of type Optional and add an Audience mapper whose Included Custom Audience is https://example.com/mcp.
Create a client scope mcp:prompts of type Optional with its Audience mapper pointing at https://example.com/mcp.
Create a client scope mcp:resources of type Optional with its Audience mapper pointing at https://example.com/mcp.

The key is that the Included Custom Audience of each client scope is identical to the value of the request’s resource parameter and to the MCP server URL. With this, if the client asks for a token with those three scopes, Keycloak issues something like:

{
 "aud": "https://example.com/mcp",
 "scope": "mcp:resources mcp:tools mcp:prompts"
}

The MCP server can now check the aud and reject any token that was not meant for it. We have tied the credential to its recipient without waiting for RFC 8707.

If you use MCP Inspector

MCP Inspector is the official tool for debugging MCP servers, and it registers clients dynamically against Keycloak by running JavaScript from its backend. For it to work you have to adjust CORS in the client registration anonymous access policies:

Allowed Client Scopes: include the scopes your MCP server supports.
Allowed Registration Web Origins: include the web origin of the MCP Inspector backend.
Trusted Hosts: include the host or IP of the machine sending the dynamic registration request, that is, where your browser runs.

For MCP 2025-11-25: client registration with Client ID Metadata Document

The latest version adds a new chapter: how clients register. It contemplates three approaches, and you choose according to your scenario:

Client ID Metadata Documents: when client and server do not know each other beforehand (the most common case).
Pre-registration: when a prior relationship already exists.
Dynamic Client Registration: for backward compatibility or specific requirements.

Keycloak supports the first, OAuth Client ID Metadata Document (CIMD), although it is worth knowing that it is an experimental feature: it may bring breaking changes in future versions. To enable it, start Keycloak with the flag:

bin/kc.sh start --features=cimd

The idea behind CIMD is that the client_id stops being an opaque identifier and becomes a URL pointing at a document with the client’s metadata. Keycloak downloads that document and processes the request with what it finds there. For it to do so, you have to create a client policy profile and a policy that trigger it.

Profile. In Realm Settings → Client Policies → Profiles, create a profile (e.g. cimd-profile), add the client-id-metadata-document executor and configure it:

Allow http scheme: allows http for the URLs (client_id, client_uri, logo_uri, jwks_uri, etc.). Development only; in production OFF.
Trusted domains: wildcard patterns of accepted domains (e.g. *.example.org). If empty, all are denied.
Restrict same domain: if ON, it requires the client_id URL, the redirect_uri and the metadata URLs to all be under the same trusted domain.
Required properties: properties the metadata document must include.
Only Allow Confidential Client: if ON, only confidential clients are accepted (with jwks/jwks_uri and private_key_jwt or tls_client_auth authentication).

Policy. In Realm Settings → Client Policies → Policies, create a policy (e.g. cimd-policy), add the client-id-uri condition and configure it:

URI scheme: schemes to recognise in the client_id (in production, https only).
Trusted domains: domains accepted in the client_id host. If filled in, the condition is only true when the host matches; if left empty, it is always false.

Then associate the cimd-profile with this policy. From then on, when a request arrives with a client_id that is an https URL from a trusted domain, Keycloak downloads the metadata document and uses it to process the request.

Global executor settings. Some parameters are not in the console and are passed as SPI options at startup:

min-cache-time: minimum document cache time (300 s by default).
max-cache-time: maximum cache time (259200 s by default, 3 days).
upper-limit-metadata-bytes: maximum document size (5000 bytes by default).

bin/kc.sh start \
 --spi-client-policy-executor--client-id-metadata-document--min-cache-time=600 \
 --spi-client-policy-executor--client-id-metadata-document--max-cache-time=86400 \
 --spi-client-policy-executor--client-id-metadata-document--upper-limit-metadata-bytes=10000

A real case: VS Code desktop as an MCP client

Visual Studio Code is an MCP client that uses CIMD. When it connects to an MCP server that requires authorisation, it sends a client_id that is an https URL hosted on vscode.dev (e.g. https://vscode.dev/mcp-client), and Keycloak downloads its metadata from there. The detail to bear in mind is that VS Code is a public client using PKCE (no client secret) and, for the OAuth callback, it starts a local server with a redirect_uri of the form http://127.0.0.1:<port>/callback. Since that redirect is not in the vscode.dev domain, you have to leave Restrict same domain OFF.

Start with --features=cimd and configure:

Profile vscode-cimd-profile (client-id-metadata-document executor):

Allow http scheme: OFF
Trusted domains: vscode.dev, 127.0.0.1
Restrict same domain: OFF (the redirect is a localhost, not vscode.dev)
Only Allow Confidential Client: OFF (VS Code is public)

Policy vscode-cimd-policy (client-id-uri condition):

URI scheme: https
Trusted domains: vscode.dev
Associate the vscode-cimd-profile profile.

With this, when VS Code launches the request, Keycloak recognises the client_id as a vscode.dev URL, downloads the Client ID Metadata Document and completes the OAuth flow with the callback on localhost.

The education continues

We have taught our MCP to identify itself and to ask permission in a standard way, reusing the identity Keycloak already governs in the rest of the house. That is no small thing: with a token properly tied to its audience, everything else starts to become possible.

But, as with any child who grows up, this does not end here. The other two subjects we mentioned at the start are still pending, separating the tenants (multitenancy) and controlling what it spends (costs), and Keycloak still has some maturing to do in places, such as native Resource Indicators support. We will cover them in future posts. For now, our MCP already knows how to say who it is. And in security, that is the first day of school.

Sources

MCP from the inside and its deep observability: the LSP of AI agents and how to see everything with OpenTelemetry

Wed, 20 May 2026 06:00:00 +0200

TL;DR

Model Context Protocol (MCP) is the standard Anthropic published at the end of 2024 and which by 2026 has become the dominant protocol for connecting AI agents to external tools and data. Its value, the reason the whole industry adopted it in under 18 months, is that it solves a combinatorial problem: before MCP, integrating M AI apps with N tools required M×N ad-hoc integrations; with MCP, M + N. It is the same move the Language Server Protocol made in 2016 for code editors. The architecture is three well-defined roles, Host (the AI app), Client (the connection, one per server) and Server (the piece that exposes capabilities); the primitives are six, three on the server side (Tools, Resources, Prompts) and three on the client side (Sampling, Roots, Elicitation); the protocol is JSON-RPC over two transports, stdio for local processes and Streamable HTTP for remote. The operational challenge appears when there are 10-20 MCP servers running simultaneously, each with several tools, connected to an agent that chains multistep calls: observing what happens, where things fail, how much each tool costs, which tenant invokes what becomes critical. The ecosystem’s answer in 2026: the new OpenTelemetry GenAI semantic conventions for MCP (already stable), trace context propagation via params._meta (because JSON-RPC does not bring it natively), FastMCP with built-in OTel instrumentation, MCP Gateways as a centralised layer (Traefik Hub, MintMCP, OpenObserve), and MCP Inspector for interactive debugging. This article walks the architecture from the outside in, puts each concept in its exact place, and goes down to the detail of observability: traces, RED metrics, real use cases and pitfalls.

This is the third post in the post-tracing series. Previous posts: Evals and Guardrails. Here we go down to the protocol that connects agents to tools, and how to see it in production.

The master analogy (in three versions)

MCP is a communication protocol. Like any protocol, it is best understood with the right analogy. I am going to give you three, because each one lights up a different facet and the combination leaves you understanding it better than any technical definition.

Version 1 — The USB-C of AI apps (the official one)

This is the analogy Anthropic adopted when presenting it. Before USB-C, every electronic device had its own connector. Your phone had microUSB or Lightning, your laptop a proprietary power port, your headphones a 3.5mm jack, your external disk USB-A at one end and mini-USB at the other. The result: three boxes full of specific cables that got lost, none of them good for two things, and buying a new device meant buying new accessories.

USB-C changed that. A single physical connector that many protocols cross: data (USB 3, USB 4, Thunderbolt), video (DisplayPort), power (Power Delivery), audio. You connect anything to anything and it works; the protocols negotiate above.

MCP plays the same role for AI apps. Before MCP, every application that wanted to integrate tools with an LLM, Claude Desktop, Cursor, Continue, in-house custom agents, invented its own way of doing it. Every tool vendor had to write N different integrations, one per app. The result: massive fragmentation, a lot of duplicated code, integrations that broke when an app changed its internal API.

With MCP, the connector is one: any app that speaks MCP can use any MCP tool. Just as your USB-C talks to printers, monitors and disks without the printer “knowing” whether the cable is connected to a Mac or to a Linux box.

Version 2 — The LSP of code editors (the most technically precise)

This is my favourite because the analogy is structurally identical, not just metaphorical.

Until 2016, if you wanted your code editor to support a new language, Rust, Go, TypeScript, somebody had to write a specific plugin for your particular editor. VSCode had its Rust plugin, IntelliJ a different one, Vim another, Emacs another. Every decent feature (go-to-definition, autocompletion, refactoring) was an implementation duplicated N times. M editors × N languages = M·N integrations.

In 2016 Microsoft proposed the Language Server Protocol (LSP): each language implements a single “language server” (a process that understands that language); each editor implements a single LSP client; when you work with Rust code in VSCode, VSCode launches rust-analyzer as a subprocess and talks LSP to it over stdio. Any LSP editor + any LSP server = it works. M + N.

MCP is literally this pattern, transferred from “editor + language server” to “AI app + tool provider”. And it shares even the technical detail: both pass JSON-RPC over stdio (among other transports). When Anthropic designed MCP, they looked at LSP. Anyone coming from the world of editors and IDEs will find MCP familiar.

Version 3 — The operating system driver (the operational one)

Finally, an analogy that helps you understand what a specific MCP server does.

An operating system does not know directly how to talk to your particular HP LaserJet printer. What it knows is a generic interface: “print document”, “query status”, “cancel job”. The printer driver is the piece that translates that generic interface into the proprietary commands of your specific printer.

An MCP server does exactly the same:

Your AI agent knows a generic interface: invoke a tool with a defined schema, read a resource by URI, ask for a prompt template by name.
The MCP server is the driver: it translates those generic operations into the concrete APIs of the underlying system, your PostgreSQL database, your filesystem, your GitHub API, your Stripe.

This leaves the AI agent free from knowing how it authenticates with GitHub, what exact SQL PostgreSQL uses, what endpoints Stripe has. It speaks MCP; the server takes care of the details.

With the three analogies combined: MCP is the layer between the LLM and the world, a standard USB-C implemented like LSP in JSON-RPC, with each server acting as a driver for one concrete underlying system.

What concrete problem MCP solves

Before going down to the architecture, it is worth pinning down the specific problem MCP solves, because without that many design decisions look arbitrary.

The problem is the quadratic cost of integrations.

Imagine you have M applications that use LLMs (Claude Desktop, Cursor, Continue, ChatGPT Desktop, your own custom agent, and so on) and N external tools those LLMs could use (filesystem, GitHub, Slack, PostgreSQL, Jira, Notion, and so on). Without a standard:

Each (application, tool) pair requires a specific integration.
Every time the application changes its internal API, N integrations have to be updated.
Every time the tool changes its API, M have to be updated.
For your new tool to be adopted, you have to write M integrations.
For your new application to support the ecosystem, you have to write N.

The real result in 2023-2024: massive fragmentation. OpenAI’s function calling was not compatible with Anthropic’s tool use; every framework (LangChain, LlamaIndex, dspy) had its own wrapper; Claude Desktop plugins did not work in Cursor; and so on.

MCP breaks the quadratic. Each application implements the protocol once; each tool implements the protocol once; any pair works. M + N.

It is exactly what happened with USB-C, with LSP, with SQL (before there were proprietary APIs per database), with POSIX (before there were proprietary APIs per operating system). The pattern repeats because it always solves the same kind of problem.

The architecture: three roles, clearly placed

We are going to pin down where each thing lives, because mixing up the roles is the number one source of confusion in MCP.

Three roles. Let us pin down what each one does and where it physically lives.

Host: the AI application

The Host is the application the user opens. Claude Desktop, Cursor, Continue, ChatGPT Desktop, a custom agent your team builds, a VSCode extension. What the user perceives as “the product”.

The Host is responsible for:

Deciding which MCP servers to connect (configured by the user in a file or via the UI).
Launching or connecting to each MCP server.
Creating one MCP Client per server (it is 1:1, they are not shared).
Embedding the LLM (or calling it via API) that takes the decisions about which tools to use.
Mediating the user’s authorisation for sensitive actions (showing the human “the agent wants to run tool X, do you allow it?”).

Important: the LLM lives inside the Host, not in the servers. The servers are dumb; they execute operations when asked. The reasoning (“should I call this tool now?”) lives in the host’s LLM.

Client: the connection, one per server

An MCP Client is a specific connection between the Host and a Server. If your Host has 5 MCP servers configured, it has 5 clients, not one shared. Each client:

Maintains its socket or stdio pipe with the server.
Negotiates capabilities in the initial handshake (which protocol version, which primitives both support).
Serialises JSON-RPC requests to the server and deserialises responses.
Is the point where the Host invokes operations on the server.

The 1:1 client-server separation matters because it lets each server have its own session state, its specific permissions and its independent authenticated context. There is no multiplexing in the client.

Server: the piece that exposes capabilities

The MCP Server is the piece that implements the tool-provider side of the protocol. It receives JSON-RPC from the client, processes it, executes the action against the underlying system and returns a response.

There are two flavours physically:

Local server: it starts as a subprocess of the Host and communicates over stdio. Its lifecycle is the Host’s (when you close Claude Desktop, the local servers die). Typical model: your Host launches node filesystem-mcp-server.js as a child.
Remote server: it runs as an independent service, reachable over HTTP. Multi-tenant, authenticated, scalable. Typical model: a company publishes https://mcp.acme.com/v1 and many hosts connect to it.

This difference has enormous consequences for observability (we will come back to it shortly).

Summary of where each thing lives

Component	Lives in	How many there are	Speaks what with whom
Host	User’s machine	1 (the open app)	UI with the user; launches clients
LLM	Embedded in the Host (or cloud API)	1 (the main one)	Reasons; asks for tools
Client	Host	1 per server	JSON-RPC with its server
Local server	Subprocess of the Host	1 per local integration	stdio with its client
Remote server	External service	1 per service	HTTP/SSE with its clients
Underlying system	External	Depends	API/DB/FS, not MCP

If you get confused in a discussion, come back to this table. The number one source of errors in MCP is saying “the server” when you mean “the host”.

The two layers of the protocol

MCP separates the data layer and the transport layer. This separation is what lets the protocol work over local stdio and over remote HTTP without changing anything in the primitives.

Data Layer: JSON-RPC with MCP extensions

The data layer defines the vocabulary of the messages. It is JSON-RPC 2.0. Each message is a JSON with jsonrpc: "2.0", a method (eg tools/call, resources/read), params, and an id to correlate request with response.

On top of JSON-RPC, MCP adds:

Lifecycle: the initial handshake (initialize, initialized) that negotiates capabilities.
The primitives (next section): tools/*, resources/*, prompts/*, sampling/* and so on.
Notifications: messages without a response (eg notifications/cancelled to abort a tool in progress).
Meta-information: the params._meta field carries cross-cutting metadata by convention (trace context, request IDs).

Transport Layer: how the messages move

The transport layer defines how the JSON-RPC messages travel. Two official transports:

stdio: the client launches the server as a subprocess and they communicate over its stdin/stdout/stderr with JSON-RPC. One message per line, separated by a newline. No network, no TLS handshake, no auth (trust is inherited from the operating system itself: if you launch the subprocess, you trust it). Minimal latency (~100 μs round-trip), maximum bandwidth (memcpy, not a socket).

Use case: local servers that live on the same machine as the host. Most of the MCP servers you see in public directories are stdio.

Streamable HTTP: the client sends a POST to an HTTP endpoint on the server; the server responds with JSON, optionally opening a Server-Sent Events stream to send asynchronous notifications or long responses. Auth by bearer token, API key or custom headers.

Introduced in the November 2025 spec, it replaces the pure SSE transport of earlier versions, which had bidirectionality limitations. Use case: remote servers serving many simultaneous clients, with authentication and multi-tenancy.

Important: the primitives are the same on both transports. A tools/call is identical over stdio and over HTTP. The transport is accidental, not fundamental.

The six primitives: placed in the architecture

Here is the meat. There are six primitives in MCP. They are often confused because several look like they do similar things. The key classification: three live on the server side (the server exposes, the client consumes) and three on the client side (the client exposes, the server consumes).

Server-side: what the server gives the host

Tools are actions the server exposes. Each tool has a schema (typed parameters, description) and an implementation. When the host’s LLM decides to invoke a tool, the client sends tools/call to the server, which executes it and returns a result.

Example: the github-mcp server exposes create_issue(repo, title, body). The host’s LLM decides “I am going to create an issue”, calls this tool, github-mcp talks to the GitHub API, returns the issue ID to the LLM.
Architectural place: the server exposes them, the LLM consumes them.

Resources are contextual data the server exposes, addressable by URI. They are not actions; they are content reads. A resource has a URI (file:///path/to/doc.md, postgres://table/users), metadata and an endpoint to read content.

Example: the filesystem-mcp server exposes the files in the authorised directories as resources. The LLM asks for resources/read with the URI file:///docs/api.md and gets the text.
Architectural place: the server exposes them, the host reads them (and optionally passes them to the LLM as context).

Key difference between Tools and Resources: Tools are verbs (they execute, they modify state, they have side effects); Resources are nouns (they exist, they are read, they are idempotent). If you have something that is “search for text in files” → probably a Tool (an action). If it is “this specific file” → a Resource. The distinction matters for auditing and permissions: tools require more control.

Prompts are parameterised prompt templates the server exposes. The user or the host can invoke them to inject a conversational pattern into the model.

Example: a code-review-mcp server exposes a prompt review_diff(diff_text, style="strict") that returns a complete, well-written prompt asking the LLM to review code.
Architectural place: the server exposes them, the user or the host invokes them, the LLM receives them as input.

Prompts are the least used of the three primitives; many servers do not even implement them. But they let a team publish good prompts as a reusable library, separate from the agent.

Client-side: what the host gives the server

This is where MCP differs from protocols like HTTP REST: the server can also ask things of the host, it is not a one-way street. Three primitives travel in that direction.

Sampling: the server asks the host to run a generation with its LLM. That is, the server borrows the host’s LLM in order to reason.

Example: the search-mcp server receives a query from the agent, searches its corpus, finds 50 results and needs to summarise them before returning. Instead of having its own LLM, it sends a sampling/createMessage to the client; the host passes this to its LLM, runs the generation with the user’s permissions, and returns the summary to the server.
Architectural place: the server asks for it, the host (with its LLM and the user’s authorisation) fulfils it.
Why it matters: the user controls which model is used, what cost is paid, what permissions apply. The server does not need its own OpenAI API key.

Roots: the host tells the server where to look. Roots are URIs (directories, repositories, namespaces) the host authorises the server to explore.

Example: your Claude Desktop starts filesystem-mcp with roots [file:///Users/me/projects]. The server knows it must only operate inside that folder, not in /etc/passwd.
Architectural place: the host declares them in the handshake, the server respects them.

Elicitation: the server asks the host for additional information from the human user via a structured UI.

Example: the stripe-mcp server is about to process a refund of €5000. Before executing, it sends an elicitation/createMessage to the client; the host shows the user “Confirm this €5000 refund” with a button; when the user confirms, it returns OK to the server, which then proceeds.
Architectural place: the server asks, the host shows the user, the user decides, the answer goes back to the server.
It is the key primitive for human-in-the-loop on sensitive actions.

Visualising the flow of the six primitives

 HOST SERVER
│ │
Server-side ─────┼─────────────────────────────────────┤
│ │
tools/list ──────┼─────── asks what tools exist ──────▶│
│◀────── returns the list ────────────│
│ │
tools/call ──────┼────────── run this tool ───────────▶│
│◀────── result ──────────────────────│
│ │
resources/read ──┼────────── read this URI ───────────▶│
│◀────── content ─────────────────────│
│ │
prompts/get ─────┼─────── give me this prompt ────────▶│
│◀────── compiled prompt ─────────────│
│ │
Client-side ─────┼─────────────────────────────────────┤
│ │
sampling ────────│◀────── I need a generation ─────────│
│── use my LLM ───┐ │
│── returns ──────▼──────────────────▶│
│ │
roots ───────────┼───── declared in the handshake ────▶│
│ │
elicitation ─────│◀────── ask the user X ──────────────│
│── shows UI ────┐ │
│── confirms ────▼───────────────────▶│

JSON-RPC in action: a concrete example

To make the theory concrete, here is a real MCP conversation between a client and the filesystem-mcp server:

// 1. Initial handshake (client → server)
{
"jsonrpc": "2.0", "id": 1, "method": "initialize",
"params": {
"protocolVersion": "2026-03-01",
"capabilities": {
"sampling": {}, // this client supports sampling
"roots": { "listChanged": true }
},
"clientInfo": { "name": "ClaudeDesktop", "version": "1.2.0" }
}
}
// 2. The server responds with its capabilities
{
"jsonrpc": "2.0", "id": 1, "result": {
"protocolVersion": "2026-03-01",
"capabilities": {
"tools": { "listChanged": true },
"resources": { "subscribe": true, "listChanged": true },
"prompts": {}
},
"serverInfo": { "name": "filesystem-mcp", "version": "0.5.2" }
}
}
// 3. The client asks for the list of tools
{
"jsonrpc": "2.0", "id": 2, "method": "tools/list"
}
// 4. The server returns its tools with a schema
{
"jsonrpc": "2.0", "id": 2, "result": {
"tools": [
{
"name": "read_file",
"description": "Read a file from the filesystem",
"inputSchema": {
"type": "object",
"properties": { "path": { "type": "string" } },
"required": ["path"]
}
},
{ "name": "write_file", "description": "...", "inputSchema": {} },
{ "name": "list_directory", "description": "...", "inputSchema": {} }
]
}
}
// 5. The LLM decides to call read_file; the client sends tools/call
{
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
"params": {
"name": "read_file",
"arguments": { "path": "/Users/me/projects/notes.md" },
"_meta": { // ← the extension where trace context will go
"traceparent": "00-abc123...-def456-01"
}
}
}
// 6. The server returns the file content
{
"jsonrpc": "2.0", "id": 3, "result": {
"content": [
{ "type": "text", "text": "# My notes\n\n..." }
]
}
}

The important thing to note: params._meta. That is the bag where MCP conventionally passes cross-cutting metadata, including trace context. We will come back to it shortly.

The observability problem: why traditional tracing is not enough

So much for theory. Let us go down to the operational problem: in a 2026 production cluster, a typical agent has 5-15 MCP servers connected simultaneously, each with 5-20 tools, and each conversation with the agent can generate dozens of chained tool calls. Without observability, debugging incidents is impossible.

Why generic tracing (Hubble, OTel without MCP conventions) is not enough:

Stdio is not visible on the network. Local servers talk over OS pipes. Your Hubble or your Datadog APM see nothing; there are no packets to capture. AgentSight (seen in the previous post in the eBPF series) with stdiocap captures it but gives you the raw JSON-RPC, without semantic context (which tool it is, which resource, which prompt).

Generic HTTP does not understand MCP either. If you trace the HTTP to a remote MCP server without MCP conventions, you see a POST to /v1 with an opaque JSON-RPC body. You lose “which tool was invoked”, “what arguments”, “was it elicitation or sampling”. RED metrics per endpoint are no use to you; you need RED per tool.

JSON-RPC does not propagate trace context natively. Unlike HTTP (the W3C traceparent header) or gRPC (metadata), JSON-RPC has no standard field for trace context. If you do not propagate it, every call to the server starts a new trace disconnected from the agent’s trace.

Multistep multi-server is very hard to follow. A single user conversation can translate into: 1) a call to github-mcp get_pr; 2) a call to filesystem-mcp read_file for several files; 3) a call to the main LLM with all the context; 4) a call to postgres-mcp query; 5) a call to slack-mcp send_message. Without propagated trace context, those are five disconnected traces. With propagation, it is one tree.

The solution: OpenTelemetry semantic conventions for MCP, already stable in 2026.

OpenTelemetry semantic conventions for MCP

The GenAI MCP semantic conventions are the set of standardised attributes for MCP-related spans and metrics. They were published as part of the GenAI subgroup of the OpenTelemetry SIG and are the first part of the GenAI semantic conventions to reach stable.

Why MCP-specific semantic conventions

Before they existed, teams instrumented MCP with the generic RPC semantic conventions (the ones you would use for gRPC or XML-RPC). It half worked. The MCP-specific conventions add:

Attributes to identify which primitive was executed (mcp.method.name = "tools/call").
Attributes to identify which specific tool/resource/prompt was touched (mcp.tool.name, mcp.resource.uri, mcp.prompt.name).
Attributes for the bidirectional flow (sampling/elicitation requests from the server to the client).
Attributes for the handshake (mcp.protocol.version, mcp.client.name, mcp.server.name).
Standardised RED metrics per tool (mcp.tool.call.duration, mcp.tool.call.errors).

The canonical attributes

The attributes any MCP-aware instrumentation should emit:

Attribute	Meaning	Example
`mcp.method.name`	JSON-RPC method	`"tools/call"`
`mcp.tool.name`	Name of the tool	`"read_file"`
`mcp.resource.uri`	URI of the resource	`"file:///docs/api.md"`
`mcp.prompt.name`	Name of the prompt	`"code_review"`
`mcp.session.id`	MCP session ID	`"sess-abc123"`
`mcp.protocol.version`	Protocol version	`"2026-03-01"`
`mcp.client.name`	Client identity	`"ClaudeDesktop/1.2.0"`
`mcp.server.name`	Server identity	`"filesystem-mcp/0.5.2"`
`mcp.transport`	Transport used	`"stdio"` or `"http"`
`mcp.error.code`	JSON-RPC error code	`-32602` (Invalid params)
`gen_ai.usage.input_tokens`	Tokens consumed (if sampling)	`1240`
`gen_ai.usage.output_tokens`	Tokens generated (if sampling)	`512`

The last two come from the generic GenAI semantic conventions and apply when the MCP call involves sampling (the server using the client’s LLM).

RED metrics per tool

Beyond spans, the semantic conventions define three core metrics:

mcp.tool.call.duration (histogram): the latency of each invocation.
mcp.tool.call.count (counter): the total number of invocations.
mcp.tool.call.errors (counter): errors per tool.

Labelled with mcp.tool.name, mcp.server.name, mcp.client.name. Pivotable in Grafana to answer “which tool is the slowest”, “which tool fails most”, “which client loads which server hardest”.

Trace context propagation: the `params._meta` trick

JSON-RPC has no headers like HTTP, so MCP cannot use the W3C traceparent header directly. The solution the ecosystem has converged on: propagate trace context in params._meta.

When the MCP client sends a tools/call, its OTel instrumentation does:

import json
from opentelemetry.propagate import inject

carrier = {}
inject(carrier) # fills it with traceparent/tracestate from the active span

params = {
 "name": "read_file",
 "arguments": {"path": "/notes.md"},
 "_meta": carrier, # ← propagates trace context
}

When the server receives it, it does the symmetrical thing:

from opentelemetry.propagate import extract

ctx = extract(request.params.get("_meta", {}))
with tracer.start_as_current_span("tools/call", context=ctx):
 # this span is a child of the client's
 return execute_tool(request.params)

The result: the server’s span is a child of the client’s span in the trace tree. When you look at the trace in Tempo or Phoenix, you see the whole chain: user → host → client → server → execution → response → client → host → response to the user.

This requires both ends to instrument consistently. If the server does not extract the context, you see disconnected spans but at least you have traceability on the client side.

Instrumentation patterns

There are three routes to instrumenting MCP, in increasing order of effort:

1. FastMCP with built-in OpenTelemetry

FastMCP is one of the most used Python frameworks for building MCP servers. It ships built-in OpenTelemetry instrumentation: every tool, resource template and prompt operation generates spans automatically with the correct MCP conventions.

from fastmcp import FastMCP
from opentelemetry.sdk.trace.export import OTLPSpanExporter

mcp = FastMCP("my-server", otel_endpoint="https://otel-collector:4318")

@mcp.tool()
def search_docs(query: str) -> str:
 """Search the corpus for matching documents."""
 # this automatically generates a span with
 # mcp.tool.name=search_docs, mcp.method.name=tools/call, etc.
 return run_search(query)

Zero instrumentation code. Spans with the correct conventions. It is the recommended pattern if you are starting an MCP server in Python from scratch.

2. Manual OpenTelemetry SDK

For servers that already exist or are in other languages (TypeScript, Go), the option is to instrument manually with the standard OTel SDK and emit the conventional MCP attributes:

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

async def handle_tools_call(req: JSONRPCRequest):
 ctx = extract_trace_context(req)
 with tracer.start_as_current_span("mcp.tools.call", context=ctx) as span:
 span.set_attribute("mcp.method.name", "tools/call")
 span.set_attribute("mcp.tool.name", req.params["name"])
 span.set_attribute("mcp.server.name", "filesystem-mcp")
 try:
 result = await execute_tool(req.params)
 return result
 except Exception as e:
 span.set_attribute("mcp.error.code", -32603)
 span.record_exception(e)
 raise

More boilerplate but it works with any existing server.

3. MCP Inspector for interactive debugging

MCP Inspector (official) is a tool for interactive protocol-level debugging. It launches a local proxy (port 6277) between your client and the server, and opens a web UI (port 6274) where you see every JSON-RPC message back and forth in real time.

It is not production observability, it is development and debugging. But it is irreplaceable during the bring-up of a new server: you see exactly which requests arrive, which responses are returned, which errors occur. It saves hours of ad-hoc logging.

MCP Gateways: the centralised piece for enterprise

When your organisation has many agents connecting to many MCP servers, managing the connection matrix becomes operationally serious. The natural question, “can there be a proxy in front of all the MCP servers that centralises auth, rate limiting, logging and observability?”, already has an answer: MCP Gateways.

An MCP Gateway is a proxy that:

Accepts MCP connections from the hosts/agents.
Routes them to the corresponding backend MCP servers.
Applies centralised authentication and authorisation (which agent can call which tool).
Applies rate limiting per agent, per tool, per tenant.
Observes: it emits OTel metrics for every operation passing through.
Propagates the identity of the agent to the backend server (with several models: token forwarding, token exchange, impersonation).

The options that have established themselves in 2026:

Traefik Hub MCP Gateway — from the Traefik team. Declarative configuration, native integration with Traefik’s Kubernetes/Helm ecosystem.
MintMCP — a gateway focused on observability and multi-tenancy. SaaS and self-host.
OpenObserve MCP Gateway — integrated with the OpenObserve observability platform.

For small deployments (one team, few agents) a Gateway can be overkill. For enterprise (dozens of agents, dozens of servers, regulated compliance), it is practically mandatory.

Real use cases for MCP observability

Let us land this with five cases where properly instrumented MCP observability delivers immediate value:

1. Audit per tool, per tenant, per agent

Question: “who ran the delete_repo tool last month?”. Without MCP observability, impossible. With OTel conventions + identity propagation: a query in your trace backend filtering by mcp.tool.name="delete_repo", grouping by mcp.client.name or by the user_id propagated in _meta. Compliance happy.

2. Cost per tool and per tenant

Question: “how much does each tool cost?”. If the tools invoke external APIs (Stripe, OpenAI sampling) or consume significant resources (a GPU for an inference tool), knowing their aggregated cost matters. With mcp.tool.call.duration + gen_ai.usage.* aggregated by tool and tenant, you build cost accountability dashboards without instrumenting anything extra.

3. Debugging multistep chains that fail

Question: “the agent failed to complete this task, where did it go wrong?”. The propagated trace connects: the user’s span → the LLM’s span with its CoT → the spans of each tool invoked → the final LLM span. If the chain broke at the third tool, in Tempo you see the red span with the specific error message. Reproducing the failure is trivial.

4. Tool latency and degradation

Question: “which tool is degrading?”. RED metrics per tool in Grafana show p95/p99 latency over time. When a tool starts climbing from 200ms to 800ms (because the underlying service is collapsing), you see it before the users complain.

5. Detecting loops and agentic anomalies

Question: “is any agent stuck in a loop?”. If an agent calls tools/call read_file 80 times in 30 seconds for the same path, something is clearly wrong. An alert on mcp.tool.call.count grouped by (session_id, tool_name) detects this. Combined with loop detection at the reasoning level, it closes the circle.

Operational pitfalls

Lack of identity propagation

Your Gateway authenticates the agent, but passes requests to the backend without propagating identity. The result: the backend’s logs say “service-account” for everything, and it is impossible to audit who invoked what. Choose a propagation strategy early: token forwarding (simple, exposes tokens to the backend), token exchange (more secure), or impersonation with cross-logging.

Stdio servers that do not show up in your APM

This is the field’s number 1 pitfall. Your Cursor agent uses filesystem-mcp over stdio; you see nothing in Datadog because there is no network traffic. Solution: instrument the stdio server with an OTel SDK that exports over OTLP to your collector (via gRPC or HTTP; the OTel collector can receive even though the server talks stdio to its client). Or use AgentSight stdiocap to capture the raw JSON-RPC and process it offline.

Multiple protocol versions in production

Different clients use different MCP versions simultaneously. Your metrics dashboard mixes apples and oranges. ALWAYS label with mcp.protocol.version and filter/group by it.

`_meta` lost when passing through a proxy

Your Gateway accepts the client’s request, rewrites it for the backend, and forgets to copy params._meta. The result: the trace is broken at the Gateway, two disconnected traces. Make sure your Gateway preserves or re-injects trace context at each hop.

Trace volume with chatty servers

Some MCP servers emit many small operations (filesystem listings, partial reads). Without sampling, they fill your backend with useless traces. Apply tail-based sampling that keeps complete sessions or keeps only traces with errors or high latency.

Cardinality in metrics

mcp.tool.call.duration with mcp.session.id as a label blows up cardinality. Do not include unique per-session IDs in labels; keep cardinality under control with labels that take few discrete values (tool name, server name, client name, error code).

Confusing client and server spans

When you look at the tree, distinguish: the client sees the total latency from its perspective (network included); the server sees only its own work. If you look only at the server span to debug the latency perceived by the user, you miss the RTT. Use both.

What we have not covered

Experimental WebSocket MCP transport: an alternative to Streamable HTTP, not yet standard.
MCP servers in cloud-native deployments with sidecars: an emerging pattern of deploying MCP servers as pod sidecars.
MCP federation: composing several servers as one (similar to GraphQL federation).
eBPF + MCP: how AgentSight’s stdiocap and Cilium’s hooks complement native instrumentation.
MCP testing and contract tests: how to validate that your server complies with the spec.

References

Specification and concepts:

Model Context Protocol — sitio oficial — entrada canónica.
MCP architecture overview.
Transports — MCP docs.
MCP Inspector (GitHub) — debugging interactivo.

OpenTelemetry GenAI MCP:

Semantic conventions for Model Context Protocol — OpenTelemetry — referencia normativa.
Adding OpenTelemetry Trace Support to MCP (Discussion #269) — historia de la propuesta.
How to Instrument MCP Servers with OpenTelemetry (OneUptime).
How to trace MCP server tool calls with OpenTelemetry and Elastic APM.
MCP Observability with OpenTelemetry (SigNoz).
Distributed tracing for agentic workflows (Red Hat Developer).
OpenTelemetry for AI Agents in MCP Workflows (MintMCP).

Frameworks and gateways:

FastMCP OpenTelemetry — instrumentación built-in.
Traefik Hub MCP Gateway — gateway de Traefik.
MintMCP — gateway con foco en observabilidad.
OpenObserve MCP Gateway guide.
What is an MCP Gateway (DEV Community).
OpenTelemetry MCP Server (Traceloop) — el patrón inverso: usar MCP para que agentes consulten traces OTel.

Cross-references:

Post anterior: Guardrails y safety.
AgentSight y el nuevo tracing de LLMs — donde se introdujo stdiocap para capturar stdio de servidores MCP locales.
Evals: la capa después del tracing.

Hubble: eBPF network observability, state of the art 2026 and the new frontier with AI agents

Tue, 19 May 2026 06:00:00 +0200

TL;DR

Hubble is Cilium’s native network observability, built on the same eBPF programs Cilium uses for enforcement. It does not duplicate the datapath or instrument the kernel its own way: it listens to the hooks Cilium already has and produces structured flow logs with Kubernetes context included, covering pod, namespace, labels, service, policy verdict and L7 payload where applicable. It is what happens when someone decides that tcpdump with grep does not scale to 10,000 pods and builds a distributed system of their own (a Hubble server per node, plus Hubble Relay as the aggregator, plus a CLI and a UI) with practically zero overhead, because the capture was already happening. In 2026 it is at version 1.19.3 (April 2026), with Cilium 1.19 marking the project’s tenth anniversary; IP options tracing has arrived, along with filtering by encryption status, the drop event tagged with the exact NetworkPolicy that caused it (direct attribution), a stabilised field mask API, and the first wave of ML anomaly detection applied to flows for predictive security in IoT/5G clusters. And, most interesting of all for 2026: a new frontier appears where the same eBPF observes AI agents such as Claude Code, Gemini CLI and MCP agents, intercepting SSL/TLS and stdio without instrumenting the code, which turns the Cilium + Hubble + Tetragon + AgentSight stack into a complete set of tools for understanding what an agentic system does inside a cluster.

This article is part 3 of the eBPF series. Part 1: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack. Part 2: Tetragon: Cilium’s security cousin that sees every syscall in the kernel. Here we complete the observability quadrant: network with Hubble, process with Tetragon, AI agent with AgentSight.

The analogy: tcpdump that speaks Kubernetes

If you have administered networks over the last twenty years, tcpdump and Wireshark have been your daily bread. They capture packets on an interface, parse them, and let you filter with tcp.port == 443 and host 10.0.0.5. They work, they have been working since the 90s, and they are the first thing you open when something smells wrong.

Now stick tcpdump on a Kubernetes cluster with 10,000 pods. The problems show up in order:

One tcpdump session per node. You wanted to “see the traffic between the frontend and the API”; you need SSH to every node, tcpdump on every NIC, synchronised timestamps, manual aggregation.
There is no K8s context. You see a packet from 10.244.5.7 to 10.244.8.42. Which pod was that? Which namespace? Which label? You have to correlate with kubectl get pod -A -o wide every single time.
No L7 understanding. You see a POST to HTTPS, and you cannot know the method and path because it is encrypted on the wire. If there is mTLS between pods, worse.
High cost: full packet capture with a copy to userspace slows down the datapath. Under dense traffic, you notice it.

Hubble is tcpdump redesigned for all of that. It reuses the eBPF programs that are already processing every packet (Cilium puts them there for enforcement) and, while they take their allow/deny decision, they emit a flow event with all the context: source and destination pod identity, namespace, labels, protocol, verdict, and, if Cilium has done L7 parsing via Envoy, the HTTP method, path, status code, DNS query, Kafka topic. That event travels through a ringbuffer to userspace, it is received by the Hubble server that lives inside the node’s Cilium agent, and it is exposed over gRPC. A separate service, Hubble Relay, aggregates the streams from all nodes and gives you a single cluster-wide API. On top of that: a CLI (hubble) and a web UI with a real-time service graph.

Zero additional copies. Zero duplicated parsing. And the result is understandable by anyone who knows what a Pod is.

Architecture: four pieces visible from outside

Hubble is made up of four logical components, all optional depending on what you want to do:

1. Hubble Server (embedded in every Cilium agent)

It lives inside the Cilium agent process (it is not a separate binary). Each node exposes a gRPC endpoint locally on the Unix socket /var/run/cilium/hubble.sock. The server listens to the events the eBPF programs emit to the ringbuffer, enriches them with Kubernetes metadata (which the agent already has in memory), and makes them available to consumers.

Enabling it: --set hubble.enabled=true in Cilium’s Helm chart. By default, the server is only accessible locally; if you want to consume it from another node, it has to be exposed (which is what Hubble Relay does).

2. Hubble Relay (aggregator)

It is a separate Deployment (typically 1 replica, scalable) that connects to every Hubble server in the cluster and aggregates their streams into a single API. When your CLI or UI asks for “the last 1000 flows in the cluster”, the Relay collects them in parallel from all nodes and returns the union.

Enabling it: --set hubble.relay.enabled=true. Without the Relay you only see the traffic of the node you are connected to, which is useful for local debugging but not for a cluster-wide view.

3. Hubble CLI (`hubble`)

A Go binary that speaks gRPC with the Relay (or with a local Hubble server). It supports two main modes:

hubble observe: a real-time stream of flows, with very expressive filters (by namespace, pod, port, verdict, protocol, label).
hubble status: the state of the Hubble cluster (how many nodes connected, lag, flow rate).

And the equivalent of tcpdump’s pcap dump: hubble observe --output jsonpb > flows.json to process afterwards with jq or other tools.

4. Hubble UI

A web frontend that connects to Hubble Relay and shows:

A service graph in real time (which Pod talks to which Service, which protocols it uses, which verdict).
A filterable list of flows.
L7 details where there are any (HTTP method/path/status, DNS query/response).

Enabling it: --set hubble.ui.enabled=true. Useful for presentations to non-CLI teams; it does not replace the CLI for serious debugging.

What you see: the Hubble flow log from the inside

A Hubble flow in JSON format looks roughly like this (simplified):

{
 "time": "2026-05-19T03:12:45.182Z",
 "verdict": "FORWARDED",
 "source": {
 "ID": 5482,
 "identity": 24871,
 "namespace": "prod-api",
 "labels": ["app=checkout", "team=payments"],
 "pod_name": "checkout-7c9f-x8j2",
 "workloads": [{"name": "checkout", "kind": "Deployment"}]
 },
 "destination": {
 "ID": 12041,
 "identity": 18356,
 "namespace": "prod-db",
 "labels": ["app=postgres", "tier=primary"],
 "pod_name": "postgres-0"
 },
 "Type": "L3_L4",
 "l4": {
 "TCP": {
 "source_port": 41982,
 "destination_port": 5432,
 "flags": {"SYN": true}
 }
 },
 "node_name": "rke2-worker-03",
 "Summary": "TCP Flags: SYN"
}

When L7 parsing is active (via embedded Envoy or Hubble’s lightweight parser), the same flow adds:

"l7": {
 "type": "REQUEST",
 "http": {
 "code": 200,
 "method": "GET",
 "url": "/api/v1/cart/items",
 "protocol": "HTTP/1.1",
 "headers": [{"key": "user-agent", "value": "checkout/1.4.2"}]
 }
}

The protocols natively supported for L7 parsing:

HTTP/1.1 and HTTP/2 (including gRPC over HTTP/2).
DNS (queries and responses, with domains, types, response codes).
Kafka (topics, API keys).
TLS handshake (SNI, not the encrypted payload by default).
MySQL, Cassandra (with optional modules).

For HTTP and gRPC, Cilium can enable the embedded Envoy proxy for the flows you want to inspect (not all of them; it is selective via CiliumNetworkPolicy with L7 rules). Without Envoy there is lightweight parsing, but less detailed.

Verdict and drop attribution

Every flow has a verdict: FORWARDED, DROPPED, ERROR, AUDIT, REDIRECTED, TRACED, TRANSLATED. For the DROPPED case, Hubble includes a structured reason (drop_reason) and, since Cilium 1.19, the exact NetworkPolicy that caused it.

That last point changes day-to-day operations. Before, when a pod could not talk to another one, the debug flow was:

See the dropped flow in Hubble.
Look at every CiliumNetworkPolicy in the namespace.
Reason manually about which one, with which labels, is blocking it.

With Cilium 1.19’s attribution, the policy_match_info field tells you directly “it was dropped by the frontend-egress policy, rule 3”. You go from “Sherlock Holmes for 20 minutes” to “kubectl get -o yaml of that specific policy”.

Prometheus metrics and Grafana dashboards

Hubble also exposes aggregated metrics in Prometheus format, separate from the gRPC flow stream. Enabling it: --set hubble.metrics.enabled=true (Helm) plus a list of the set you want to export.

The usual metric groups:

flow: total flows by verdict, source/dest, protocol.
http: requests by method, response code, latency (histogram).
dns: queries, response codes, top-N domains.
tcp: handshakes, retransmissions, congestion window.
drop: drops by reason, with NetworkPolicy attribution.
port-distribution: histogram of active ports.
policy: hits by policy and verdict.

These metrics have rich K8s labels (source_workload, destination_workload, namespace, and so on) that make them pivotable in Grafana. There are prebuilt dashboards on Grafana Labs covering the common cases; importing one and having immediate visibility costs five minutes.

Cost: metrics with many K8s labels can blow up cardinality in Prometheus. For large clusters (>1,000 pods), it is worth reviewing which set you export and using drop rules in Prometheus to keep a lid on it.

Deployment: Helm on one screen

The canonical Cilium install with full Hubble:

# values.yaml
hubble:
 enabled: true
 metrics:
 enabled:
 - dns:query;ignoreAAAA
 - drop
 - tcp
 - flow
 - port-distribution
 - icmp
 - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
 serviceMonitor:
 enabled: true # auto-discovered by kube-prometheus-stack
 relay:
 enabled: true
 rollOutPods: true
 ui:
 enabled: true
 rollOutPods: true
 ingress:
 enabled: true
 className: cilium
 hosts:
 - hubble.example.local

And the install itself:

helm upgrade --install cilium cilium/cilium -n kube-system -f values.yaml

After installing, validate with cilium status (the Cilium CLI) that the Hubble section shows OK, and try your first flow with:

cilium hubble observe --namespace prod-api --pod checkout-7c9f-x8j2

State of the art in 2026

Cilium 1.19 was released in February 2026, marking the project’s tenth anniversary. Hubble reached version 1.19.3 on 22 April 2026. The relevant news:

Direct attribution of drops to a NetworkPolicy

Already covered above; it is probably the most valuable operational change in the release. Any dropped flow carries the name, namespace and specific rule of the policy responsible. It is also available via Prometheus metrics, which allows alerts of the form “policy X is dropping >N requests/second”.

Tracing with IP options

Hubble can now trace individual packets with IP options enabled. It is a mechanism similar to traceroute, but at L3: you put a mark on the packet and Cilium reports it every time the packet crosses a node or an eBPF decision. Useful for debugging multi-cluster paths, fabric mesh, or NetworkPolicies applied at different layers.

Filtering by encryption status

A new CLI flag: hubble observe --encryption-status=encrypted (or unencrypted). Useful for validating deployments with WireGuard or IPsec enabled pod-to-pod: you confirm that the traffic that should be encrypted is, and you spot regressions quickly.

Hubble field mask API stabilised

The field_mask lets you ask for only the parts of the flow you care about, hugely reducing bandwidth and processing when all you need is, say, source/dest and verdict. It used to be experimental; now it is stable and it is default-on in the CLI.

AI-driven anomaly detection (predictive security)

This is the most talked-about addition of 2026. Cilium 1.19 adds hooks so that an external consumer, typically an ML system, can process flows in streaming and detect statistical anomalies: pods that suddenly talk to new destinations, latency spikes on an API, odd DNS sequences. The detection part happens outside the Cilium agent (nobody wants heavy ML in the datapath), but Cilium exposes the flows with the pre-computed features the model needs. The published use cases focus on IoT and 5G, where traffic is high in volume and low in variety, ideal conditions for anomaly detection.

Scaling to 10,000+ pods

Cilium 1.19 has done serious work on scalability: Hubble Relay can now aggregate streams from hundreds of nodes without saturating; the default field_mask reduces inter-node bandwidth; and flows can be sampled under high load if your use is statistical analysis rather than forensic debugging.

Cilium 1.20 in development

Cilium 1.20 is in the development branch. The most relevant bits for Hubble:

Unification of preferIpv6: the hubble.preferIpv6 flag was deprecated in favour of the global preferIpv6 that applies to every Cilium component.
tetragon-python SDK: although it belongs to Tetragon, not Hubble, it sets a trend, namely eBPF policies written in Python instead of YAML. Hubble will probably follow a similar path.

The new frontier: eBPF and AI agents

So far, the classic Hubble content. But there is a 2026 twist worth covering, because it closes the loop with the other series on this blog.

When a Kubernetes cluster starts running AI agents, such as Claude Code, Gemini CLI, or LangGraph-based agents calling APIs and MCP servers, the observability problem changes shape. It is no longer enough to know “which pod talked to which pod” (that is Hubble) or “which process ran what” (that is Tetragon). You need to know:

Which external APIs the agent is calling and with which prompts.
Which MCP tools it is invoking, with which arguments.
How many tokens it consumes, which model it picks, how much it costs.
Whether the agent drifts from expected behaviour (out-of-policy queries, jailbreak attempts, secret leakage).

The traditional solutions, instrumenting the agent’s code with OpenTelemetry or parsing structured logs, do not work well when the agent is a third-party binary (Anthropic’s Claude Code, Google’s Gemini CLI) or when the MCP servers live in other languages with stdio as their transport.

AgentSight: zero-instrumentation for LLM agents

AgentSight (a project from the eunomia-bpf group, the same ecosystem behind several high-profile eBPF runtimes) attacks this problem with the same philosophy as Hubble: do not instrument; listen. It puts eBPF hooks at two critical points:

uprobes on SSL/TLS libraries (libssl, boringssl, rustls). It captures the plaintext before encryption on send and after decryption on recv. For an HTTP call to https://api.anthropic.com/v1/messages, AgentSight sees the full JSON of the prompt and the response without decrypting anything in transit, simply because it has reached the syscall level before the TLS layer does its work.
stdiocap BPF: it captures read, write and dup on a process’s stdin/stdout/stderr file descriptors. This is what makes it possible to observe MCP servers that speak stdio with their client, the usual pattern for local MCP servers. You capture the JSON-RPC going back and forth without either the client or the server knowing.

Reported overhead: <3% CPU, comparable to Hubble in its own regime.

How it fits with Hubble and Tetragon

The three complement each other cleanly:

Hubble tells you: “the agent’s pod opened a TCP connection to api.anthropic.com:443 with verdict ALLOW”.
Tetragon tells you: “the claude-code process with PID 1843 did a connect() to that IP” (plus the binary, the arguments, the pod namespace).
AgentSight tells you: “the HTTPS content of that connection was a prompt messages=[{role:'user', content:'analyze this repo and modify the firewall config'}] and the response included a tool call to read_file with argument /etc/passwd”.

It is the difference between flow, process and semantics. For a security team that wants to watch AI agents in production, all three are necessary. For anyone who wants to understand cost, all three are useful (Hubble for network latency, Tetragon for resource usage, AgentSight for tokens and chosen model).

Emerging use cases

The patterns consolidating in 2026:

Agent audit trail: recording every LLM call and every tool call for compliance, especially in regulated sectors.
Jailbreak and prompt injection detection: applying rules over the prompts captured by AgentSight (similar to Tetragon’s TracingPolicy, but over semantic content).
Cost accountability: seeing which team/agent consumes which tokens, without instrumenting.
Replay and debugging: reproducing an agent’s reasoning in production without asking it to run again (which is non-deterministic).

It is a young field, AgentSight is months old rather than years, but the “eBPF as zero-instrumentation observability” pattern is very clearly spreading beyond network and process. The coming year will see consolidation and, probably, native integration with Hubble.

Common Hubble use cases

Back to Hubble proper, the cases in which any organisation deploys it:

1. NetworkPolicy debugging

The classic use: “this pod cannot reach this Service”. Without Hubble, it meant SSH, tcpdump, comparing rules. With Hubble:

hubble observe --from-pod prod-api/checkout --to-pod prod-db/postgres --verdict DROPPED

If there are drops, you see the policy responsible (Cilium 1.19+). If there are no drops, the problem is not policy: it is DNS, routing or the target service.

2. Auditing inter-namespace communication

For compliance: validating that isolated namespaces are not communicating against what was declared.

hubble observe --from-namespace prod-payments --to-namespace 'NOT prod-db' --output json

3. Exfiltration detection

Outbound traffic to suspicious public destinations. Hubble detects them by IP/SNI, not by payload (which is encrypted):

hubble observe --to-fqdn 'NOT *.example.com' --to-fqdn 'NOT *.internal' --protocol tcp

Combined with Prometheus metrics and Grafana alerts, this gives you an exfiltration radar at zero cost.

4. Real-time service SLOs

The hubble:http:response_time_seconds metrics with source_workload, destination_workload, method and status_code labels allow SLO dashboards without needing to instrument the apps. The SRE sees the p95 latency of checkout → catalog directly.

5. Performance debugging

hubble:tcp:retransmissions_total and hubble:tcp:flags_total{flag="RST"} are early signals of network problems. A rise correlated with a latency regression points you at something in the infrastructure (NIC, switch, MTU) before you go down to investigate the app.

6. Post-incident forensics

Configuring Hubble to export flows to persistent storage (via OTLP to Tempo/Loki, or hubble observe --output jsonpb to S3) gives you forensic capability: if at T+30 days you detect that something was wrong at T, you can reconstruct the traffic.

Hubble and the rest of the observability stack

Hubble does not replace Prometheus, Loki, Tempo or Jaeger; it complements them:

Prometheus: receives Hubble’s aggregated metrics. Hubble exports a native Prometheus endpoint.
Loki: receives the structured flow logs if you export them as logs. Hubble has no native Loki exporter, but a Fluent Bit with an OTLP plugin or a custom one bridges it easily.
Tempo / Jaeger: the Cilium Operator has an OTLP exporter of flows in trace format (each HTTP/gRPC flow is a span). It integrates with Tempo or any other OTLP tracing backend.
Grafana: there are already public Hubble dashboards. Combined with Prometheus, Loki and Tempo, they give you a unified panel: metrics, logs, traces, all correlated by K8s labels.

The full-stack pile seen in production in 2026 (described in Building a Production eBPF Observability & Security Stack for Kubernetes in 2026):

Data: Cilium + Hubble (network), Tetragon (process), AgentSight (AI agent).
Pipeline: OTLP Collector as the single router.
Storage: Prometheus (metrics), Loki (logs), Tempo (traces).
UI: Grafana with domain-specific dashboards.
Alerting: AlertManager with rules over the Hubble + Tetragon metrics.

Comparison with the alternatives

System	Layer	Focus	Model
Hubble	L3-L7 network	K8s cluster with Cilium	eBPF, pull metrics, push flows gRPC
GKE Dataplane v2 obs	L3-L7 network	GKE managed	eBPF (Cilium-based, managed)
Tigera Calico Whisker	L3-L7 network	Cluster with Calico	eBPF + pcap, own UI
Tetragon	Process/syscall	K8s cluster	eBPF, push events gRPC
Falco	Process/syscall	K8s cluster	eBPF in userspace or kernel module
AgentSight	LLM agent	Agentic systems	eBPF (SSL uprobes + stdio)
Beyla (Grafana)	Application	App L7 + tracing	eBPF (uprobes on libs)
Pixie	App + system	Broad cluster visibility	eBPF + PXL script
Parca	CPU/mem profiling	Performance	eBPF profile sampling

If your CNI is Cilium, Hubble is the natural entry point and it does not compete with the rest: it complements them. For Calico clusters, Whisker is the equivalent. For profiling, Parca. For AI agents, AgentSight. The era of “one tool for everything” is passing: the modern stack combines several specialised pieces, all eBPF-based, exposed via OTLP.

Operational traps

Cardinality in Prometheus

Hubble metrics with all the K8s labels can blow up Prometheus. Measure cardinality before exporting everything. The most prolific metrics are flow and httpV2; start with drop and port-distribution and add the rest incrementally.

L7 visibility costs CPU

Enabling L7 parsing via embedded Envoy adds load to the agent (not to the base datapath, but yes to the node’s envoy proxy). For heavy HTTP traffic, measure. For flows where you only need L4, leave Envoy disabled.

Hubble Relay without HA

A single Relay replica is a single point of failure for the CLI and the UI (not for the local agent, which keeps working). For production, deploy with replicas: 2+ and topologySpreadConstraints so both do not go down together.

Encryption status reporting depends on the Cilium config

The new --encryption-status filter only gives real data if Cilium has encryption enabled (WireGuard or IPsec). Without that, everything is unencrypted and the filter contributes nothing.

UI exposed without auth

Hubble UI has no native auth. If you expose it through Ingress, there has to be authentication in front of it: OIDC via oauth2-proxy, mTLS, IP allowlist. It is not optional.

Storage that does not scale

If you keep flows for days for forensics, the volume is serious. For a cluster with 100 active pods, easily 1-10 GB/day of flow logs. Plan the lifecycle (compaction, retention, cold storage) before enabling it.

What we have not covered

Mesh / multi-cluster Hubble: aggregating flows from several Cilium clusters into a single Relay. Use case: a cross-cluster view, debugging a distributed service mesh.
hubble export: local persistence on the agent’s disk for forensics with low retention.
Anomaly detection with your own models: how to connect the gRPC stream to a custom ML consumer.
AgentSight in depth: the project deserves its own article. Next instalment.
eBPF for LLM serving profiling: how to measure vLLM’s TTFT, TPOT and throughput without instrumenting, using uprobes on libcudart.

References

Hubble and Cilium:

Hubble GitHub — main repo.
Hubble — Network Observability (Cilium docs) — official reference.
Cilium 1.19 release notes (InfoQ, feb 2026) — tenth anniversary and 1.19 news.
Cilium releases — all releases.
Hubble L7 HTTP Metrics — Grafana dashboard 19423 — ready to import.
End‑to‑end L7 Visibility with Cilium Hubble (cloud-cod.com, mar 2026).
Cilium Hubble Observability Platform Internal Analysis (Young-ju).
CiliumNetworkPolicy Python Hubble: L7 Visibility 2026 — one of the threads on the Python SDK.

State of the art 2026 and the full stack:

eBPF + AI agents:

AgentSight (GitHub eunomia-bpf) — the referenced project.
Harnessing eBPF for High‑Performance LLM Workloads (Klizo Solutions).

Cross-references:

Part 1: eBPF from zero to Cilium.
Part 2: Tetragon: Cilium’s security cousin.
LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators — where the traffic Hubble observes carries the prompts AgentSight inspects.

Mcp on lo0 — Tech Blog

Making the model prefer our tools: the priority that exists in no layer, the two primitives that do, and the `_meta` that crosses the gateway

TL;DR

You are here: the question that arrives after the third server

The analogy: the counter with two trays

What the specification says: nothing

The priority that exists and is no use

The only blessed filtering is by credential

Discovery does not discover tools

What there is in proposals

The layer that decides: the client

Claude Code

The other clients

The API layer

The test: the _meta crosses the gateway

The setup

The result

The three routes where it is lost

The library trap

What we have not tested

What LiteLLM can and cannot do

The cheap lever: the description, not the name

What does not work

The ladder, updated

The risk that has to be declared

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

TL;DR

You are here: the resource side

The analogy: the visa and the border checkpoint

Part 1. The split of obligations

What is new in the revision and affects the setup

Part 2. What the SDKs give, and what they do not

Protected resource metadata

Audience validation

The authentication challenge

Scopes, which are not per tool

Which revision each one announces

Part 3. Building the protected resource

Part 4. The hop from the gateway to the MCP server

What the gateway does today

Part 5. The enterprise extension, which changes the picture

And here comes the problem

What is already in the gateway, undocumented

Part 6. Per-tool authorisation

Part 7. What still has no standard

Part 8. Who covers what

Reference configuration

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

TL;DR

You are here: the cross-cutting layer

The analogy: the civil registry and building permits

Part 1. What Keycloak is today

Project status

The internal architecture in five pieces

The minimum conceptual model

Part 2. Who can authenticate against it

Part 3. Humans and machines

People

Workloads with no person behind them

Keycloak against SPIFFE

Part 4. MCP, and the two missing standards

What the specification requires

What Keycloak supports

Part 5. The confused deputy

Part 6. Two hours of task against five minutes of token

Part 7. Sizing and operating

On Kubernetes

The official sizing figures

The benchmark

Security

The `priority` that exists and is no use

The test: the `_meta` crosses the gateway

`allow_all_keys`

`require_key_mcp_access_defined`

A hardened `config.yaml`

What does not work: `priority` in LiteLLM

By credential: `model_group_alias` at key or team level

By `User-Agent`: `tag_regex`

A different strategy per pool with `routing_groups`