Tool-Use on lo0 — Tech Blog

Making the model prefer our tools: the priority that exists in no layer, the two primitives that do, and the `_meta` that crosses the gateway

Tue, 15 Sep 2026 03:30:00 +0200

A follow-up to LiteLLM’s MCP gateway, which treated the tool catalogue as cost and as surface. This one treats the next question, the one that arrives in the meeting afterwards: so then, how do I get the model to use mine. Verified against LiteLLM 1.102.0 and MCP specification 2026-07-28, with a test of our own run on 14 September 2026.

TL;DR

Tool priority does not exist as a concept in any layer of the stack. Not in the specification, not in the clients, not in the providers’ APIs. What exist are two distinct primitives: removing the alternative from the catalogue, and leaving it behind a search step. Everything else is persuasion, and the model ignores it when it suits.

The specification has no ranking field. The Tool object is name, title, icons, description, schemas, five annotations that are unreliable hints by their own declaration, and _meta. There is a trap I see repeated: MCP does define a numeric priority, but in another structure, the one that applies to content blocks, resources and prompts. It does not govern tool selection.

The only filtering the specification blesses is by credential. The current revision says the set of tools must not vary by connection, and that it may vary according to the authorisation presented. That is exactly what a gateway with per-key lists does.

The client is the one that decides, and only one of them has anything resembling priority. Claude Code lets you exempt a server from deferral with alwaysLoad, so that its tools load in full at start-up while the rest require a prior search. It is not called priority and it works as such.

We proved that the per-tool mark crosses the gateway. We set up an MCP server with one marked tool, served it through LiteLLM 1.102.0 and captured the wire: the _meta arrives whole and with the correct key. The chain is clean on purpose, with a comment in the code that says so.

And it is lost on exactly three routes, which are the gateway’s three catalogue-trimming modes. Proxy mode, tool search driven by key permissions, and the REST listing. In other words: the per-tool mark and LiteLLM’s catalogue trimming are incompatible. You have to choose one.

There is a library trap that makes the field disappear silently. The SDK’s Tool model declares the alias without allowing population by field name, so building it with the field name puts the dictionary in the extras and serialises it under the wrong key. There is no error. The field simply does not arrive.

The cheap lever is the description, not the name. Swapping descriptions shifts the selection distribution substantially; changing only the name has minimal and inconsistent effects. And the gateway’s overrides really do replace what is sent to the client.

Reordering the catalogue is a weak lever. The model is already attending to the correct tool eighty per cent of the time when it fails. Interventions on prompt order repair at most 23 % of the failures.

And the asymmetry you build is attack surface. Whoever controls the order controls the selection: with injection rates of 1.2 % you can take over the head of a tool ranking between 91 % and 97 % of the time.

You are here: the question that arrives after the third server

The previous article closed with a decision ladder for the catalogue: trim, rewrite, split by route, and use progressive disclosure when the catalogue grows. That ladder answers how much catalogue to show.

The question that arrives afterwards is a different one, and it is more uncomfortable. A real agent has native tools from the client hosting it, it has the in-house gateway with the house servers, and it has two or three third-party servers that somebody connected. All of them compete. And what you want is not to show less catalogue: you want that, faced with a query three tools could serve, yours wins. The audited one, the one with access control in place, the one that leaves a trace.

The answer everybody gives is to write a rule in the agent’s instruction file politely asking it to prefer the in-house server. There is even a rule published in a template marketplace that does exactly that. It is text in the prompt and the model follows it when it feels like it.

This article is what lies underneath.

The analogy: the counter with two trays

Let us go back to the switchboard operator from the previous article, the one who also hands out keys. Now the building has three key providers: the house cabinet, the maintenance contractor’s, and the cleaning service’s. All three have a key to the store room and all three open it.

The head of security wants the house one used, because it is the one with a log. And he discovers he can do three things, not one more.

He can take the other two off the counter. It always works and it annoys whoever needed them.

He can leave his own in the front tray and the other two in a drawer, so that to take one you have to ask. It works almost always and annoys nobody.

And he can change the label on his key so that it describes better when it is the right one. It works sometimes, it is the cheapest, and it is the only one that also protects against the contractor changing his label without warning.

What he cannot do is put a number from one to ten on each key and trust the operator to respect it. That number does not exist. The rest of the article is why it does not exist and what to do instead.

What the specification says: nothing

The current revision is 2026-07-28. Read against the source schema, the Tool object has name, title, icons, description, input schema, output schema, annotations and _meta. There is no field for priority, weight, rank, cost, group or tag.

The annotations are exactly five: title, and the read-only, destructive, idempotent and open-world hints. The schema itself warns that all of them are hints and that a client should not make tool-use decisions based on them when they come from untrusted servers.

The `priority` that exists and is no use

Here is the mistake I see repeated most. MCP does define a priority field, numeric between zero and one. It is in the annotations interface that applies to content blocks, to resources and to prompts, alongside the audience and the last-modified date.

It is not in Tool. It has no relation whatsoever to which tool the model picks. Anyone who finds it by searching the word in the schema and concludes that tool priority is standardised has the wrong structure.

The only precedence the specification defines for a tool is one of presentation: to display the name, first the title, then the annotations title, then the name.

The only blessed filtering is by credential

There is a new sentence in the current revision that is genuinely useful and that goes unnoticed. The set of tools must not vary by connection nor as a side effect of other requests, and it may vary according to the authorisation presented in the request.

Translated: segmenting the catalogue by key, by team or by token is the legitimate way to do it. Segmenting it by session state stopped being legitimate when the July revision removed protocol sessions.

Discovery does not discover tools

server/discover, which the current revision adds and which servers have to implement, does not return tools. It returns supported versions, capabilities, server instructions, identity and the cache freshness fields. It accepts neither filters nor a cursor.

Listing is still asking for everything and paginating. The listing request accepts only a cursor: there is no query, no filter, no limit, no groups, no tags.

And on collisions between servers, the specification explicitly and reasonedly washes its hands: clients or proxies that aggregate tools from several servers will encounter collisions and should implement a disambiguation strategy, for example prefixing. And it adds that the server name is not guaranteed to be unique and should not be used to disambiguate. There is no precedence rule between servers. There is not going to be one soon.

What there is in proposals

Nothing accepted. The official proposals index only lists those in a final state, and none of them deals with priority, groups or search. In draft there is one asking for a free-text query in the listing and a filtering capability; another on groups and tags that ended up superseded; and one carrying the word priority but which is per-tool model routing, which is another thing.

The interest group exploring the grouping of primitives declares in its own minutes that it is not going to pick a canonical pattern soon, and the documents in its repository are empty. The year’s roadmap does not mention the problem.

The layer that decides: the client

If the specification gives you nothing, what is left is whatever each client has built on its own. And here there is a clear winner.

Claude Code

Tool search is on by default, and what it does is defer: MCP tools are not loaded in full at start-up, the model sees just enough and discovers them when it searches. It requires a model that supports tool reference blocks, that is Claude Sonnet 4.5, Haiku 4.5, Opus 4.5 and later.

On top of that, the lever:

{
 "mcpServers": {
 "gateway-casa": {
 "type": "http",
 "url": "https://litellm.interna.svc/mcp",
 "alwaysLoad": true
 }
 }
}

The documentation describes it bluntly: if a server’s tools must always be visible without a search step, you set alwaysLoad to true and all of that server’s tools are loaded into context at the start of the session, regardless of the global search setting. And it recommends using it for a small number of tools that the model needs on every turn, because each tool loaded up front consumes context.

That is visibility asymmetry, and it works as priority even if it is not called that. Ours in front, the rest behind one step.

There is an operational consequence that the documentation mentions and that matters more in a gateway than in an ordinary server: setting alwaysLoad makes start-up wait for that server to return its tools, capped by the standard five-second connection timeout. A gateway that aggregates eight upstream servers and does not cache the listing, as is the case, lists live against all eight every time. If one of them is slow, the session’s start-up feels it.

There is also the fine-grained variant. An MCP server can mark individual tools as always loaded by including "anthropic/alwaysLoad": true in the tool’s _meta object, with the same effect for that tool alone. That would let you load three tools from our gateway up front and leave the other forty from the same gateway deferred. It is the lever that matters, and it is the one that motivated the test in the next section.

The rest of this client’s arsenal is pure exclusion, and it is worth knowing one detail that decides whether it works or not. A deny rule with the bare tool name removes it from the model’s context; with parentheses, it does not. The documentation puts it in a table: plain WebFetch removes the tool and the model cannot search at all, whereas WebFetch(domain:*) keeps the tool and rejects every request. The difference is not cosmetic: in the second case the model still sees the tool, still tries it and still spends the context its definition occupies.

The --tools flag is the scalpel: it accepts the empty string to disable all the built-ins, and it does not affect MCP tools. That is, --tools "" leaves the model with only what the gateway serves. It is a command-line flag and has no equivalent as a settings-file key; the functional equivalent there is deny rules.

The other clients

Client	Disable the native ones	Filter the MCP ones	Priority
Claude Code	`--tools ""`, deny by bare name	deny by pattern, subagents	`alwaysLoad`
Codex CLI	`features.shell_tool` to false	`enabled_tools` and `disabled_tools` per server	No
Gemini CLI	`tools.core` as an allowlist	`includeTools` and `excludeTools` per server	No
Copilot in VS Code	Not documented	picker and tool sets	No
Cursor	No	per-server switches	No

Two notes. Copilot cuts off at 128 tools per request and its virtualisation threshold does not go above that; its embedding-based routing is internal and has no configuration surface. And Cursor’s rules are purely indicative: they are text prepended to the context.

The API layer

If the client is in-house, the margin is wider, though narrower than it looks.

Forcing a set is not possible. In Anthropic’s case, tool choice accepts auto, any, a specific one by name, or none, and the documentation says explicitly that pointing at a set of MCP tools or at a member is not supported. In vLLM it is the same: a specific one, all of them or none. There is no way to say “any of mine from the gateway”. The proposal that would introduce grammars restricting tool names to the request’s set has been open for months and is not merged.

Filtering the catalogue is possible, in all three of the big ones. In Anthropic, with an MCP tool-set block where the default configuration disables and specific ones are enabled. In OpenAI, with the allowed tools list inside the MCP block itself, which the documentation justifies on latency and cost grounds, avoiding the model seeing unnecessary definitions. And in Gemini, the remote MCP server configuration accepts restricting which of the server’s tools the agent may call.

It is worth not confusing two things that are named almost identically in OpenAI: the allowed tools list inside the MCP block filters the catalogue, and that is what matters here; the identically named tool-choice type is another thing, it is a selection restriction, and there is evidence that it does not combine well with hosted tools, without the current documentation recording it. If anyone depends on that, let them test it against their model: it fails as a request error, so it shows up on a cold start.

And there is a 2026 lever that almost nobody is using. Anthropic has in beta a mechanism for mid-conversation tool changes, with add and remove blocks that accept referencing an entire MCP set. The reason it exists is in its documentation: the tools array sits even earlier in the chunked prefix than the system field, so editing it invalidates the cache for the whole conversation; by declaring the full set at the start and using the blocks, the array never changes and the cached prefix stays intact.

Applied to our case: you can remove a rival server’s set hot, mid-session, without paying for the whole prefix. It is not available on every model.

The test: the `_meta` crosses the gateway

The fine-grained variant of Claude Code’s lever depends on a question no documentation answers: whether a gateway that aggregates upstream servers propagates each tool’s _meta all the way to the client, or loses it along the way.

We checked.

The setup

A minimal MCP server with the Python SDK, serving two tools over HTTP, one of them with the mark in place:

types.Tool(
 **{
 "name": "always_loaded_tool",
 "description": "tool with _meta",
 "inputSchema": {"type": "object", "properties": {}},
 "_meta": {"anthropic/alwaysLoad": True, "probe": "UPSTREAM_META_MARKER"},
 }
)

In front of it, a LiteLLM from the main branch with that server declared in config.yaml, with no database, with a master key. And an MCP client querying the listing against /mcp.

The result

This is what comes out on the wire:

{
 "_meta": {"litellm.ai/server_outcomes": {"probe": {"status": "ok", "tool_count": 2}}},
 "tools": [
 {
 "name": "probe-always_loaded_tool",
 "description": "tool with _meta",
 "inputSchema": {"type": "object", "properties": {}},
 "_meta": {"anthropic/alwaysLoad": true, "probe": "UPSTREAM_META_MARKER"}
 }
 ]
}

The mark arrives whole and with the correct key. The client parses it without trouble.

Tracing the code, the chain is clean on purpose. LiteLLM keeps the SDK objects as they are when it lists against the upstream, instead of rebuilding them. The name prefixing does a deep copy and mutates only the name, with a comment in the code declaring the intention to preserve every field including the _meta by avoiding mutation. The two permission filters are list comprehensions that reuse the same objects. And the name and description overrides mutate in place, so they do not lose anything either. Somebody thought about this.

One operational detail: the name arrives prefixed with the server alias, governed by the configurable separator and the short prefix mode.

The three routes where it is lost

And here is the finding that changes the previous article’s recommendation.

Proxy mode returns only its three fixed tools, with no _meta and with no upstream tool at all. Verified against the wire as well.

Tool search activated by key permissions does the same with four virtual tools.

And the REST listing builds its response object by hand and discards the field: the tool that did carry a mark comes out with the field null. The class inherits from the type that has the field; it simply is not filled in. The fix fits on one line.

Those three routes are, exactly, the three ways LiteLLM has of trimming the catalogue. The operational conclusion is uncomfortable and it is worth saying plainly: either you use the gateway’s catalogue trimming, or you use the client’s per-tool mark. They cannot be combined. The previous article recommended proxy mode as the fifth rung of the ladder; with this in hand, that rung and the fine-grained mark are mutually exclusive.

The library trap

This deserves its own section because it takes out anyone writing a server or a proxy, and it gives no warning.

The SDK’s tool model declares the field with an alias, and does not enable populating it by name. The consequences, both checked by running them:

Building with the field name populates nothing. The dictionary slips into the model’s extras and comes out serialised under the wrong key, without the leading underscore, which is a key no client interprets.

And dumping and revalidating without asking for aliases loses the field, for the same reason.

There is no exception, no warning in the log, nothing. The field disappears. If someone marks their tools and does not see them marked in the client, this is the first place to look, before the gateway.

What we have not tested

For honesty’s sake, and because it is the link that remains: we have shown that the mark survives the gateway. We have not shown that the client acts on it when the tool arrives with the name prefixed by the gateway. Claude Code’s documentation describes the per-tool mark in a single sentence and says nothing about aggregating servers or about prefixes.

It is a half-hour test for anyone who has the setup in front of them: two tools from the same gateway, one marked, and look at which one appears loaded at the start of the turn. If someone does it before I do, I am interested in the result.

What LiteLLM can and cannot do

With the above, the gateway’s inventory of levers comes out like this.

Lever	Serves external MCP clients	Preserves the `_meta`
`allowed_tools` and `disallowed_tools` per server	Yes	Yes
Name and description overrides	Yes	Yes
Tool sets per route	Yes	Yes
Proxy mode and virtual tools	Yes	No
Search by key permissions	Yes	No
REST listing	Yes	No
Semantic filter	No	Not applicable

Two clarifications on the previous article’s table, now that the code has been read again.

The allowed and disallowed lists are applied in the listing, not only on the call. They are the blunt, effective instrument, and they do not break the cache because they are static.

And the overrides have a limit that was not accounted for: they are applied in the MCP protocol listing, but they are skipped in proxy mode and not applied on the REST route. Anyone rewriting descriptions and also turning on proxy mode is not serving what they think.

What the gateway cannot do is inject _meta. There is no setting equivalent to the ones for name and description. The mark has to be put there by the origin server, which for in-house servers is trivial because we write them. To mark third-party tools you would have to patch, and it is small: one field in the type and three lines in the overrides function if configuring it by file is enough.

There is also an ordering lever that did not appear in the previous article. In the proxy’s search mode there is a core tools setting that puts them at the front of the ranking and that also does not count against the results cap. It is the closest thing to a declarative priority in the whole stack, and it lives inside the one mode that discards the _meta.

And a nuance about proxy mode’s results cap. The client cannot ask for more than five, that is true, but the operator can raise it by configuration. The previous article implied it was immovable.

The cheap lever: the description, not the name

If you cannot exclude, what is left is to bias. And bias comes in through the description.

The work that measures selection bias between functionally equivalent tools puts the combined bias of the evaluated models between 0.25 and 0.38, that is, you would have to redistribute between 25 % and 38 % of the probability mass for equivalent tools to be picked equally. On that basis, swapping two tools’ descriptions shifts selection substantially, whereas perturbations that touch only the name produce smaller and more inconsistent effects.

Another piece of work measures that a single pass of automatic description rewriting improves the metric over a large corpus, almost as much as iterative refinement, and with two orders of magnitude less time. And it adds a warning worth retaining: descriptions tuned against a fixed set of candidates do not generalise to the dynamically retrieved set. You have to optimise for the regime you serve in.

Applied: the gateway’s description override is at once token trimming, selection bias and mitigation of description change after approval. It is the lever with the best effort-to-effect ratio in the whole article, and it is static, so it does not touch the prompt cache.

Name prefixing, by contrast, costs tokens and probably does not change which tool the model goes to.

What does not work

Reordering the catalogue. A June paper measures that, when the model fails, it was already attending to the correct tool eighty per cent of the time, well above chance. The bottleneck is not at the input but in the late layers. Interventions on prompt order recover at most 23 % of the failures, against the 59 % to 91 % of those acting on the final readout. Reordering is cheap and that is why it gets recommended a lot; it is also weak.

Rules and instructions. They are text. They help and they do not decide.

Your own per-request top-K filters. This is covered in the previous article and it still holds: four out of five measured strategies perform worse than not filtering, and on top of that rewriting the tool block on every turn invalidates the entire cached prefix.

The ladder, updated

Exclude in the client. It is the only deterministic thing. --tools "" or deny by bare name for the competing native tools.
Mark your own as always loaded. At server level with alwaysLoad, or per tool with the mark in the _meta if you control the origin server. Counting on start-up waiting for the server.
Trim with static lists per server and per key, and split a catalogue by agent profile across routes.
Rewrite the descriptions with the gateway’s overrides.
Remove hot the rival sets, if the client is in-house and the model supports it.
Choose: the gateway’s catalogue trimming, or the per-tool mark. Not both.
Do not: reorder, rename, or trust prompt rules.

The risk that has to be declared

The asymmetry you build is also a surface. Whoever controls which tools head a ranking controls the selection, and that can be attacked: there is work measuring that by injecting adversarial tools at rates of 1.2 % you take over the head of the ranking between 91 % and 97 % of the time.

In a system under ENS (Esquema Nacional de Seguridad, Spain’s national security framework) this fits under change management, not access control: the set of tools a model sees and the order in which it sees them are security configuration, and today nobody versions them. Neither does the specification fix a hash of the description or of the schema, nor does the gateway. The specific Annex II codes are worth checking against the current text before taking them into a compliance document; the approach is developed in the MCP gateway article.

There is a precedent within the ecosystem itself that points the way. The MCP skills extension, which is in a final state, requires a manifest with a per-file hash, mandatory verification before use, and approval tied to the set of files and their digests, so that any change revokes the approval. And it obliges hosts to prevent two skills with the same name from silently replacing one another. It is exactly the pattern missing for tools. Anyone who needs it today has to implement it in their gateway.

Checklist

Decide the strategy before touching anything: catalogue trimming in the gateway, or the per-tool mark in the client. They are mutually exclusive.
If you pick the mark, put it in the origin server, because the gateway cannot inject it.
Build the tool object with the alias key, never with the field name, or the field disappears without warning.
Do not revalidate tool objects from a dump without asking for aliases.
Count on marking a server as always loaded making session start-up wait for that server, and on a gateway listing live against all of its upstreams.
Exclude in the client the native tools that compete, by bare name and not with a pattern in parentheses.
Rewrite the descriptions of your own tools, and do not waste time renaming.
Do not turn on proxy mode if you depend on the description overrides, because they are not applied there.
Version the exposed tool set and its order as security configuration, with a hash of the description and the schema.
If you use the gateway’s REST route for anything, know that it discards the _meta.

Traps and things that are not what they look like

MCP’s priority exists, and it is not about tools. It lives in the annotations for content, resources and prompts.
Tool annotations are hints and the schema itself says they are not reliable from untrusted servers.
The specification gives no precedence rule between servers, and it says besides that the server name is no use for disambiguating.
The tool set may indeed vary by authorisation, and that is the only blessed segmentation.
server/discover does not return tools.
A deny rule with parentheses does not remove the tool from the context, it only rejects the calls.
--tools is a command-line flag, with no equivalent in the settings file.
Marking a server as always loaded delays start-up by up to five seconds per server.
Tool(meta=...) does not populate the field and serialises it under the wrong key.
Proxy mode, search by permissions and the REST listing discard the _meta.
Description overrides are skipped in proxy mode and are not applied on REST.
Proxy mode’s cap of five results can be raised by the operator, even though the client cannot ask for more.
The core tools of the proxy’s ranking are the only declarative priority in the stack, and they live in the mode that loses the _meta.
You cannot force “any of mine from the gateway” in any API.
In OpenAI, the allowed list inside the MCP block and the identically named tool-choice type are different things.
Gemini does accept a per-tool allowlist on its remote MCP servers.
Reordering the catalogue repairs at most 23 % of the failures.

Closing

The starting question had a trap in it, and the trap is the word. When someone asks how you prioritise an MCP server, they are assuming a dial exists somewhere. It does not, and I have spent the whole article showing the places where it is not.

What there is instead is poorer and more manageable: you remove what competes, you leave your own in front, and you write the description better. Three things, none of them elegant, all three effective in that order.

What does seem to me worth taking away is the shape of the problem. The tool catalogue a model sees is security configuration in the full sense, because it determines what the agent can do and which system it is going to talk to. And today it is managed like a list of connections: things get added, it is not versioned, it is not signed, and nobody detects a change. The ecosystem has already solved that problem once, for skills, with hashes and tied approval. For tools, not yet.

In the meantime, the asymmetry has to be built by hand, server by server, and knowing that it rests on the _meta of an object that a library can empty without saying anything.

Sources

Model Context Protocol, herramientas en la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/server/tools.
Model Context Protocol, server/discover: https://modelcontextprotocol.io/specification/2026-07-28/server/discover.
Model Context Protocol, paginación: https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/pagination.
Model Context Protocol, caché de listados: https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/caching.
Model Context Protocol, esquema fuente de la revisión: https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/schema/2026-07-28/schema.ts.
Model Context Protocol, extensión de habilidades: https://modelcontextprotocol.io/extensions/skills/overview.
Model Context Protocol, grupo de interés sobre agrupación de primitivas: https://modelcontextprotocol.io/community/interest-groups/primitive-grouping.
Anthropic, Claude Code y MCP (alwaysLoad, búsqueda de herramientas): https://code.claude.com/docs/en/mcp.
Anthropic, Claude Code, referencia de línea de comandos (--tools): https://code.claude.com/docs/en/cli-reference.
Anthropic, Claude Code, permisos (denegación por nombre pelado): https://code.claude.com/docs/en/permissions.
Anthropic, conector MCP y conjuntos de herramientas: https://platform.claude.com/docs/en/agents-and-tools/mcp-connector.
Anthropic, cambios de herramientas a mitad de conversación: https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages.
Anthropic, búsqueda de herramientas: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, MCP y conectores (lista de herramientas permitidas): https://developers.openai.com/api/docs/guides/tools-connectors-mcp.
OpenAI, búsqueda de herramientas y caché: https://developers.openai.com/api/docs/guides/tools-tool-search.
Google, llamada a funciones en la Interactions API: https://ai.google.dev/gemini-api/docs/function-calling.
vLLM, llamada a herramientas y decodificación restringida: https://docs.vllm.ai/en/latest/features/tool_calling.html.
vLLM, propuesta de decodificación guiada por regiones y gramáticas de herramienta: https://github.com/vllm-project/vllm/issues/39848.
LiteLLM, código del gateway MCP: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, tool_search.py, litellm/experimental_mcp_client/tools.py: https://github.com/BerriAI/litellm.
SDK de Python de MCP, modelo Tool y serialización por alias: https://github.com/modelcontextprotocol/python-sdk.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307v2.
Looking Is Not Picking, atención frente a lectura final y techo de las intervenciones sobre el prompt: https://arxiv.org/html/2606.16364v1.
A Single Rewrite Suffices, reescritura de descripciones y generalización al conjunto recuperado: https://arxiv.org/html/2606.30775.
ToolFlood, saturación adversaria de la capa de recuperación: https://arxiv.org/html/2603.13950.

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

Fri, 11 Sep 2026 01:45:00 +0200

Sixth post in the operational track on the control layer. The previous five treat the gateway as the piece the tokens pass through: the pair with Langfuse, day 2, virtual keys, humans and agents and prefix routing. This one is about the other door. Verified against LiteLLM 1.102.0, commit of 10 September 2026.

TL;DR

The MCP gateway is not a module, it is a complete second surface. A native MCP endpoint with four route spellings, a proxy mode with three fixed tools, four REST routes, some thirty-five administration routes and a long set of OAuth discovery endpoints. It still lives under _experimental and it is nearly 20,000 lines.

I correct what I said in the previous post: access is NOT open by default. In 1.102.0, a key with no declared permissions and no team resolves zero servers. What does open things up is a per-server flag, allow_all_keys, and that flag overrides the key’s explicit scope unless you enable a setting that is off by default.

Permissions are enforced on execution, not just on the listing. Calling a tool that did not appear in tools/list returns 403. There are three chained checks before the call, including one on the allowed arguments. The same predicate governs listing and calling, by declared design.

The hierarchy is built out of intersections, with two escapes and a fail-open. The key’s access groups are additive on top of the key and team ceiling. The organisation ceiling replaces rather than intersects when there is nothing below it. And on an indeterminate error while resolving tool permissions, the code returns “unrestricted”. It is a deliberate decision and it has to be declared in the risk analysis.

Per-tool cost is worth zero and that is why its metric disappears. There are exactly two Prometheus metrics for MCP, and the spend one is only incremented if the cost is greater than zero. With the default configuration, that series never exists. There is no listing metric, no latency metric, no error metric, no server health metric.

The cost that does matter is not attributed to MCP anywhere. It is the tokens of the tool definitions, resent on every turn inside the prompt_tokens of the model call. Five typical MCP servers add up to 58 tools and some 55,000 tokens. The tool’s spend row, with its zero cost, has nothing to do with that.

And the gateway’s semantic filter does not act on the MCP door. It only applies when it is LiteLLM calling the model. A client connected to /mcp receives the whole catalogue, filter enabled or not. And even if it did act, filtering per request rewrites the cached prefix of the prompt, which is the opposite of what Anthropic and OpenAI do with their own tool searches.

The good observability ships switched off. Without LITELLM_OTEL_V2, a tool call comes out as a generic span with the model set to MCP: name. With it enabled there is a dedicated span and correct attributes, and the arguments and the result stay hidden unless you opt in. And in neither case is trace context propagated towards the upstream MCP server: the trace is cut at the gateway.

Tool arguments are written, and they are written in the clear. They go in full into the metadata column of the spend table, without passing through the switch that governs prompt storage nor through the redaction function, which only knows about messages and response. Whoever enabled redaction for GDPR probably believes they are covered.

The 1.83 branch concentrated four serious vulnerabilities, and one of them belongs to the MCP gateway itself. The endpoint for testing a connection before saving accepted the stdio transport’s command, args and environment in the request body. A low-privilege key got command execution on the proxy host.

And the specification moved underneath. The revision in force since July 2026 removed protocol sessions, the initialisation handshake and the session header. LiteLLM 1.102.0 advertises the June 2025 revision and its version enumeration does not even contemplate the two later ones.

You are here: the door nobody chose

The gateway was chosen for the reasons in the switchboard post: licence, fit inside the cluster, maturity. What was evaluated was the inference traffic.

Later, somebody connects an agent, the agent needs tools, and the same process that is already authenticated and deployed turns out to speak MCP too. The decision to turn it into the tool door is almost never taken: it is inherited. And that door has neither the same controls, nor the same observability, nor the same cost model.

When MCP grows covered the identity of your own MCP servers. MCP from the inside covered the protocol and its instrumentation. This post covers the piece in between.

The analogy: the switchboard that also hands out keys

An office building’s telephone switchboard. Its original job is to route calls: it decides who can call outside, how much each department spends, and it leaves a record of every conference call.

Over time, the switchboard operator is also given the key cabinet. They are already at reception, they already know who everybody is, it looks like the natural place. And from that moment there are two services at the same desk with very different rules.

Calls are priced by the minute and show up on the bill. Keys have no tariff, so on the department’s bill they come out at zero, and somebody might conclude that the key cabinet costs nothing. It does cost: it costs the operator’s time, it costs having to read the whole key catalogue every time somebody asks what is available, and it costs that nobody checks that the label on a key still says what it said the day its loan was authorised.

The analogy ends at the interesting spot. When somebody takes a key, the record says which key it was and what they said they wanted it for. That record is simultaneously the evidence of the action and a piece of data that perhaps should not be stored in the clear.

The real routes

The first thing that surprises you when you look at the code is how many spellings get you to the same place.

The native endpoint

The MCP sub-application is mounted at /mcp and inside it has four mounts: the root as a wildcard, /mcp, /{name}/mcp and /sse.

And there is also a hand-declared route for bare /mcp on the main application, with a comment that explains why: the mount cannot match its own bare prefix, and the resulting 307 redirect breaks MCP clients sitting behind a proxy that terminates TLS. It is the kind of detail you discover in production on a Friday.

/mcp/sse holds a surprise. It does not use the classic SSE transport: it uses the same HTTP session manager in stateless mode. The SseServerTransport object is constructed and is not referenced on any other line of the file. It is dead code, and the /mcp/sse/messages endpoint that this object advertises does not exist as a working route.

The /{name}/mcp route resolves in a documented order: server alias, comma-separated list with a maximum of 16 elements, tool set, access group. If nothing matches, 404.

And /mcp/proxy exposes a fixed surface of three tools. It is the progressive disclosure pattern, and it will come up again further down because it is the most important mitigation in this post.

The REST routes

Four, under the /mcp-rest prefix:

Method	Route	Authentication
GET	`/mcp-rest/tools/list`	Virtual key
POST	`/mcp-rest/tools/call`	Virtual key
POST	`/mcp-rest/test/connection`	Key plus administrator role
POST	`/mcp-rest/test/tools/list`	Key plus administrator role

The variants without -rest that appear in the internal types have no router registering them. The /mcp/* wildcard absorbs them as JSON-RPC.

The last two are the stars of the security section.

Administration and discovery

Some thirty-five routes under /v1/mcp, almost all of them with virtual key authentication. Three details worth knowing:

GET /v1/mcp/registry.json carries no authentication dependency. It returns 404 if the public registry is not enabled, which is the real protection.

POST /v1/mcp/server requires the proxy administrator role. POST /v1/mcp/server/register does not: it is the proposal path, which requires a key with a team, explicitly forbids the stdio transport and leaves the entry pending review.

And on top of that there is a long set of OAuth .well-known endpoints, with dynamic client registration, authorisation, token, revocation and introspection, plus their per-server variants.

Configuration and transport

The mcp_servers schema in config.yaml has fifty-odd fields, all of them optional. A warning for anyone coming from earlier versions: spec_version no longer exists in 1.102.0.

The default transport when omitted is http, that is, streamable HTTP. The other two values are sse and stdio. The string streamable_http is not valid in the configuration file, although it is accepted as a synonym when importing connector files.

Tool prefixing uses the format {server}{separator}{tool} with - as the default separator, configurable through an environment variable read at import time. The prefix chosen is the alias, or the name, or the identifier, in that order. There is an optional short mode that derives three characters from the hash of the server identifier and resolves collisions by rehashing.

Access control, and the correction to the previous post

In the humans and agents post I closed the list of traps with this one: “access to MCP servers is open by default if no level defines a list”. Verified in 1.102.0, that is false.

The function that resolves the allowed servers does the opposite. If the team declares nothing, it inherits from the key; if the key declares nothing either, the base set is empty and the scope is marked as unbounded with zero servers. There is no branch that expands to “the whole registry” other than the administration role one.

What does open things up, and this is the important part, is something else.

`allow_all_keys`

It is a per-server flag, in configuration, in the API and in the database, defaulting to false. When set to true, that server joins the set of open servers and any key can see it.

The detail to know: by default it overrides the key’s explicit scope. A key that declares access only to server A will also see server B if B has the flag. The key’s scope is only respected if you enable general_settings.mcp_allow_all_keys_respects_mcp_scope, which is off by default.

There is a second opening case: anonymous subjects, with no user identifier, role or key, additionally receive the servers configured with authentication delegation towards the upstream and the pass-through ones. It is consistent with the design of those two modes, and it is a surface you have to know about.

`require_key_mcp_access_defined`

It lives in general_settings, defaults to false, and needs no licence. What it inverts is only the empty-key inheritance towards the team: with the flag on, a key with no list of its own gets an empty set instead of inheriting the team’s.

It does not affect keys that do declare servers, which still intersect with the team. And it does not close off the access group grants, which are additive and are the escape route you have to audit separately.

There is a sibling, require_end_user_mcp_access_defined, with the same shape and applied to the end user.

The hierarchy

The function’s docstring is normative and says that all the rules are intersections. In detail:

Key and team: intersection when both declare. Inheritance in one direction when one is empty.
Key access groups: additive union on top of the previous result.
End user: intersection if it declares anything.
Agent: intersection.
Internal user: ceiling that only narrows.
Organisation: ceiling that intersects if there are restrictions below, and that replaces by becoming the ceiling when there are none.

For subjects with no key, admitted by SSO or by the gateway, the model changes to a per-source union instead of an intersection.

The fail-open

This is the point you have to take to the risk analysis and not leave in a technical note.

On an indeterminate failure while resolving the tool permissions of a key or a JWT, the code returns “unrestricted”, that is, all tools allowed. The organisation ceiling is skipped in the same situation. For subjects with no key the behaviour is the opposite, fail-closed.

There is a well-resolved intermediate case: when a permission is named but cannot be read, a specific error is raised and it is denied in both cases.

The doctrine is documented in the class docstring, which is appreciated. And it remains a behaviour that, in a system under the ENS, you have to declare in writing instead of discovering it during an audit.

What is done well

Permissions are enforced on the call, not just on the listing. The execution path chains three checks before contacting the server: allowed or forbidden tools, per key and team tool permission, and validation of the arguments against the allowed ones. All three return 403.

Listing and calling share the same predicate, and the docstring declares it as an explicit invariant. A client that calls a tool directly when it did not appear in its listing receives 403, it does not execute. The same over REST and over the responses API route.

The x-mcp-servers and x-mcp-access-groups headers only narrow, and they fail closed if they do not resolve.

Header forwarding towards the upstream is an explicit per-server allowlist, with a centralised decision about when to strip the caller’s Authorization, and a protection against cross-forwarding: in a multi-server listing, the global header is withheld if more than one server would consume it. That is RFC 9700 applied.

The bill that never shows up

Here is, in my opinion, the most important thing in this post, and it has nothing to do with security.

Every tool the gateway aggregates takes up space in the model’s context. Name, description and parameter schema travel in the request’s tools block, on every turn, and they are paid for as input tokens of the real model.

The figures published by Anthropic in November 2025 are the best public reference there is. Five connected MCP servers add up to 58 tools and some 55,000 tokens. The breakdown: GitHub, 35 tools and some 26,000 tokens. Slack, 11 tools and some 21,000. Sentry, five and some 3,000. Grafana, five and some 3,000. Splunk, two and some 2,000.

With tool search enabled, that set drops from some 72,000 to some 8,700 tokens, and the usable context goes from 122,800 to 191,300.

The second cost of a large catalogue is not economic and is not billed: it is that the model chooses worse. It has its own section further down, because of the two it is the one that decides whether the agent is any good.

How to measure it with what is already there

The gateway stores enough data, even if it does not cross-reference it for you.

In the listing’s spend row, with call type list_mcp_tools, it enriches the metadata with allowed_server_count, tool_count_total, per_server_tool_counts and per_server_list_outcomes.

In the model call rows, the proxy_server_request column keeps the request body with the tools block inside it.

The practical method: pair the listing rows with the LLM rows of the same session by session_id, read the total number of tools from the first, and pass the tools array of the second through litellm.token_counter(model=..., tools=...), which accepts that argument. Multiply by the number of turns, because the block is resent on all of them.

For the relative cost, a quicker comparison: mean prompt_tokens of keys with MCP enabled against those without, same model and same period.

Tool selection

The previous section treats the catalogue as a bill. This one treats it as what determines whether the agent gets it right. This is ground where the talk runs well ahead of the measurement, so it is worth separating what has been measured from what gets repeated.

How many tools a model can take

Anthropic’s official figure is that Claude’s ability to pick the right tool degrades when going from 30 to 50 available tools. The same documentation gives the inverse criterion, which is the more useful one: the full catalogue with nothing else is the right option with fewer than ten tools, when all of them are used on every request, or when the definitions add up to fewer than a hundred tokens.

There is also an administrative ceiling before the semantic one. VS Code with Copilot cuts off at 128 tools per request and returns an error when you exceed it.

The full curve gets cited far more than it has been measured. The soundest primary data I found is from May 2025: varying the number of candidates from 1 to 11,100 over a registry of more than 4,400 servers, below some 30 tools the success rate is above 90 %, between 31 and 70 intermittent failures appear, and past the hundred mark the degradation becomes dominated by retrieval. It is a qualitative heat map, not a numeric table.

The figure that circulates most (78 % with ten tools, 40 % with a hundred, 13.6 % with seven hundred) comes from a conference talk with no paper, no repository and no published methodology. Do not take it to a client presentation.

And there is a warning that affects everything else. An audit of 496 tasks from four tool calling benchmarks found 18.5 % misalignment between the label and reality. In the BFCL v4 subset, 80 % of the failures assigned to the agent came from fragile state comparisons. In one of the benchmarks with an automatic judge, 23 identical re-runs gave a range from 57.9 % to 76.8 %, almost 19 points of spread, enough to reorder the ranking. Any absolute figure in this section deserves a generous margin of error.

How it fails

The work with the cleanest taxonomy evaluates 36 servers, 220 tools and 1,000 tasks against twenty frontier models, with an average of 15.2 tools exposed per task of which 4.1 are relevant.

Failure mode	Share
Not using a tool when one was needed	10.5 %
Choosing the wrong tool	9.0 %
Malformed parameters	6.9 %
Not recovering from an error	10.3 %
Cognitive failures, not tool failures	63.3 %

The interesting part is in the per-model breakdown: the dominant failure mode changes with the model, not with the size of the catalogue. In one of them, 40.1 % of the failures are simply not calling any tool. In others the malformed parameter dominates. In another, the wrong tool. There is no canonical failure to attack.

On tools that do the same thing, which is the real case of a gateway with fifteen servers, there is a good measurement. A study with ten groups of five functionally equivalent tools and a thousand query pairs puts the selection bias between 0.3 and 0.4, that is, you would have to redistribute between 30 and 40 % of the probability mass for equivalent tools to be chosen equally. The same work shows that a biased continued training takes the selection of a specific endpoint from 0.6 % to 12.8 %.

That study contributes another figure that reorders priorities: shuffling a tool’s description moves the selection distribution substantially, whereas changing only the name has minimal and inconsistent effects. Applied to the gateway’s server-tool prefixing: it costs tokens and probably does not change which tool the model goes to. There is no study isolating the effect of name spacing, and it is a real gap in the literature.

If the description is what carries weight, the state of the estate is the problem. An audit of 856 tools from 103 MCP servers found undeclared limitations in 89.8 %, missing usage guidance in 89.3 % and opaque parameters in 84.3 %. Only 2.9 % were free of problems across all their components.

The optimum is not a number

The May 2026 work I cited above proposes a chance-corrected metric and uses it to size the list adaptively. Over 370 functions it reaches 90.3 % coverage while seeing an average of 7.4 tools, against 90.8 % seeing a fixed 50.

But the same paper publishes its own counterexample, and it is honest to cite it: over another benchmark of 3,251 tools, the fixed K of 5 wins in aggregate, 64.7 % against 61.9 %. The adaptive advantage concentrates in the hard queries, where the right tool sits between sixth and twentieth by similarity: there it finds 16.7 % of the cases where the fixed K finds 0 %.

The March 2026 work over 121 tools puts the plateau at K=3, with 97.1 % that does not improve going up to K=5. With the caveat that it measures retrieval against labels, not model accuracy.

And a July 2026 work gives the formulation that seems to me the right one: ranking by score is inconsistent with optimal acquisition when costs are heterogeneous, so the optimal K is a function of cost, not a constant. Trimming from 7 to 4.4 exposed tools while keeping success is possible; fixing a universal number is not.

Filtering can make the result worse

This is the part that appears in no product blog. A June 2026 benchmark with three registry sizes (25, 100 and 250 tools), six filtering methods, seven models and 26,460 runs:

Strategy	Success	Visible tools	Tokens
Expose the whole catalogue	32.1 %	125.00	56,062
Keyword, top-5	22.1 %	4.80	3,200
Keyword, top-10	22.4 %	9.54	5,356
State-aware	24.0 %	25.95	13,522
Full causal path	24.0 %	26.24	13,697

Four of the five filtering strategies perform worse than not filtering. The fifth, which does win comfortably, exposes 0.99 tools per step and requires hand-written precondition and effect contracts for every tool, plus a search of the causal path to the goal state. It is a planner with the goal given, not a retrieval filter, and it is not comparable with anything a gateway offers.

There is more evidence in the same direction. A work over a corpus of 43,000 tools measured that the agent’s completion rate drops when using retrieved sets against hand-annotated sets: the filter introduces its own recall ceiling. And top-K breaks the tasks that need several tools: on a benchmark of 16,464 endpoints, a fine-tuned retriever reaches a Recall@3 of 68.6 % but only 39.7 % of the queries have their complete tool set within the first three. Almost 29 points of gap between “the one I needed is there” and “all the ones I needed are there”.

Filtering is not free on security either. A March 2026 work attacks the retrieval layer by injecting adversarial tools that saturate the top-k: with injection rates of 1.2 % it manages to dominate the top-k between 91 % and 97 % of the time. It is an attack surface that exposing the whole catalogue does not have.

The most elegant counterpoint comes from Anthropic’s own numbers. With the same catalogue of more than fifty tools, the gain from tool search drops from 25 points on one model to 8.6 on the next, because the unfiltered baseline rises from 49 % to 79.5 %. The margin left by filtering shrank to a third in one generation of models. Any architecture decision taken here has a short shelf life.

The prompt cache trap

Here is what makes a homemade top-K filter a bad idea even if it worked.

Tool definitions do not go just anywhere in the prompt: they go at the beginning. Anthropic documents the order of the cacheable prefix as tools, then system, then messages, with a hierarchy where each level rests on the previous one. And the invalidation rule is explicit: modifying tool definitions, be it names, descriptions or parameters, invalidates the entire cache, and a change at one level invalidates that level and all subsequent ones.

OpenAI says the same and adds a detail: the prefix must remain unchanged, and order counts. The same set of tools served in a different order is a cache miss.

With vLLM serving an open model there is no tool block of its own. The tools parameter is serialised by the chat template inside the first system message, behind the system prompt text. For automatic prefix caching the effect is identical, because it is still near the absolute start of the prompt.

The consequence is the one that matters. A filter that returns a different set of tools on the next turn does not just invalidate the tool block: it invalidates everything behind it, which is the system prompt and the whole conversation, which is exactly the part that grows with the turns. At list prices, a prefix token goes from costing 0.1 times when read from cache to 1.25 times when written. An agent with sixty thousand tokens of stable prefix goes from some six thousand equivalent tokens per turn to some seventy-five thousand, plus the prefill that gets paid again in latency.

The two big providers have separately arrived at the same solution, and it is the opposite of filtering in the prefix. Anthropic excludes deferred tools from the prefix and, when the model discovers one through search, injects it inline inside the conversation: “the prefix stays intact, so prompt caching is preserved”. OpenAI: “all tools are loaded at the end of the model’s context window (…) this allows the cache to be preserved from one request to the next”. Anthropic’s operational recommendation is the stable core pattern: leave the three to five most used tools undeferred.

The closest academic work, from January 2026, states the problem without measuring it and recommends exactly that, keeping a fixed set of reusable general-purpose functions and implementing dynamic capability through code generation.

The July 2026 MCP specification picks up both halves of the matter. Servers “should return the tools from tools/list in a deterministic order to allow caching in the client and improve the hit rate of the model’s prompt cache”. And it adds mandatory ttlMs and cacheScope in listing results, so that the client can freeze the catalogue for a known window instead of re-querying it and propagating any variation to the prompt.

Nobody has publicly measured the cost in euros of this effect. The only attempt I found is a self-published audit of ten thousand turns that puts it at 2.4 % to 3.5 % of cache misses, with non-reproducible methodology. It is a gap in the literature, and the order of magnitude in the previous paragraph comes from applying list prices, not from a measurement.

And now, LiteLLM

With all of the above in mind, the gateway’s levers read differently.

The semantic filter does not act on the MCP endpoint. It is registered as a generic LiteLLM callback, and its only dispatcher is the proxy’s precall hook, which additionally discards anything that is not completion, acompletion or aresponses. The tools/list path does not invoke that hook at any point; there is a comment in the code that admits it in passing, when explaining why the listing’s JWT signature is done separately. An external MCP client connected to /mcp receives the whole catalogue, filter enabled or not.

Lever	Serves external MCP clients	Breaks the prompt cache
Per-server `allowed_tools`	Yes	No, it is static
Name and description overrides	Yes	No, it is static
Selection by route or header	Yes	No, if it is stable per agent
`/mcp/proxy` and virtual tools	Yes	No, the surface is fixed
Semantic filter	No	Yes, it rewrites the block per request

When it does act, the filter has details you need to know before enabling it. It embeds only each tool’s description, with the name as a fallback if it is missing; the parameter schema does not go in. Of the query it embeds only the last user message: on turn twelve of an agent conversation, that might be “yes, go ahead”. The tool embeddings are cached and indexed at start-up; the query’s is computed on every request, with a synchronous call inside a coroutine.

Its failure modes are almost all open, which is the reasonable thing: if the embedding model does not respond, if nothing passes the threshold, if the names do not match the catalogue, it returns all the tools. The exception is context overflow, which fails closed with a 400, and which if it happens while building the index during start-up is memoised and makes all subsequent requests fail until you restart. The defaults are text-embedding-3-small, top_k of 10 and a threshold of 0.3, under litellm_settings.mcp_semantic_tool_filter.

/mcp/proxy does serve external clients, and that is where its real value lies. It replaces the catalogue with three tools with opaque identifiers. How search_tools searches deserves a line, because it is not what you would assume: by default it is neither a regular expression nor a semantic match, it is a keyword counter, how many tokens of the query appear as a substring in the concatenated name and description. It only moves to embeddings if you configure a model in mcp_tool_search. The result cap is hard-coded at five in this mode, and the client cannot ask for more. In exchange, call_tool goes through the same permission checks as a normal call plus a validation of the argument schema that the normal route does not do at that point.

There is a related variant: with mcp_tool_search_enabled in the key’s permissions, the catalogue is replaced by four virtual tools, and there the search one does expose top_k to the client.

The catalogue order is not deterministic, and that breaks the client’s cache without anybody filtering anything. The list of allowed servers comes from iterating a Python set of strings, so with the random hash seed the order differs between processes, between uvicorn workers and between restarts. There is no sorting anywhere in the listing path, and there is no tool cache: every tools/list queries the upstreams live, so the order within each server also depends on what the upstream returns that time. The gateway does not offer today the guarantee that the July specification recommends. The fix, sorting the catalogue before serving it, fits on one line.

Nor is there any pagination towards the client. The catalogue is returned whole, always.

And the lever almost nobody uses is the static one. Every MCP server accepts tool_name_to_display_name and tool_name_to_description, and those overrides really do replace what is sent to the client in tools/list, not just what the administrator sees in the interface. The tools/call routing translates from the new name to the real one before any permission check. With that you can trim a four-hundred-token description written by a third party, rename an ambiguous tool so it does not compete with a similar one, and do it without touching the prompt cache, because it is static and holds for all turns.

It is also the rug pull mitigation left pending from the security section: if the description the model sees is the one the operator wrote, whatever the upstream server changes stops reaching the context. Two limits. They are configured through the database or the management API, not through config.yaml. And the reverse name translation is by exact equality and is not disambiguated between servers, so two servers with the same display name route unstably, made worse by the non-deterministic order of the previous paragraph.

Alongside that, per-server allowed_tools remains the blunt and effective instrument: it filters the listing the client sees, reduces tokens, and is static.

The ladder, in order

Trim. Per-server allowed_tools, plus a per-key allowlist. Cheap, static, does not break the cache and reduces the permission surface at the same time.
Rewrite the descriptions with the overrides. That is where the evidence is, because the description carries weight and the name almost none, and along the way it pins the contract against the rug pull.
Split by route. Each agent to /{server}/mcp or to a tool set, never to the aggregated endpoint. Stable per agent, so the cache survives.
Sort the catalogue before serving it if you control the deployment, while the gateway does not do it.
/mcp/proxy when the catalogue goes past thirty or fifty tools. It is progressive disclosure and it works with external clients, which is more than you can say for the semantic filter.
The semantic filter, only when the one calling the model is LiteLLM. With a generous top_k, knowing that it embeds only the last user message and that context overflow is a closed and sticky failure.
What not to do: a home-grown top-K filter that rewrites the tools array on every turn. It costs the whole prefix and, according to four of the five measured strategies, probably makes selection worse.

The declared cost, which is worth zero

The calculation has four precedence steps: cost set by a post-call hook, per-tool cost in tool_name_to_cost_per_query, server default cost, and 0.0.

Without configuring mcp_server_cost_info, all MCP traffic is accounted at zero and consumes neither key nor team budget.

There are exactly two Prometheus metrics for MCP:

litellm_mcp_tool_calls_total
litellm_mcp_tool_call_spend_metric

Both with the same eight labels: tool name, server name, key hash, key alias, team, team alias, user and end user.

And the second is only incremented if the cost is greater than zero. With the default configuration, that time series never appears in Prometheus. It is not that it is worth zero: it is that it does not exist.

Nor is there a tools/list metric, or MCP latency, or MCP errors, or upstream server health. For that you are left with the generic request metrics.

In the reports, /spend/calculate has no MCP branch: it accepts a model and messages, or a completion response, and it always calls the LLM cost calculation. The per-team report does include MCP calls, grouped by the model column, which for these rows is worth MCP: tool_name. That is: every tool appears as if it were a model, with zero input and output tokens. Only the sessions endpoint separates the two call types explicitly.

Observability, and the trace that gets cut

There are two OpenTelemetry stacks in LiteLLM and which one runs depends on an environment variable.

Without LITELLM_OTEL_V2=true, the old logger runs, and it has no MCP handling: the only appearance of MCP in the whole file is the metadata key in a list. The tool call comes out as a generic litellm_request span with the model set to MCP: name.

With v2 enabled there is a dedicated span, with a dedicated role, client kind and a parent in the proxy’s request span. The span name is tools/call tool-name. The attributes are correct and follow the conventions:

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`mcp.method.name`	`tools/call`
`mcp.session.id`	MCP session identifier
`gen_ai.tool.name`	Tool name
`gen_ai.tool.call.arguments`	Only with content capture enabled
`gen_ai.tool.call.result`	Only with content capture enabled
`server.address`, `server.port`	From the upstream URL, redacted
`litellm.cost.total`	The cost that is probably worth zero

Content capture is off by default. Without it you see neither the arguments nor the result in the trace.

A note for anyone working with semantic conventions: mcp.tool.name does not exist. The MCP registry has exactly four attributes (mcp.method.name, mcp.session.id, mcp.protocol.version and mcp.resource.uri), all four in development, and the tool name goes in gen_ai.tool.name. LiteLLM gets it right. And the GenAI conventions were extracted into their own repository which as of today has no release at all, so there is no version number to cite.

In Langfuse, the vendor mapper returns an empty dictionary for any span that is not an LLM call. An MCP call appears as a raw span with its attributes, with no native input, output or model fields.

The trace is cut at the gateway

Incoming context is read. The traceparent the client puts in params._meta, following the specification’s propagation proposal, is extracted and turned into a link on the span, never into its parent. The reason is documented in the code: parenting to the remote trace would leave the span hanging off a trace whose root never reaches the backend. The client’s baggage is discarded on purpose to avoid forging identity attributes.

Outgoing context is not propagated. There is no propagator injection call anywhere in the tree. The headers that go out towards the upstream MCP server are built with authentication and the authorised extra headers, and nothing else.

The operational consequence: there is no end-to-end trace. If the upstream MCP server is instrumented, its spans live in another trace and there is no way to correlate them automatically.

Auditing and the conflict with privacy

A tool call leaves a normal spend row, with call type call_mcp_tool, spend equal to the cost per query, model MCP: name, zero tokens in the three token columns, a dedicated mcp_namespaced_tool_name column, and the usual key, team, user and session fields.

And in the metadata column, in JSON, the complete structure of the call: name, arguments, result, server, namespaced name, MCP session identifier, authentication mode and server resource with the URL redacted.

The problem

That structure is written into the metadata column without passing through the privacy switch. The assignment is direct and is not conditioned by the function that governs whether prompts and responses are stored, which does control the messages and response columns.

And the redaction function does not cover it either: it only touches messages and response. A search for MCP in the redaction file returns nothing.

The practical result: an operator who enables turn_off_message_logging for GDPR keeps sending the full arguments of every tool to every logging destination they have configured. To the SIEM, to Langfuse, to S3.

If an argument carries a national ID number, an address, a case number or a medical record, it has been exported.

There is an aggravating factor that does not depend on LiteLLM. The revision of the specification in force introduces an extension that allows mirroring tool arguments into HTTP headers visible to load balancers, proxies and web application firewalls. The specification recommends not marking sensitive parameters that way, and it is a recommendation, and the upstream server decides.

The fit with the ENS

The conflict is real and has a familiar shape: activity logging (op.exp.8) requires the call to be traced and non-repudiable; protection of personal data (mp.info.1) requires that log not to become a secondary repository of personal data with a different purpose. The tool argument is simultaneously the evidence of the action and the data. Redacting it destroys non-repudiation; storing it in the clear creates the repository.

The defensible way out you can build today: a hash of the argument in the audit log, the argument in the clear with encryption (mp.info.3), separate retention and its own access control. The hash sustains non-repudiation and the clear text is only opened under procedure.

There are two other measures this material touches directly. Change management (op.exp.5) because registering an MCP server and, above all, changing the description of an already approved tool are changes to the security configuration. And maintenance and updates (op.exp.4) because, as you can see in the next section, an old version of the gateway is a non-compliance with a name and a number.

Two honesty warnings. The codes follow the numbering of Annex II of RD 311/2022 and you have to check them against the text in force before citing them in a compliance document. And on NIS2: as of the date of this post the directive is still not transposed in Spain, with the draft bill on cybersecurity coordination and governance going through the process and a reasoned opinion from the Commission over the delay. What is enforceable today in the public sector and its suppliers is the ENS.

What is not logged

An administrative gap: registering, modifying and deleting MCP servers generates no audit log. There are three TODO comments in the management code that say so, one of them with an if is_audit_logging_enabled(): pass. In the same deletion block there are two more TODOs about not cleaning up orphaned permissions on keys and teams.

Security

The vulnerability you need to know about

CVE-2026-42271, published on 8 May 2026. Affects LiteLLM from 1.74.2 up to versions prior to 1.83.7. CVSS 3.1 of 8.8.

The POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list endpoints, which exist to preview an MCP server before saving it, accepted the stdio transport’s command, args and env in the request body. A low-privilege key executed arbitrary commands on the proxy host. It also affects Red Hat OpenShift AI on several branches.

It is the story of this post condensed into one line: the gateway you put in to contain MCP was the hole.

It does not come alone. The 1.83 branch concentrates three more serious ones:

CVE	Component	CVSS	Fixed in
`CVE-2026-42208`	SQL injection in key validation, with no prior authentication	9.8	1.83.7
`CVE-2026-35030`	OIDC userinfo cache keyed on `token[:20]`, collision and impersonation	9.1	1.83.0
`CVE-2026-40217`	Remote execution through bytecode rewriting in the guardrails test endpoint	8.8	no version recorded

Plus two privilege escalations fixed in 1.83.10 and 1.83.14.

The operational conclusion is short: a deployment below 1.83.14 is exposed to remote execution or to an unauthenticated SQL injection. The current branch is 1.102.0.

And a note for context: the whole MCP ecosystem has the same pattern. The unauthenticated remote execution in MCP Inspector, the command injection in mcp-remote when connecting to an untrusted server, DNS rebinding disabled by default in the Python and TypeScript SDKs until late 2025, the directory escapes in the reference servers. Connecting to an untrusted MCP server compromises the client, not just the other way round.

The rug pull nobody covers

Documented since April 2025: a server can change a tool’s description after the client has approved it. The description is what the model reads to decide when and how to use it, so changing it is changing the behaviour without asking for permission again.

The specification in force does not address it. It defines list change notification and cache freshness fields, which are recency mechanisms, not integrity ones. The closest normative warning is that clients must treat tool annotations as untrusted unless they come from trusted servers.

LiteLLM does not cover it either. The only “pinning” in the code is of the configuration’s server identifier, so that it does not change when you edit the YAML. There is no hash or comparison of the description or the input schema between successive listings. A search for related terms in the MCP module: nothing functional.

The control exists outside the gateway. mcp-scan, from the same researchers who documented the attack, implements description hash pinning precisely to detect this.

In ENS terms it is an unauthorised change to the security configuration that today goes undetected. And the home-made fix fits in a cron job: store the hash of the description and schema of every approved tool, compare against tools/list periodically, alert on the difference.

Three more holes in the code

No URL validation towards the upstream. A new server’s url field has no validator. The SSRF protection utilities exist in the tree and are used in OAuth discovery and in downloading OpenAPI specifications, but not in the egress of tools/call. A server pointing at a private range or at the cloud metadata IP is accepted as is.

The stdio transport is local execution on the proxy host. There is a command allowlist: npx, uvx, python, python3, node, docker, deno, extendable through an environment variable. The code’s own comment admits the residual risk: the permitted runtimes can execute code through arguments. With docker and npx in there, it is de facto arbitrary execution for any proxy administrator.

Two server modes skip the gateway’s authentication. Pass-through and authentication delegation to the upstream completely omit virtual key validation. They are well hardened (fail-closed if the target is mixed or unresolvable, strict comparison against truthy values, explicit blocking of the client credentials flow so as not to lend out the proxy’s service account), and the result is still that those servers have no gateway access control.

The limit that truncates in silence

The tools/list pagination stops and returns what it has accumulated in three cases: repeated cursor, one thousand pages, or expiry of a global deadline. The docstring declares it: a slow or faulty upstream produces a partial catalogue instead of an error.

The first two log a warning. The third, the deadline, exits without logging anything at all. A slow server makes part of its tools disappear from the catalogue with no trace left behind.

Correction on the slot leak

In the previous post I took issue 34534 at face value, according to which every MCP tool call acquires a max_parallel_requests slot and does not release it. Tracing the 1.102.0 code, that diagnosis no longer holds: the release path exists, on both success and failure, and the limiter is registered as a litellm callback and not just as a proxy hook.

With one honest caveat: I found no regression test covering the full cycle for MCP, and the definitive confirmation is a real proxy with max_parallel_requests: 2 and five MCP calls in a row. If anybody reproduces it today, I would like to know.

The specification moved underneath

The MCP revision in force is 2026-07-28, published on 28 July 2026, and in between there was another one, 2025-11-25. The changes in the latest one are not cosmetic:

Protocol sessions and the session header are removed. Cross-request state becomes an explicit identifier in the tool arguments.
The initialisation handshake is removed. Every request carries version and capabilities in _meta. MCP becomes stateless.
server/discover appears as a mandatory call.
Elicitation changes shape: out go server-initiated requests, in comes a multi-round pattern where the server returns an “input required” result and the client retries with the answers.
ping, logging/setLevel and the roots change notification are removed. SSE stream resumption is removed.
Standard HTTP headers are mandatory on POST, which lets a gateway or a web application firewall decide without parsing the JSON.
On authorisation: mandatory issuer validation in the client, and dynamic client registration is deprecated in favour of client identifier metadata documents.
Roots, sampling and logging are deprecated, with a minimum window of twelve months.

LiteLLM 1.102.0 advertises 2025-06-18. Its version enumeration has three values and none of them is later than that date. The literal is sent in the initialize against the upstream.

One nuance that is not entirely bad: the version the gateway advertises to its clients is not hard-coded in LiteLLM, the SDK negotiates it. And there is partial implementation of the intermediate revision, because the elicitation and sampling handlers reference it in their docstrings. That is: partial implementation of one revision while advertising an earlier one.

In practical interoperability it does not hurt yet, because the client ecosystem is still mostly on the 2025 revisions. It hurts in planning: July’s stateless redesign invalidates the gateway’s session model, and that is work ahead.

Two ecosystem notes for anyone choosing. There are clients that speak remote streamable HTTP natively (Claude and Claude Code, ChatGPT with OAuth and dynamic registration, VS Code with Copilot, Cursor, Zed, Kiro) and others that still demand stdio or a local bridge with mcp-remote, among them Windsurf, Cline and Continue.

Tool execution from the proxy itself

Besides serving MCP outwards, the proxy can consume it itself. It is enabled with an mcp block in the body, with server_url equal to the litellm_proxy sentinel, and it works both on /v1/responses and on /chat/completions.

The flow: it separates the gateway’s tools, lists them, filters by allowed, deduplicates, translates them into OpenAI function format and concatenates them. It calls the model. If there are tool calls and auto-execution is permitted, it runs them against the MCP servers. It builds the follow-up and makes one second call.

Two limits you need to know before designing on top of this.

The loop is a single hop. There is no multi-step iteration. The calls from the first response are executed and the follow-up is returned. A plan that chains tools requires the client to call again.

Auto-execution fails closed and as a block. It requires all MCP references to have approval set to “never”; a single one requiring approval disables auto-execution for the whole request.

And a third one, which is the one that bites: the multi-turn state of /v1/responses is rehydrated from the spend table. The query is literal, it looks up the session of the previous request_id and retrieves all the rows of that session ordered by time.

With disable_spend_logs: true nothing is written, the query comes out empty and the history is lost in silence, with no 4xx error. The code itself acknowledges it: it does not expect retries because deployments that do not write spend logs have nothing to wait for. It is a coupling between a performance option and a conversation feature that nobody expects to find.

Health and day-to-day operation

Health checks exist, they are on demand and they are not periodic. GET /v1/mcp/server/health accepts repeated identifiers and returns per-server status. The implementation opens a client and runs an empty operation inside the session, with a ten-second timeout.

It skips the check entirely if the server requires per-user authentication, and those always return unknown status.

The surprising part: the status, last check and error columns have existed in the database since a May 2025 migration, and the check function returns an object without writing to the database. Status is recalculated on every request. There is no periodic MCP health job.

When an upstream server is down, the behaviour is to degrade and not block. The listing fans out in parallel and each failing server contributes an empty list plus a result classified into six categories. Those results reach the client in the response’s _meta under a vendor key, without filtering the upstream’s text or URL. The reason for the design is in the code: an empty contribution with no signal makes a broken upstream indistinguishable from a healthy server with no tools.

There is no cache of the previous list. A server that is down disappears from that turn’s tools.

On registering and deregistering without restarting, there are two worlds. Database servers can be created, edited and deleted through the API, and every operation reloads the registry with an atomic swap. Propagation between replicas goes through Redis publication on the configuration change channel, if there is Redis, and through polling every 30 seconds if there is not.

Servers declared in config.yaml are loaded once, at start-up. There is no re-read. Changing them requires a restart.

And there is a precedence trap between the two worlds, with two different warnings in the code. A database row with the same identifier as a configuration entry hides the configuration one entirely. And the other way round, a configuration identifier that matches the name or alias of a database server captures its permissions, because identifiers are resolved before names. Both cases emit a log warning, once.

A hardened `config.yaml`

general_settings:
 # Access is not open by default, but this closes the empty-key
 # inheritance towards the team, which is the one that surprises people.
 require_key_mcp_access_defined: true
 require_end_user_mcp_access_defined: true
 # Without this, allow_all_keys overrides the key's explicit scope.
 mcp_allow_all_keys_respects_mcp_scope: true

litellm_settings:
 # Does not cover MCP arguments. See the auditing section.
 turn_off_message_logging: true
 callbacks: ["langfuse_otel"]

mcp_servers:
 inventario:
 url: https://mcp-inventario.interna.svc/mcp
 transport: http
 auth_type: bearer_token
 auth_value: os.environ/MCP_INVENTARIO_TOKEN
 # Explicit allowlist. Also enforced on tools/call.
 allowed_tools:
 - buscar_articulo
 - consultar_stock
 # Never true on a server with data.
 allow_all_keys: false
 access_groups: ["operaciones"]
 timeout: 20
 max_concurrent_requests: 8
 mcp_info:
 description: "Inventario de almacen, solo lectura"
 # Without this, MCP spend is zero and its metric does not exist.
 mcp_server_cost_info:
 default_cost_per_query: 0.0005
 tool_name_to_cost_per_query:
 consultar_stock: 0.0002

And in the process environment:

# Without this there is no MCP span, only a generic litellm_request.
LITELLM_OTEL_V2=true
# Full block hashes, not truncated, if correlating
# with engine events.
LITELLM_MCP_CLIENT_TIMEOUT=45
LITELLM_MCP_TOOL_LISTING_TIMEOUT=20

For clients, the rule is not to use the aggregated endpoint. Each agent points at /{server}/mcp or at a tool set, or at the three-tool proxy mode if the catalogue is large.

Checklist

Check the version. Below 1.83.14 there is remote execution and an unauthenticated SQL injection with an assigned CVE.
Inventory which servers have allow_all_keys and enable the setting that makes it respect the key’s scope.
Audit the access groups separately, because they are additive on top of the key and team ceiling.
Declare in writing the fail-open behaviour on tool permission resolution errors.
Measure the real cost: number of tools per server from the listing row, and tools block tokens from the stored request body.
Choose a catalogue strategy before connecting the third server: three-tool proxy mode, semantic filter, or selection by route. Not the aggregated endpoint.
Declare mcp_server_cost_info even with a token value, or the spend metric will not exist.
Enable the OTel v2 logger, and decide deliberately whether to enable content capture.
Assume the trace is cut at the gateway and plan correlation with the upstream by other means.
Treat the spend table’s metadata column as a personal data repository: encryption, its own retention and separate access control.
Set up the rug pull check, because nobody does it: hash of the description and schema of every approved tool, periodic comparison against the listing.
Restrict egress towards MCP servers at the network level, because the gateway does not validate the destination URL.
Do not enable the stdio transport unless you need it, and if you do, review what is on the command allowlist.
If you use /v1/responses with multi-turn state, do not enable disable_spend_logs.

Traps and things that are not what they seem

MCP access is not open by default in 1.102.0. I got that wrong in the previous post. What opens things up is per-server allow_all_keys.
allow_all_keys overrides the key’s explicit scope unless you enable a setting that ships disabled.
require_key_mcp_access_defined only closes key-to-team inheritance. Access group grants remain additive.
On an indeterminate failure resolving tool permissions, the code fails open for keys and JWTs.
The default cost of an MCP call is 0.0, and the spend metric is only emitted if the cost is greater than zero, so the series does not exist.
MCP calls appear in the per-team report as if they were models, named MCP: tool and with zero tokens.
The OTel logger with MCP support is off by default. Without the environment variable there is no MCP span.
Trace context is not propagated towards the upstream MCP server. There is no end-to-end trace.
mcp.tool.name does not exist in the semantic conventions. The name goes in gen_ai.tool.name.
Message redaction does not cover tool arguments. They are written in the clear into the spend table.
Registering, modifying and deleting MCP servers generates no audit log. There are three TODOs in the code.
The semantic tool filter is not applied on tools/list. Only on /chat/completions and /v1/responses.
The catalogue order is not deterministic, because it comes from iterating a set of strings. It breaks the client’s prompt cache even though nobody filters anything.
search_tools in proxy mode counts keywords by default, it uses neither regular expressions nor embeddings, and its cap of five results is hard-coded.
Description overrides really do replace what the client sees, and they are the only catalogue lever that does not break the cache. They are only configurable through the database or the API.
spec_version no longer exists in the configuration schema.
/mcp/sse does not use the SSE transport, and the messages endpoint its object advertises is dead code.
Listing pagination truncates in silence when the deadline expires, without leaving a single log line.
A database row hides a configuration entry with the same identifier, and the other way round a configuration identifier can capture the permissions of a database server.
Servers declared in config.yaml require a restart. Only the database ones reload hot.
The health check does not write its result to the database even though the columns exist, and there is no periodic job.
The gateway does not validate the upstream MCP server’s URL. The anti-SSRF utilities exist and are not used on that path.
docker and npx are on the stdio command allowlist.
The /v1/responses tool loop is a single hop, and disable_spend_logs: true breaks its multi-turn behaviour with no error.
LiteLLM advertises the June 2025 revision of the protocol, two behind the one in force.

Closing

There is a thread running through the six posts in this track that is more visible here than anywhere else. The gateway is chosen for one job and ends up doing five, and each new job arrives with its own defaults, which are almost never the ones you would have chosen.

In the case of MCP the asymmetry is twofold. On one side, the part that looks dangerous is better built than you would assume: access control is enforced on execution and not just on the listing, the hierarchy is built out of intersections, header forwarding is an allowlist, and default access is closed, contrary to what I myself wrote two days ago. On the other, the part nobody looks at is worse: tool arguments come out in the clear bypassing the privacy switch, spend is worth zero and its metric never gets to exist, the trace is cut, and a change to the description of an approved tool goes undetected by anybody in the whole chain, neither the specification nor the gateway.

The architecture decision that comes out of this is that the MCP door has to be treated as what it is, an interconnection of systems with third parties, and not as one more function of the LLM proxy. That means its own inventory, formal registration, change control over every tool’s description, network isolation towards the upstream and a log designed knowing that it is going to contain personal data.

And it means accepting that the cost you really pay for connecting fifteen MCP servers is in no column of the spend table. It is spread across the prompt_tokens of every model call, on every turn, and across the accuracy you lose when the catalogue is too large for the model to choose well. That number can be computed with what the gateway already stores. The first step is wanting to compute it.

Sources

LiteLLM, MCP Gateway: https://docs.litellm.ai/docs/mcp.
LiteLLM, código: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, auth/user_api_key_auth_mcp.py, cost_calculator.py, faults/list_outcomes.py, litellm/proxy/management_endpoints/mcp_management_endpoints.py, litellm/proxy/spend_tracking/spend_tracking_utils.py, litellm/litellm_core_utils/redact_messages.py, litellm/integrations/otel/, litellm/responses/litellm_completion_transformation/session_handler.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en llamadas MCP: https://github.com/BerriAI/litellm/issues/34534.
NVD, CVE-2026-42271, ejecución de comandos por los endpoints de prueba de MCP: https://nvd.nist.gov/vuln/detail/CVE-2026-42271.
NVD, CVE-2026-42208, inyección SQL sin autenticación en la validación de clave: https://nvd.nist.gov/vuln/detail/CVE-2026-42208.
NVD, CVE-2026-35030, colisión de clave de caché de userinfo OIDC: https://nvd.nist.gov/vuln/detail/CVE-2026-35030.
NVD, CVE-2025-49596, ejecución remota en MCP Inspector: https://nvd.nist.gov/vuln/detail/CVE-2025-49596.
NVD, CVE-2025-6514, inyección de comandos en mcp-remote: https://nvd.nist.gov/vuln/detail/CVE-2025-6514.
NVD, CVE-2025-66416, protección de DNS rebinding desactivada por defecto en el SDK de Python: https://nvd.nist.gov/vuln/detail/CVE-2025-66416.
Invariant Labs, MCP Security Notification: Tool Poisoning Attacks (tool poisoning, rug pull, shadowing): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks.
Invariant Labs, Introducing mcp-scan (fijado de herramientas por hash): https://invariantlabs.ai/blog/introducing-mcp-scan.
Model Context Protocol, changelog de la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/changelog.
Model Context Protocol, changelog de la revisión 2025-11-25: https://modelcontextprotocol.io/specification/2025-11-25/changelog.
Model Context Protocol, Security Best Practices (confused deputy, indicadores de recurso, validación de audiencia): https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.
NSA, Model Context Protocol: Security Design Considerations for AI-Driven Automation, mayo de 2026: https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDF.
CISA y agencias aliadas, Careful Adoption of Agentic AI Services, mayo de 2026: https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services.
Anthropic, Advanced tool use (coste en tokens de las definiciones, búsqueda de herramientas, precisión de selección): https://www.anthropic.com/engineering/advanced-tool-use.
Anthropic, Code execution with MCP: https://www.anthropic.com/engineering/code-execution-with-mcp.
Anthropic, Prompt caching (jerarquía tools → system → messages e invalidación): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.
Anthropic, Tool search tool (herramientas diferidas fuera del prefijo, umbral de 30 a 50 herramientas): https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, Prompt caching (el orden de las herramientas cuenta): https://developers.openai.com/api/docs/guides/prompt-caching.
OpenAI, Tool search (las herramientas descubiertas se cargan al final del contexto): https://developers.openai.com/api/docs/guides/tools-tool-search.
vLLM, Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html.
Microsoft, Use tools in chat (límite de 128 herramientas por petición en VS Code): https://code.visualstudio.com/docs/copilot/agents/agent-tools.
RAG-MCP, prueba de estrés de 1 a 11.100 servidores candidatos, mayo de 2025: https://arxiv.org/abs/2505.03275.
Benchmarking the Benchmarks, desalineación del 18,5 % entre etiqueta y realidad, junio de 2026: https://arxiv.org/html/2607.02577v1.
MCP-Atlas, taxonomía de fallos sobre 220 herramientas y veinte modelos, febrero de 2026: https://arxiv.org/html/2602.00933v3.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307.
MCP Tool Descriptions Are Smelly, auditoría de 856 herramientas de 103 servidores, febrero de 2026: https://arxiv.org/html/2602.14878v1.
ToolMenuBench, seis estrategias de filtrado y 26.460 ejecuciones, junio de 2026: https://arxiv.org/html/2606.15508.
Retrieval Models Aren’t Tool-Savvy (ToolRet), corpus de 43.000 herramientas, marzo de 2025: https://arxiv.org/abs/2503.01763.
Tools Are Not Islands, brecha entre Recall@3 y conjunto completo, julio de 2026: https://arxiv.org/html/2607.25718.
ToolFlood, saturación adversaria de la capa de recuperación, marzo de 2026: https://arxiv.org/html/2603.13950.
Scores Are Not Decisions, el K óptimo como función del coste, julio de 2026: https://arxiv.org/html/2607.27083v1.
Don’t Break the Cache, caché de prompts en tareas agénticas de horizonte largo, enero de 2026: https://arxiv.org/html/2601.06007v2.
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer, mayo de 2026: https://arxiv.org/html/2605.24660v1.
Semantic Tool Discovery for LLMs, marzo de 2026: https://arxiv.org/abs/2603.20313.
OpenTelemetry, convenciones semánticas de GenAI y MCP: https://github.com/open-telemetry/semantic-conventions-genai.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
IBM ContextForge, gateway y registro MCP: https://github.com/IBM/mcp-context-forge.
Docker MCP Gateway: https://github.com/docker/mcp-gateway.
agentgateway: https://github.com/agentgateway/agentgateway.

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

Thu, 04 Jun 2026 10:00:00 +0200

This post goes deeper into the agentic reasoning mechanism that extends the RAG described in RAG with reranker and hybrid retrieval. The retriever invoked when the LLM chooses vector_search is exactly the pipeline in that article. The JSON Schema that defines each tool call is structured output applied to the tool interface. And the agent’s requests pass through the L7 inference gateway before reaching the model.

TL;DR

A naive RAG always queries the same source. Function calling gives the LLM the ability to decide which tool to invoke (vector store, SQL, web search) according to what the query actually needs. The ReAct pattern chains those invocations in a reasoned loop until there is enough evidence. A 3-iteration pipeline with Llama-3.1-70B on on-premise hardware takes ≈ 1.1 s against the ≈ 300 ms of a single-step RAG; the gain is not in speed but in queries that a naive RAG simply cannot answer. The critical eval metric is tool selection accuracy: the percentage of turns in which the model picks the right tool, measured over a synthetic eval set.

The analogy: the detective who knows which file to ask for

A detective in a novel does not go to the same filing cabinet regardless of the case that lands on his desk. When he takes a case, he reasons first: what kind of evidence do I need? If there are witnesses, he asks for the statements (vector search over unstructured documents). If there are financial transactions, he asks the bank for the accounting records (SQL over the structured database). If the suspect has recent activity that the company cannot possibly have indexed, he goes to the press archive (web search). He does not consult all three sources at once on every case: he picks the one the evidence requires, receives the result, reasons again if he needs more, and only writes the report when he has enough evidence.

A bad detective always goes to the same filing cabinet. A naive RAG is that bad detective: it vectorises the query, goes to the vector store, and returns whatever it finds even if the question was “how many contracts?”, something no PDF chunk can answer better than a COUNT(*) in SQL.

Function calling is giving the LLM the ability to reason about which source to ask for, and to invoke it in a structured way. The analogy has three edges worth keeping in mind:

The detective does not improvise the file he asks for: there is a catalogue of available sources with a description of what each one contains. The tool description in the system prompt does that job.
The detective can ask for several pieces of evidence at once if they are independent (parallel tool calling).
The detective knows when to stop: if after N rounds he reaches no conclusion, he declares that he does not have enough evidence. The agent has an iteration limit for the same reason.

What function calling is: the anatomy of a tool call

Function calling, also called tool use, is a mechanism by which the LLM, instead of generating free text as its answer, generates a structured JSON object representing a tool invocation. The system intercepts that JSON, executes the real tool, and returns the result as a message with the tool role in the conversation.

Defining tools in the system prompt

Each tool is defined through a JSON Schema that specifies name, description and parameters. This JSON Schema is exactly the same mechanism described in structured output, applied here to the tool interface:

{
 "tools": [
 {
 "name": "vector_search",
 "description": "Search internal company documents about policies, contracts and procedures. Use when the query requires unstructured text, document context or semantic similarity.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Natural language search query"},
 "top_k": {"type": "integer", "default": 5}
 },
 "required": ["query"]
 }
 },
 {
 "name": "sql_query",
 "description": "Query the SQL database for structured metrics, counts, aggregations and financial data. Use when the query requires exact numbers, filters, sums or joins over structured records.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Parameterized SQL query with $1, $2 placeholders"},
 "params": {"type": "array", "items": {}, "description": "Parameter values for the placeholders"}
 },
 "required": ["query"]
 }
 },
 {
 "name": "web_search",
 "description": "Search public web for real-time information, recent news or current prices. Use only when data is public and not covered by internal sources.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string"}
 },
 "required": ["query"]
 }
 }
 ]
}

The life cycle of a tool call

When the LLM decides to invoke a tool, the message it generates in place of free text has this structure (OpenAI-compatible format, the same one vLLM supports):

{
 "role": "assistant",
 "tool_calls": [
 {
 "id": "call_01",
 "type": "function",
 "function": {
 "name": "sql_query",
 "arguments": "{\"query\": \"SELECT COUNT(*), SUM(amount) FROM contracts WHERE amount > $1 AND year = $2 AND provider_region = $3\", \"params\": [100000, 2025, \"EU\"]}"
 }
 }
 ]
}

The system executes the tool and returns:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"count\": 47, \"total\": 8300000}"}

The LLM receives that message as a continuation of the conversation and decides whether it needs more information or can generate the final answer.

Support in OSS models

In 2026, native function calling support (not emulated via system prompt) is available in:

Llama 3.1 / 3.3+: native tool call format, supported in vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json
Qwen 2.5+: native support with --tool-call-parser hermes
Mistral NeMo / Mistral 7B Instruct v0.3+: native format with --tool-call-parser mistral
Qwen3 (2025+): extended native support with parallel tool calling

Without --enable-auto-tool-choice, vLLM can force the tool call format via guided decoding (JSON Schema), but the ability to decide when to invoke a tool versus generating free text requires specific model training.

The ReAct pattern: Reasoning + Acting

ReAct (Yao et al., 2022) is the loop that turns an LLM with tools into an agent with reasoning. The central idea is that the model interleaves thoughts (explicit reasoning about what to do) with actions (tool invocations) and observations (results received). This loop repeats until the model reckons it has enough information to answer, or until the iteration limit is reached.

Thought → Action → Observation → Thought → Action → Observation → ... → Response

A full ReAct trace: a real production query

Query: “How many contracts above 100k€ did we sign in 2025 with EU providers, and which special conditions recur?”

This query requires two different sources: the exact numbers live in SQL, and the context of special conditions lives in the contract PDFs indexed in the vector store.

Iteration 1

Thought: The query asks for aggregated data (count and sum) with exact filters over structured fields (amount, year, region). This is a SQL query, not a semantic search.

Action: sql_query

SELECT COUNT(*) AS num_contratos, SUM(amount) AS total_eur
FROM contracts
WHERE amount > $1 AND year = $2 AND provider_region = $3

params: [100000, 2025, "EU"]

Observation: {"num_contratos": 47, "total_eur": 8300000}

Iteration 2

Thought: I have the numbers. Now I need the special conditions that recur in these contracts. That kind of information is in the free text of the PDFs, not in the structured table. I go to the vector store.

Action: vector_search

"EU provider contracts 2025 amount above 100k special conditions recurring clauses"

top_k: 5

Observation:

[chunk_1] Provider A contract — clause 8.3: late-delivery penalty of 2% per month on the invoiced amount...
[chunk_2] Provider C contract — clause 7.1: price revision linked to the euro area CPI capped at 4% per year...
[chunk_3] Provider B contract — clause 8.3: late-delivery penalty of 2% per month...

Thought: I have enough data from both sources. Clause 8.3 on late-delivery penalties appears in at least two of the retrieved contracts. I can synthesise the answer.

Response: “In 2025 you signed 47 contracts with EU providers above 100,000 €, for a cumulative total of 8.3 M€. The special condition that appears most often in the higher-value contracts is the late-delivery penalty clause (8.3 in the contract templates of Provider A and B): 2% per month on the invoiced amount. The price revision linked to the euro area CPI, capped at 4% per year (clause 7.1), also recurs.”

This trace has 2 iterations. A more complex query (comparing with the previous year, cross-referencing with payment status) would add further iterations. Common practice is to set a limit of 5–10 iterations maximum to avoid loops.

The three sources and when to use each one

The choice of source is neither arbitrary nor left solely to the LLM: the architecture defines which tools exist and how they are described. The table below summarises the selection criteria:

Characteristic	Vector store (Qdrant)	SQL / structured (PostgreSQL)	Web search
Data type	Free text, documents, PDFs	Tables with a fixed schema	Public pages, news
Structure	Unstructured	Highly structured	Semi-structured
Temporality	Static index (periodic refresh)	Real time (transactional)	Real time (crawl)
Typical latency	5–50 ms	10–200 ms	200–2,000 ms
Privacy	Internal data, full sovereignty	Internal data, full sovereignty	Public data only
Natural query	Yes (natural language → embedding)	No (parameterised SQL)	Yes
Exact aggregations	No	Yes (`COUNT`, `SUM`, `GROUP BY`)	No
When to use	Document context, semantics, PDFs	Metrics, counts, exact filters, joins	Data that does not exist internally and is public

The most important rule of thumb: if the question contains words like “how many”, “sum”, “total”, “more than X”, “in year Y” and the data is in a structured table, the right answer is sql_query. If the question asks for context, explanations, clauses, procedures or examples from documents, the answer is vector_search. If it asks for the current price of something or recent news about a third party, web_search, but only if no data sovereignty is involved.

Tool routing: how the LLM picks the right tool

The description of each tool in the system prompt is the most critical factor for routing accuracy. An LLM with good function calling capabilities can still choose badly if the descriptions are ambiguous or overlap.

Descriptions that work versus those that do not

Weak description (leads the LLM to use the wrong tool):

"search_docs" — Searches for information in the available sources.
"query_data" — Gets data from the system.

Strong description (delimits precisely when to use each one):

"vector_search" — Search internal company documents about policies, contracts and procedures.
Use when the query requires unstructured text, document context or semantic
similarity. NOT for counts, sums or exact filters.
"sql_query" — Query the SQL database for structured metrics, counts, aggregations and
financial data. Use when the query requires exact numbers, filters, sums or
joins over structured records. NOT for finding document context.

The difference lies in two elements: (1) positive use-case examples, and (2) explicit exclusions with NOT for. Both reduce the semantic overlap between tools and improve tool selection accuracy.

Parallel tool calling

When two tools are independent of each other, that is, when the result of one does not affect the query of the other, the LLM can invoke them simultaneously in the same turn:

{
 "tool_calls": [
 {"id": "call_01", "function": {"name": "sql_query", "arguments": "..."}},
 {"id": "call_02", "function": {"name": "vector_search", "arguments": "..."}}
 ]
}

The system executes both in parallel and returns both observations before the LLM’s next turn. This cuts total latency when the queries are independent: instead of 2 sequential iterations (2 × tool_latency), you pay 1 × max(sql_latency, vector_latency). For the detective example: if he needs both the accounting records and the witness statements to answer, he can ask for them at the same time.

Tool selection accuracy: the eval metric

Tool selection accuracy is the percentage of turns in which the LLM picks the right tool given a set of evaluated queries:

\[ \text{TSA} = \frac{\text{turns with correct tool chosen}}{\text{total turns with expected tool call}} \]

It is measured over a synthetic eval set built with (query, expected_tool, expected_args) triples. An example of a minimal eval set:

Query	Expected tool	Failure indicator
“How many orders in March?”	`sql_query`	LLM uses `vector_search`
“What does the returns policy say?”	`vector_search`	LLM uses `sql_query`
“What is the price of copper today?”	`web_search`	LLM uses `vector_search`
“Sum the Q3 contracts”	`sql_query`	LLM uses `vector_search`

A TSA < 0.85 in a production agent is a sign that the tool descriptions need revising before the model does. For more detail on how to build these evals, see LLM evals.

Error handling in tool calls

SQL injection via prompt

The most serious risk in tool-augmented retrieval is that the LLM generates malicious SQL, either because a user induced it via prompt injection, or because the model hallucinated a destructive query. This attack vector is covered in detail in LLM Guard, but the minimum rules on the tool side are:

Rule 1: parameterised queries always, never direct interpolation.

# NEVER this:
cursor.execute(f"SELECT * FROM contracts WHERE provider = '{llm_output}'")

# Always this:
cursor.execute("SELECT * FROM contracts WHERE provider = $1", [llm_output])

Rule 2: DB user with minimum permissions. The user the agent runs SQL as must have SELECT on the necessary tables and nothing else. No DROP, INSERT, UPDATE or DELETE.

Rule 3: table allowlist. The system validates that the query generated by the LLM only references tables on an allowlist before executing it.

Rule 4: per-query timeout. Queries that block for too long, potentially induced to DoS the database, are cancelled with a configured timeout.

Rate limits, timeouts and tool errors

When a tool fails, the error is returned to the LLM as an observation:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"error\": \"timeout after 5s\", \"tool\": \"web_search\"}"}

The system prompt must instruct the LLM on what to do in this case:

If a tool returns an error or is unavailable, acknowledge the limitation in your response.
Do not retry more than once. If web_search is unavailable, state that real-time data
is not accessible at this moment and answer with available internal sources only.

This stops the agent from entering retry loops and handles graceful degradation: if web_search is unavailable, it answers with what it has in the internal sources.

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Each iteration of the ReAct loop has three latency components:

\[ T_{\text{iter}} = \text{TTFT}_{\text{LLM}} + T_{\text{tool}} + \Delta_{\text{context}} \]

where:

TTFT_LLM: time to the LLM’s first token (dominated by the prefill of the accumulated context)
T_tool: tool execution time
Δ_context: overhead of the growing context window (each iteration adds the previous output to the context)

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

With Llama-3.1-70B in FP8 on a node with 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s), the typical values in production are:

Component	Value
TTFT with context < 4k tokens	≈ 150 ms
TTFT with context 8k tokens	≈ 220 ms
`sql_query` (simple query, indexed)	≈ 50 ms
`vector_search` (top-5, Qdrant in RAM)	≈ 20 ms
`web_search` (external API)	≈ 600 ms

Total latency by number of iterations

1-iteration pipeline (simple query, a single tool):

\[ T_1 = 150 + 50 = 200 \text{ ms} + \text{final synthesis} \approx 200 + 300 = 500 \text{ ms} \]

2-iteration pipeline (SQL + vector_search in sequence):

\[ T_2 = (150 + 50) + (180 + 20) + 400 = 800 \text{ ms} \]

The context in the second iteration already includes the result of the first, so TTFT rises slightly to ≈ 180 ms.

3-iteration pipeline (the most common case in complex queries):

\[ T_3 = (150 + 50) + (180 + 20) + (200 + 50) + 450 \approx 1,100 \text{ ms} \]

Parallel tool calling (SQL + vector_search in parallel, a single iteration):

\[ T_{\text{parallel}} = 150 + \max(50, 20) + 400 = 600 \text{ ms} \]

When the two queries are independent, parallel tool calling trims latency from ≈ 800 ms to ≈ 600 ms: a 25% improvement over the 2 sequential iterations case.

Comparison with a naive RAG

Configuration	Latency	Queries it can answer
Naive RAG (1 retriever, 1 step)	≈ 300 ms	Document context queries
ReAct 1 iteration (SQL)	≈ 500 ms	Structured aggregation queries
ReAct 2 iterations (SQL + vector)	≈ 800 ms	Hybrid numeric + context queries
ReAct 3 iterations	≈ 1,100 ms	Complex multi-source queries
ReAct with web_search	≈ 1,500 ms	Queries requiring real-time data

The practical conclusion: multi-hop ReAct is 3–5× slower than a single-step naive RAG. The gain is not in speed but in query coverage: a naive RAG cannot answer “how many contracts?” because that answer is in no text chunk at all. For applications with a strict latency SLO (< 500 ms), you have to work out whether the use case really needs ReAct or whether a well-configured RAG with hybrid retrieval covers 90% of the queries.

On-premise hardware for ReAct agents

A ReAct agent with Llama-3.1-70B in production has different requirements from a naive RAG, because the context grows with each iteration and prefill throughput is more critical.

Recommended configuration: 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s)

Llama-3.1-70B in FP8: fits in 2×H100 (70B params × 1 byte/param ≈ 70 GB + KV cache). With 4×H100 you can serve it with tensor parallelism TP=4, cutting prefill TTFT by ≈ 2×.
Qdrant instance: it can sit on the same node (if the collection fits in RAM) or on a dedicated node. For collections < 50M vectors of 768 dims: ≈ 150 GB, which fits in the RAM of a dual-socket server.
PostgreSQL: separate node or managed instance. The agent adds no unusual load to SQL; the queries are simple and bounded by timeout.
vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json --max-model-len 16384: a 16k token context comfortably covers the 5–10 turns of a ReAct pipeline.

Minimum viable configuration: 2×H100 SXM (160 GB)

Llama-3.1-70B in FP8 with TP=2. TTFT ≈ 250–300 ms for 4k token contexts (a 60–100% increase over TP=4).
Good enough for internal workloads with < 20 concurrent requests.
Not advisable for an SLO < 1 s with more than 5 concurrent users and a long context.

What we have not covered

Agentic retrieval loops with planning. ReAct is the simplest agent pattern. When a query requires decomposition into sub-tasks with dependencies, you need orchestration frameworks such as LangGraph (state graphs), smolagents (Hugging Face, agents with Python code as actions) or llama-index Agents (planning + retrieval pipeline). These introduce a planning step before the execution loop.

MCP (Model Context Protocol). Anthropic’s emerging standard, with OSS implementations, for defining tools portably across frameworks and hosts. Instead of defining each tool’s JSON Schema separately in every application, MCP centralises those definitions in an MCP server that any compatible client can discover and invoke. Adoption in 2025–2026 is fast among OSS frameworks (LangChain, smolagents, OpenWebUI).

Tool caching. If the same tool call (same arguments, same tool) is going to be invoked several times within the same context or in very similar contexts, the result can be cached. The mechanism is analogous to the semantic cache described for RAG: before executing the tool, the hash of the arguments (or its embedding, for semantic matching) is compared against a cache with a TTL. Especially valuable for sql_query with frequent queries and data that changes little.

Multi-agent. When an orchestrating agent delegates sub-tasks to specialised agents, one for SQL, one for document retrieval, one for code generation, you enter multi-agent system territory. Each sub-agent can have its own tool set and its own LLM (possibly smaller and more specialised). Coordination between agents introduces additional tracing and observability complexity.

References

Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
vLLM documentation. Tool calling. https://docs.vllm.ai/en/stable/features/tool_calling.html
Qdrant documentation. Search. https://qdrant.tech/documentation/concepts/search/
OpenAI. Function calling. https://platform.openai.com/docs/guides/function-calling
Meta AI. Llama 3.1 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
Qwen Team (Alibaba). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115
Anthropic. Model Context Protocol. https://modelcontextprotocol.io
OWASP. LLM Top 10 for Large Language Model Applications. LLM01: Prompt Injection. https://owasp.org/www-project-top-10-for-large-language-model-applications/

Tool-Use on lo0 — Tech Blog

Making the model prefer our tools: the priority that exists in no layer, the two primitives that do, and the `_meta` that crosses the gateway

TL;DR

You are here: the question that arrives after the third server

The analogy: the counter with two trays

What the specification says: nothing

The priority that exists and is no use

The only blessed filtering is by credential

Discovery does not discover tools

What there is in proposals

The layer that decides: the client

Claude Code

The other clients

The API layer

The test: the _meta crosses the gateway

The setup

The result

The three routes where it is lost

The library trap

What we have not tested

What LiteLLM can and cannot do

The cheap lever: the description, not the name

What does not work

The ladder, updated

The risk that has to be declared

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

TL;DR

You are here: the door nobody chose

The analogy: the switchboard that also hands out keys

The real routes

The native endpoint

The REST routes

Administration and discovery

Configuration and transport

Access control, and the correction to the previous post

allow_all_keys

require_key_mcp_access_defined

The hierarchy

The fail-open

What is done well

The bill that never shows up

How to measure it with what is already there

Tool selection

How many tools a model can take

How it fails

The optimum is not a number

Filtering can make the result worse

The prompt cache trap

And now, LiteLLM

The ladder, in order

The declared cost, which is worth zero

Observability, and the trace that gets cut

The trace is cut at the gateway

Auditing and the conflict with privacy

The problem

The fit with the ENS

What is not logged

Security

The vulnerability you need to know about

The rug pull nobody covers

Three more holes in the code

The limit that truncates in silence

Correction on the slot leak

The specification moved underneath

Tool execution from the proxy itself

Health and day-to-day operation

A hardened config.yaml

Checklist

Traps and things that are not what they seem

Closing

See also

Sources

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

TL;DR

The analogy: the detective who knows which file to ask for

The `priority` that exists and is no use

The test: the `_meta` crosses the gateway

`allow_all_keys`

`require_key_mcp_access_defined`

A hardened `config.yaml`