Security on lo0 — Tech Blog

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

Sat, 12 Sep 2026 09:00:00 +0200

A follow-up to Keycloak in an AI platform, which flagged the gap: resource indicators are not supported and protected resource metadata belongs to the MCP server. This post tries to close it. Verified against revision 2026-07-28 of the specification, the official Python and TypeScript SDKs, Keycloak 26.7.3 and LiteLLM 1.102.0.

TL;DR

The specification splits three obligations and the authorisation server covers only one. The MCP server must publish its protected resource metadata. The client must send the resource indicator in both requests, and must do so even if the authorisation server does not support it. The MCP server must validate that the token was issued for it. Keycloak does not understand the indicator and considers the metadata someone else’s, so all three end up in the resource layer.

The Python SDK mounts the metadata on its own, but degraded. As soon as the resource server URL is configured, the route appears. What that automatic route publishes carries a single authorisation server, reuses as its scope catalogue the ones the middleware requires, and leaves the name and the documentation null. To publish a complete document you have to mount the route by hand.

Audience validation comes switched off in Python and does not exist in TypeScript. In Python, if the resource URL is configured and the corresponding flag is not enabled, the SDK emits a deprecation warning and behaves as if it were disabled; the docstring itself promises that version 3 will set it to true. In TypeScript the bearer verification does three things, and none of them is looking at the audience. Neither of the two ships a signature verifier with public keys.

Token passthrough is forbidden in writing, and the substitute has a name. The specification says the MCP server must not accept or relay tokens that were not issued for it, and that if it calls upstream APIs the token must be a different one. The mechanism is token exchange, which Keycloak supports in its standard version with a limitation you need to know: the audience is a client identifier, not a resource URL.

There is an official extension that changes the whole picture, and it is stable. Enterprise-managed authorisation replaces the redirect to the MCP authorisation server with an exchange at the corporate identity provider, which evaluates the policy and issues a grant based on an identity assertion. It has been stable since June 2026 and there are servers in production.

Keycloak implements it halfway and in experimental. It only knows how to act as receiver, not as issuer, behind a feature flag, and against draft 01 when the working group is on 04. The official documentation literally says not to use it in production.

LiteLLM already implements it on the client side and does not document it. In 1.102.0 there is an authentication mode towards MCP servers that runs the two stages of that grant. There is not a single documentation file that mentions it.

You are here: the resource side

The previous post walked through the identity provider. This one walks through what sits on the other side of the token, which is where the MCP specification puts almost all its normative obligations.

The natural reading order is When MCP grows up for the basic setup, Keycloak in an AI platform for the identity piece, and this one for the missing layer. The MCP gateway seen from the inside is in its own post.

The analogy: the visa and the border checkpoint

A consulate issues visas. A border post checks them. They are two different offices, and the classic mistake consists of assuming that because the visa is genuine it is good for entering through any door.

The visa has written on it which country it is good for. That is the token’s audience, and here the problem appears: the consulate we work with does not know how to write the destination the traveller asks for, because it does not understand that field on the form. It knows how to write a destination if asked for it under a different name, by way of a pre-agreed stamp, which is the workaround of scopes and the audience mapper.

And then there is the other half, the one almost nobody builds. The border post has to exist, it has to announce where it is and from which consulates it accepts visas, and it has to read the destination written on the visa before letting anyone through. If the border post limits itself to checking that the stamp is genuine and does not look at the destination, any valid visa from the same consulate is good for entering. That is exactly what the two official SDKs do by default.

Part 1. The split of obligations

From revision 2026-07-28, with the normative words as they are:

Obligation	Whose	Status with Keycloak
Publish protected resource metadata (RFC 9728)	MCP server, MUST	Out of its scope, by design
Send the resource indicator (RFC 8707) in authorisation and in token	Client, MUST, even if the authorisation server does not support it	It does not understand it
Validate that the token was issued for oneself	MCP server, MUST	The resource’s responsibility
Not accept or relay other tokens	MCP server, MUST NOT	The resource’s responsibility
Validate the issuer of the authorisation response (RFC 9207)	Client, MUST; authorisation server SHOULD emit it	Supported
Consent for each dynamically registered client	MCP proxy with a static identifier, MUST	Configurable

The most confusing part of that split is the second line. The specification obliges the client to send the parameter regardless of whether the authorisation server supports it. With Keycloak, that parameter is lost: the official documentation says it cannot recognise it, and the standard behaviour towards an unknown parameter is to ignore it. The client complies, the token comes out, and what does not come out is the correct audience. The failure appears later and somewhere else, which is the worst kind of failure.

It is worth knowing where that work stands. The matter is issue 14355 of the project, open, with a milestone in 26.8.0. There was a complete implementation in pull request 35711, with a mapper of its own and an extension point for resolving resources, which went back to draft in October 2025 because of a total lack of tests and because it was decided to redo it in phases, starting with a single resource per client. In March 2026 a new issue was opened for experimental support, also with a milestone in 26.8.0 and with no associated pull request. In 26.7.x there is no feature flag that enables it.

What is new in the revision and affects the setup

Four things changed with respect to the previous revision and are worth collecting:

RFC 9207. A new section on validating the authorisation response. The client must record the issuer from the validated metadata document, and the comparison is literal: you are not allowed to normalise case, elide the default port, add or remove the trailing slash or re-encode characters before comparing.
Offline tokens leave the catalogue. A new section on refresh tokens: MCP servers should not include the corresponding scope either in the challenge header or in the supported scopes of their metadata.
Dynamic registration becomes deprecated. It goes from optional to optional and deprecated, retained for compatibility with authorisation servers that do not support client identifier metadata documents. And a new requirement is added to registration: the application type is mandatory, and a client must not reuse credentials from another authorisation server, it has to register again.
Step-up elevation is rewritten. The menu of three server strategies disappears and is replaced by two rules: the challenge’s scope attribute describes what is needed for the requested resource, with no obligation to include what has already been granted, and scope accumulation becomes the client’s responsibility. With a new requirement for the server: it must take scope hierarchies into account, where a broad one implies the narrow ones.

Part 2. What the SDKs give, and what they do not

Here is the uncomfortable part, and it is the main reason for writing this post. Verified by reading the code of the Python SDK and of the TypeScript 2.0.0-alpha one.

Protected resource metadata

In Python the complete model exists and it mounts on its own. The metadata class carries the RFC fields: resource, authorisation servers with a minimum of one, key set URL, supported scopes, bearer methods with a default value of header, name, documentation, policy, terms, and the client certificate and DPoP fields. The handler serves the document with a one-hour cache directive, and the URL is built by inserting the well-known path in front of the resource path, as the RFC requires.

As soon as the resource server URL is configured, the route appears with no further work. But what that automatic route publishes is degraded in three points: it passes a single authorisation server, it takes as its scope catalogue the ones the middleware requires (which are not the same as the ones the resource supports), and it passes neither the name nor the documentation, which come out null. To publish a complete document you have to call the route creation function by hand.

In TypeScript it exists and does not mount on its own. The function that builds the document emits only five fields: resource, authorisation servers, supported scopes, name and documentation. It does not emit the supported bearer methods, nor the key set URL, nor anything about DPoP. And serving it is explicit: either you call the response function from your own handler, or you mount the metadata router. There is no point in the SDK that does it on its own.

Audience validation

This is the one to fix on day one.

In Python it comes disabled. The token verifier is a single-method protocol, that is to say a hole the implementer fills. The access token’s resource field is the indicator you set yourself. And the comparison only happens if you ask for it: there is a function that normalises as a URL and ignores the trailing slash, but the resource server URL reaches it empty unless the validation flag is enabled.

The detail to read twice is in the configuration: if the resource URL is set and the flag is not, the SDK raises a deprecation warning and behaves as if the flag were false. The docstring says version 3 will set the default to true. Until then, a Python server configured with authorisation accepts tokens issued for another resource, unless the implementer enables the flag or validates the audience inside their own verifier.

In TypeScript it is not validated at all. Bearer token verification does exactly three things: strip the prefix, check the required scopes, and require that the expiry is present and not past. A search for audience or key set across the server and middleware packages returns no validation. It is all left in the hands of whichever verifier you plug in.

Neither of the two ships a signature verifier with public keys. In Python the only example is introspection against the authorisation server, and its audience check sits behind a flag that also comes out false. In TypeScript the interface is empty. That is: the component that validates the token, which is the one holding up the most important normative requirement of the specification, is your own code in both cases.

The authentication challenge

Python builds the challenge with the error and its description, and adds the metadata URL only if it is configured. It returns 401 with invalid token and 403 with insufficient scope, both from the same function, so both carry the metadata URL. What it never emits is the scope parameter, which revision 2026-07-28 asks to be included, and which is exactly what the client needs in order to know what to ask for.

TypeScript does emit the scope when there are required scopes, in addition to the metadata URL, and it correctly maps invalid token to 401 and insufficient scope to 403.

That is, each SDK gets a different half right. The Python one validates better and warns worse; the TypeScript one warns better and does not validate.

Scopes, which are not per tool

In both SDKs scopes are a static list at the transport mounting level, not per tool. There is no per-tool scope declaration and no step-up elevation help on the server side. Emitting a 403 with the specific scope that call requires, which is what the specification describes, is your own code.

That is the origin of almost all the fine-grained authorisation you have to build, and it is why there is a Part 6.

Which revision each one announces

Python is already on 2026-07-28. The revision appears in the known versions and in the list of modern versions, described as the ones that use the stateless envelope per request. The server discovery method is registered with a default handler, and the method and name headers are validated against the body.

TypeScript has it, but in a separate list. The latest protocol version constant is still 2025-11-25, because that list is only the one for the initial handshake. The modern era lives in a separate module, with a comment explaining the reason: keeping them deliberately separate so that adding a revision there never leaks a modern version string into a 2025-era handshake.

And it is worth recalling the mismatch that already came up in the MCP gateway post: LiteLLM 1.102.0 still announces 2025-06-18.

Part 3. Building the protected resource

With the above, the list of what you have to write is short and concrete.

1. A token verifier that validates the signature and the audience. Against the realm’s key set, checking issuer, expiry and that the audience contains the MCP server’s canonical URL and only things that concern it. Here the side effect of the Keycloak workaround comes in: if a client asks for two scopes from two different resources, out comes a token with two audiences, because each mapper contributes its own to the array. There is no client policy condition or executor in 26.7 that limits the number of audiences per token. The practical defence is the resource’s: reject tokens whose audience includes other resources, instead of merely checking that its own is present.

2. The complete metadata document, with all the authorisation servers you accept, the real catalogue of scopes the resource understands, and the name and documentation filled in. In Python, mounting the route by hand instead of leaving the automatic one. In TypeScript, mounting it, full stop.

3. The authentication challenge with the scope. In Python you have to add it, because the SDK does not emit it. And you have to take scope hierarchies into account, which is a new requirement of the revision.

4. The decision between local validation and introspection. Validating the signature locally costs no network, but revocation does not take effect until the token expires: there are no revocation lists for signed tokens. Introspection costs a round trip per request and only confidential clients can invoke it. The reasonable combination is a short token life and introspection on the operations that change state.

5. Binding state to the user. The new revision is explicit about this, because with protocol sessions gone, state is carried in handlers that travel as an ordinary tool argument. Servers must verify every incoming request and must not treat possession of a handler as authentication; and they should bind the handler to the user on the server side, for example storing the state under a key that combines the user identifier derived from the verified token with the handler, and reject the handler if someone else presents it. It is a change in the way of working compared with the sessions of before.

Part 4. The hop from the gateway to the MCP server

The real shape in an inference platform is not client against MCP server: it is client against gateway, and gateway against MCP server. That second hop is where it is decided whether the architecture is correct.

The specification leaves no room. The MCP server must not accept or relay tokens that were not issued for it. And if it makes requests to upstream APIs, it may act as an OAuth client of theirs, but the token it uses there is a different token, issued by the upstream authorisation server, and it must not forward the one it received.

The standard mechanism for that is token exchange, and here you need to know three Keycloak details.

First, the permission model changed. The old version required fine-grained administration permissions and an explicit exchange authorisation on the target client. The standard version, supported since 26.2, does not require them: it is enough for the requesting client to be confidential and to have the corresponding switch enabled. In exchange there is a surprising condition: the subject token has to carry the requesting client in its audience, unless it exchanges its own token. That is, the gateway needs to appear in the audience of the user token, which is achieved with an audience mapper in a default scope of the gateway. Only then can it narrow the audience to that of the specific MCP server.

Second, the audience is a client identifier, not a resource URL. The parameter filters audiences, that is to say it narrows them, which is what you want. But it takes the identifier of a client registered in the realm. The practical consequence is that the MCP server has to be registered as a client, and its identifier should be its canonical URL, so that the resulting audience matches what the resource validates. It is a naming trick, and it has to be documented for whoever comes after. The documentation itself acknowledges it: token exchange does not yet support the resource parameter.

Third, the subject token type is limited. The standard version only accepts access tokens as subject.

There is also a new experimental feature in 26.7 that adds a parameterised scope type for validating whether the requesting user is authorised to act on behalf of another. Interesting for delegation, but experimental.

What the gateway does today

In LiteLLM 1.102.0 the authentication modes towards MCP servers number twelve, with a default value of none. The ones that matter for this discussion are four.

oauth2_token_exchange implements the standard exchange. It sends the correct grant type, the subject token and its type, and the audience, never the resource. The omission is deliberate according to the code itself: fabricating a target risks an invalid target error. It caches the resulting token under a key that combines the subject token with the whole configuration, and with a lifetime equal to the token’s minus one minute. And it blocks, never degrades: with no incoming token it returns 401, which the edge turns into a challenge with the resource metadata URL; a rejection from the identity provider returns 401; a gateway configuration error returns 500; a transport failure, 503.

true_passthrough and oauth_delegate forward the client’s authorisation header as is. That is token passthrough, by the name the specification gives it, and its legitimate use is narrow: when the token the client presents was already issued for the target MCP server and the gateway is a mere transport. Outside that case, it is non-compliant.

oauth2_id_jag is the surprise, and it deserves a part of its own.

Part 5. The enterprise extension, which changes the picture

In June 2026 the MCP project published an authorisation extension that solves a different problem from the one the standard flow solves, and which in an organisation with an identity provider of its own is the real problem.

The normal flow is per user: each employee authorises each client against each MCP server. It works for consumer applications and it does not work in a company, because onboarding a person requires authorising dozens of services one by one and offboarding requires revoking them one by one.

The io.modelcontextprotocol/enterprise-managed-authorization extension inverts that. It is in the stable specifications directory of the extensions repository, it comes from SEP-990, and the flow is this:

The MCP client authenticates the user against the corporate identity provider through the normal flow, and stores the identity assertion, which may be an OpenID identity token or a SAML assertion.
When the server indicates that enterprise-managed authorisation is needed, the client exchanges that assertion at the corporate provider for an authorisation grant based on an identity assertion. The provider evaluates the organisation’s policy there: group membership, roles, conditional access.
The client presents that grant to the MCP authorisation server and obtains an access token.
The sentence that defines the extension, verbatim: the user is not redirected to the authorisation endpoint of the MCP authorisation server.

The MCP authorisation server validates the signature against the corporate provider’s key set, plus audience, issuer and expiry, and uses the subject as the user’s stable identifier, with the email as a fallback for linking earlier accounts.

What that buys is exactly what a compliance file asks for: policy in a single place, an auditable decision at the identity provider, and centralised revocation that takes effect across all clients at once. The employee who loses access stops receiving grants, without touching any server.

The underlying standard is a draft from the IETF’s OAuth working group, under the name of JWT identity assertion authorisation grant, revision 04 of May 2026, signed by people from Okta, Ping Identity and an independent author. It profiles identity chaining across trust domains by combining token exchange with the JWT profile.

And here comes the problem

Keycloak implements it only halfway. It has its own documentation page, behind a feature flag. And three limitations you need in front of you before designing anything:

It only acts as receiver. It accepts assertions issued by an external provider and issues local tokens. Native support for acting as issuer, the documentation says, is not yet fully implemented. In the extension’s flow, the issuer is the corporate provider: if that provider is Keycloak, the missing piece is exactly the one you need.
It is based on draft 01 while the working group is on 04.
The official documentation says not to use it in production.

The asymmetry is striking and worth pointing out: Keycloak does not support a standard published eight years ago such as resource indicators, and does partially support a draft from a few months back.

What is already in the gateway, undocumented

In LiteLLM 1.102.0, the oauth2_id_jag mode runs the two stages of the flow. In the first it asks the corporate provider for a token exchange with the requested token type specific to this grant, sending the user’s identity token as subject and, optionally, audience, resource and scope. In the second it presents the resulting assertion to the resource’s authorisation server with the JWT bearer grant type. In both the gateway authenticates as a client with a JWT signed by private key.

Two operational warnings that come out of the code and are nowhere else:

The subject comes from the caller’s identity token, or from the one captured at login. And only the generic OIDC provider captures assertions: with Google, Microsoft or SAML there is none, and all users fail. The code raises a warning when loading the configuration.
A badly configured server refuses with a 500 instead of falling back to the static credential. It is the right decision, and it is worth knowing before it happens.

And the main warning: there is no documentation. Not a single file in the repository that mentions this mode. Anyone using it is reading code, like this post.

Part 6. Per-tool authorisation

None of the previous pieces answers the question the business ends up asking: whether this person may call this tool with these arguments. The identity provider gives roles and scopes, and with that you do not model a graph of relations, as was already argued in the previous post.

What is published today, with sources:

OpenFGA is the only one with a dedicated product guide, updated on 9 September 2026. The pattern is a tool type with an invocation relation, checked on each call, and a listing query so the client only sees the tools it may invoke. It is a pattern and a modelling language, not a reference implementation.
Pomerium ships a per-tool policy criterion, with exact, prefix, suffix and list matching.
Kong offers it as a product, with per-tool control lists.
Traefik Hub goes as far as parameter-level restrictions, which is the finest published grain.
Envoy has an MCP filter that extracts protocol attributes for fine-grained access control, declared to be in active development and with no OAuth of its own: the hook is external authorisation towards a policy engine.

The lowest-friction option in a platform that already has a service mesh is external authorisation against a policy engine. The most expressive one, the relations model.

Part 7. What still has no standard

Two gaps the specification does not cover and that have to be closed from outside.

Tool definition change. Neither revision 2026-07-28 nor the SDKs pin a hash of the description or of the input schema. The best practices page covers confused deputy, token passthrough, server-side request forgery in discovery, state handler hijacking, server confusion and validation of the authorisation URL scheme, but there is no integrity or pinning requirement. The tool that does it is the Invariant Labs scanner, under the Apache 2.0 licence, which pins hashes to detect definition change after approval and also ships a proxy mode with guardrails. It already came up in the MCP gateway post and it remains the only answer.

The public guidance that does get into this is the NSA’s on MCP security design considerations, from May 2026, done with Carnegie Mellon’s software engineering institute. Two of its observations are directly actionable: that authorisation in MCP is optional and that the specification imposes no requirement on token lifecycle management, so expiry and rotation are left in the organisation’s hands; and the warning against dynamic tool discovery with no origin verification and no authorisation checks, which is the most direct acknowledgement of the risk in a state guidance document.

Part 8. Who covers what

With all of the above, the decision table for the layer you put in front of the MCP server:

Piece	Publishes resource metadata	Validates audience	Token exchange	Per-tool authorisation
Python SDK, automatic route	Yes, degraded	Not by default	No	No
TypeScript SDK	Does not mount on its own	No	No	No
Traefik Hub (commercial)	Yes, automatic	Not documented	Not documented	Yes, down to parameter
Kong, OAuth plugin for MCP	Not documented	Yes	Not documented	Yes, per-tool lists
Pomerium	Partial	Not documented	Not documented	Yes, per-tool criterion
agentgateway (Solo.io)	Yes	Yes, by JWT policy	Yes, including the assertion grant	Yes, with expressions
mcp-context-forge (IBM)	Not in release notes	Not documented	Yes, since 1.0.6	Yes, per-role control
Envoy, MCP filter	No	No	No	Via external authorisation
LiteLLM 1.102.0	Yes, and authorisation server metadata too	Not on the incoming JWT	Yes, and the assertion grant	Yes, per-key and per-team permissions

Two warnings about that table. The Kong plugin requires a minimum version of 3.12, is in technical preview and needs a licence for its AI edition; it also introduces a breaking change in 3.13, which starts treating all traffic as MCP in order to close a possible authentication bypass. And on oauth2-proxy I found no primary documentation either for or against protected resource mode: I am not claiming it does not have it, I am claiming it is not documented where I could look.

Reference configuration

Keycloak: the per-resource scope, optional, with its audience. That it is optional and not default is the key piece, because it is what makes the client have to ask for it explicitly and makes it work as a substitute for the resource indicator:

kcadm.sh create client-scopes -r plataforma \
 -s name=mcp:inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

And the MCP server registered as a client with its canonical URL as identifier, so that token exchange produces the right audience:

kcadm.sh create clients -r plataforma \
 -s clientId=https://mcp.ejemplo.es/inventario \
 -s enabled=true -s publicClient=false -s consentRequired=true

The gateway, with token exchange instead of token passthrough:

mcp_servers:
 inventario:
 url: "https://mcp.ejemplo.es/inventario"
 transport: "http"
 auth_type: "oauth2_token_exchange"
 client_id: "litellm-gateway"
 client_secret: os.environ/GW_SECRET
 token_exchange_endpoint: "https://sso.ejemplo.es/realms/plataforma/protocol/openid-connect/token"
 audience: "https://mcp.ejemplo.es/inventario"
 subject_token_type: "urn:ietf:params:oauth:token-type:access_token"
 token_exchange_profile: "rfc8693"

The resource verifier, which is what nobody hands you ready-made. The essential bit is the last check, the one that rejects other resources’ audiences instead of settling for finding its own:

RECURSO = "https://mcp.ejemplo.es/inventario"

async def verificar(token: str) -> AccessToken | None:
 claims = jwt.decode(
 token, await jwks(),
 algorithms=["RS256"],
 audience=RECURSO, # validates that I am there
 issuer=ISSUER,
 )
 aud = claims["aud"]
 aud = [aud] if isinstance(aud, str) else aud
 # and that nobody else is: the scopes workaround allows
 # tokens with two audiences, and that reopens the confused deputy
 if set(aud) - {RECURSO, CLIENTE_GATEWAY}:
 return None
 return AccessToken(
 token=token,
 client_id=claims["azp"],
 scopes=claims.get("scope", "").split(),
 expires_at=claims["exp"],
 resource=RECURSO,
 )

And when mounting the server, the flag that does not come set:

auth = AuthSettings(
 issuer_url=ISSUER,
 resource_server_url=RECURSO,
 validate_token_resource=True, # without this there is only a warning
 required_scopes=["mcp:inventario"],
)

Checklist

Enable audience validation in the Python SDK, or write it into the verifier if it is TypeScript. Without that, the central normative requirement is not met.
Reject tokens with other resources’ audiences, not just check that your own is present.
Mount the metadata document by hand, with all the authorisation servers and the real catalogue of scopes.
Add the scope parameter to the challenge, in Python, and account for scope hierarchies.
Register each MCP server as a Keycloak client with its canonical URL as identifier, so that token exchange produces the right audience.
Replace any token passthrough mode with token exchange, except the narrow case where the token was already issued for the target.
Check that the gateway appears in the audience of the user token, or the exchange will fail.
Bind state handlers to the user on the server side, and never treat them as authentication.
Decide between local validation and introspection knowing that the first delays revocation until expiry, and shorten the token life accordingly.
Pin a hash of the description and the input schema from outside, because the specification does not cover it.
Put a policy engine in for per-tool authorisation, and do not try to model it with roles.
If you are considering the enterprise extension, check first who issues the grant: Keycloak today only knows how to receive it, in experimental and against an earlier draft.

Traps and things that are not what they look like

The Python SDK mounts the metadata document on its own, and that is why it looks solved. What it publishes carries a single authorisation server and no name and no documentation.

The audience validation deprecation warning does not enable anything. It warns and carries on behaving as disabled.

The TypeScript SDK does not validate the audience at all. It is not configuration, it is that there is no code for it.

Python’s challenge does not carry the scope, so a client that receives a 403 does not know what to ask for.

The scopes workaround allows two audiences in the same token. Each mapper contributes its own, and there is no client policy that limits it. The defence is in the resource.

The token exchange audience is a client identifier, not a URL. If the MCP server is not registered with its URL as identifier, the resulting audience will not match what the resource validates.

Token exchange requires the gateway to be in the audience of the user token. It is the condition that wastes the most time the first time round.

Dynamic client registration is deprecated as of this revision, and the substitute, client identifier metadata documents, is experimental in Keycloak and has an open bug with documents carrying unknown fields, which blocks login with real clients.

LiteLLM’s identity assertion grant depends on the generic OIDC provider. With Google, Microsoft or SAML no assertion is captured and all users fail.

Step-up elevation is no longer the server’s responsibility. Scope accumulation moved to the client in this revision.

The authentication elevation signal goes in a 401, not a 403. Insufficient scope is 403; asking for a higher authentication level is a different thing and a different code.

Closing

The summary of this post fits in one sentence: the identity provider is the cheap half of the problem, and the expensive half is the protected resource, which no SDK hands you ready-made and which in the two official ones comes with audience validation disabled or absent.

From that comes an order of work. First the token verifier, with the audience validated in both directions, because without it everything else is decoration. Second, the complete metadata document, mounted by hand. Third, replacing any token passthrough with exchange, with the MCP server registered with its URL as identifier. Fourth, a policy engine for what roles do not model.

And a scheduling decision worth taking with the information in front of you. The enterprise-managed authorisation extension is the right route for an organisation with a provider of its own, it is stable and it has servers in production behind it. What is not ready is Keycloak as issuer. Anyone who wants that flow today has two honest options: delegate the mechanics to a gateway that implements it, or wait. Putting it into production with a feature the official documentation marks as experimental and expressly advises against is not one of them.

Sources

Especificación MCP, revisión 2026-07-28: autorización, registro de clientes y buenas prácticas de seguridad.
Extensión de autorización gestionada por la empresa y su especificación estable, del SEP-990. Anuncio de 18 de junio de 2026.
draft-ietf-oauth-identity-assertion-authz-grant, revisión 04 de 21 de mayo de 2026, grupo de trabajo OAuth del IETF.
Código de los SDK oficiales de MCP: python-sdk y typescript-sdk 2.0.0-alfa, clonados el 12 de septiembre de 2026.
Integración de Keycloak con Model Context Protocol, intercambio de tokens y concesión de autorización por aserción de identidad, consultados el 12 de septiembre de 2026.
Incidencias de Keycloak 14355, 47117 y 51039, y petición de cambios 35711.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
NSA y SEI, Model Context Protocol: Security Design Considerations, mayo de 2026.
OpenFGA, autorización de servidores MCP, actualizado el 9 de septiembre de 2026.
mcp-scan de Invariant Labs, Apache 2.0.
Documentación de agentgateway, Traefik Hub, Kong y Pomerium, consultadas el 12 de septiembre de 2026.

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

Sat, 12 Sep 2026 07:30:00 +0200

This post opens the identity block of the operations track. The previous one covered the gateway side: the JWT that does not validate audience, the user who never reaches the trace. This one covers the piece on the other side. Verified against Keycloak 26.7.3, revision 2026-07-28 of the MCP specification and the text of Royal Decree 311/2022 in the BOE.

TL;DR

Keycloak 26.7.3 was released on 31 August 2026 and is still incubating at the CNCF, which it entered in April 2023. The cadence is four minor releases a year and support in practice covers a single branch: the twenty security bugs fixed in 26.7.3 were not backported to the 26.4, 26.5 or 26.6 branches at the time of release.

The Quarkus-based distribution separates build and runtime configuration. What is set in the build phase is frozen into the image, and changing it at startup forces a rebuild. In containers, the pattern is to build into the image and start with the optimised flag.

Storage is relational and only relational. The alternative engine that was in development was withdrawn in version 25.0. Since 26.0 sessions are persisted to the database by default, which changes the load profile: the dominant cost in the official benchmarks is database CPU.

Organisations solves multi-tenancy inside a realm, fully supported since 26.0. It groups members, associates email domains and links identity providers per organisation. It is the alternative to replicating realms per customer.

The measured bottleneck is password hashing. Since 25.0 the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. The official sizing guide gives one vCPU for every 15 password logins per second, against 120 per second for client credentials and for refreshes.

Keycloak does not implement two standards the MCP specification marks as mandatory. Protected resource metadata belongs to the MCP server, not the authorisation server, so there is nothing to hold against it there. Resource indicators do belong to it and are not supported. Keycloak’s own documentation declares partial conformance with the last three MCP revisions for that reason. The practical substitute is scopes plus an audience mapper.

There is a recent critical bug worth looking at today. CVE-2026-18963, scored 9.1, allowed resetting any user’s password without authenticating, administrators included. Fixed in 26.7.2, released on 19 August 2026.

You are here: the cross-cutting layer

Identity is not a stage of the pipeline, it is a layer that crosses all of them. It shows up in stack hardening as one of the seven layers of defence, in the model’s chain of trust as workload identity, in the gateway as token validation, and in the post about authenticating MCP servers as authorisation server.

What was missing was the post about the piece itself, and above all the decision of where to put it and what to expect from it.

The analogy: the civil registry and building permits

A small town hall has a population register, which says who lives there, and it has permits, which say who can do what. They are two different things and it is worth not confusing them, because the register is good for the whole town while a permit is good for one specific job on one specific street.

Keycloak is the register. It knows who each person is, identifies them reliably, knows which groups they belong to and issues a credential document that expires. What it does not do is decide whether this person can modify this particular file: that is the permit, and it lives in each service.

Confusing the two produces the most expensive design mistake in platforms of this kind, which consists of putting into the token everything that will need to be decided later. The register document grows, gets handed out everywhere, and when a permission changes you have to wait for everyone’s document to expire.

There is a third element the analogy captures well. A credential document issued for presenting at the tax office should not be good for entering the municipal depot. That is the token’s audience, and it is the central subject of the second half of this post.

Part 1. What Keycloak is today

Project status

Item	Value
Latest stable version	26.7.3, released on 31 August 2026
Previous ones	26.7.2 (19 August), 26.7.0 (9 July), 26.6.0 (8 April)
Cadence	four minor releases a year
CNCF	incubating since 10 April 2023, not graduated
Support	the current branch; earlier ones do not receive fixes reliably

That last point deserves underlining because it conditions the upgrade policy. The analysis of 26.7.3 shows that its twenty security fixes were not backported to earlier branches at release time. Operating Keycloak means keeping up to date with the current branch, not staying on a “stable” version for a year.

Since 26.0 there is a backward compatibility guarantee across minor versions for features marked as fully supported, and breaking changes are opt-in.

The internal architecture in five pieces

Two startup modes. The development one enables HTTP, does not resolve the hostname strictly and disables the theme cache. The production one requires a hostname and TLS, or it will not start. Confusing them in production is the first antipattern on the final list.

The build phase. The build command materialises the optimisations into the image. Build options, such as the database engine, the enabled features or metrics, are frozen. Runtime ones, such as the database password or the hostname, are applied at startup. Changing a build option at startup forces a rebuild with the corresponding latency cost. The correct pattern in containers is to build into the image and start with the optimised flag.

Extensions via SPI. Authenticators, protocol mappers, user storage, event listeners, hashing algorithms and secret vaults. They are packaged as JARs and resolved in the build phase.

Infinispan cache. Embedded by default, with database-based discovery as the recommended mechanism. The discovery alternatives via Kubernetes DNS, TCP and UDP are deprecated. There are local caches, with default capacities of 10,000 entries in several of them, and replicated caches for sessions, client sessions, offline sessions, authentication sessions, login failures and action tokens. Remote mode, with an external Infinispan, is needed for first-generation multi-site deployment.

Relational storage. JPA over a database. The alternative engine was removed in version 25.0. The databases supported today include PostgreSQL from 14 to 18, MariaDB and MySQL in their extended support versions, SQL Server 2019 and 2022, Oracle 19c and 23.x, and the managed variants Aurora PostgreSQL and Azure SQL.

The minimum conceptual model

Before touching anything you need to be clear on eight concepts, and the order in which they relate:

Realm: the isolation boundary. Its own users, its own signing keys, its own policies. Two realms share nothing.
Client: an application that asks for tokens. Public if it cannot keep a secret, such as a browser application; confidential if it can, such as a service; and bearer-only if it merely validates tokens without requesting them.
Client scope: a reusable bundle of scopes and mappers, assigned to clients as default or as optional. It is the unit with which you bound what a token carries.
Protocol mapper: what projects an attribute, a role or an audience into the token. The audience mapper is the one that solves the central problem of this post.
Role, realm or client, and composite role, which aggregates others.
Group, hierarchical, with inherited roles and attributes.
Identity provider: federation towards another external OIDC or SAML. User federation: LDAP, Active Directory and Kerberos, with synchronisation and attribute mapping.
Authentication flow: a sequence of executions with required, alternative, conditional or disabled requirements, and subflows. It is where you build a second factor or a check of your own.

And one feature that changes architectures: organisations, fully supported since 26.0. It solves multi-tenancy inside a single realm, grouping members, associating email domains and linking identity providers per organisation, so that provider discovery is done by the user’s email domain. The classic alternative, one realm per customer, multiplies administration and shares no configuration.

Part 2. Who can authenticate against it

This is the table you need before drawing anything. The right-hand column is usually the one that decides.

Piece	Protocol	Cost
Kubernetes API	native OIDC, or the structured authentication configuration file	Free
LiteLLM	OIDC and JWT, plus UI SSO	JWT and SSO above five users: Enterprise
vLLM	static key only	No OIDC
Langfuse	generic OIDC and Keycloak provider	SSO free; per-project access control: Enterprise
Grafana	generic OAuth yes in the open edition	Team synchronisation: Enterprise
Backstage	generic OIDC and community-maintained Keycloak provider	Free
Open WebUI	OIDC, with role management by claim	Free
LibreChat	generic OIDC, with Keycloak documented	Free
Argo CD	native OIDC or Dex, with groups into its RBAC file	Free
MinIO	OIDC with role assumption via web identity	See the note below

Two clarifications that change decisions.

Kubernetes. The modern mechanism is structured authentication configuration, which allows several JWT issuers plus validations and mappings with CEL expressions. The corresponding proposal, KEP-3331, went alpha in 1.29, beta in 1.30 and is stable since 1.34. It is the route to use today, and it replaces the loose API server flags.

vLLM has no identity. All there is is a static key via environment variable. This is not a defect to fix, it is an architectural fact: the engine sits behind the gateway and its access control is network-based, not identity-based. Any design that exposes vLLM directly to users is giving up per-person traceability.

About MinIO a note of honesty is in order: the public documentation now redirects to the commercial edition, so the detail of how it maps policies from a claim is verified there and not in the open edition. Anyone depending on that mechanism should confirm it against their own version before designing on top of it.

Part 3. Humans and machines

They are two different problems and it is worth solving them separately.

People

Authorization code with PKCE, with the S256 method. The MCP specification requires it when the client is technically capable, and it goes further: it mandates rejecting the flow if the authorisation server metadata does not publish the supported challenge methods.

Workloads with no person behind them

Here the obvious option is client credentials with a Keycloak service account, authenticated with a secret or, better, with a JWT signed by private key.

But there is a less known and better option for workloads inside the cluster: federated client authentication, introduced in January 2026 under the banner of doing away with secrets. It lets a client authenticate with a token issued by an external provider, and one of those providers is Kubernetes. The announcement placed full support for the Kubernetes service account variant and for OIDC in version 26.6; the SPIFFE variant, by contrast, expressly says it will stay in preview until the SPIFFE client authentication standard is finalised at the IETF.

That means a pod can authenticate against Keycloak with the token the kubelet already projects into it, with no additional secret to rotate. It is worth confirming the exact status and the flag name against the release notes of whichever version you have deployed, because the feature is recent and has changed status between 26.5 and 26.6. It is preferable to token exchange, whose standard version is supported but is internal-to-internal only, while the old one, which covered external-to-internal and impersonation, is deprecated and in preview.

Keycloak against SPIFFE

The question always comes up and the answer is that they solve different things with a small overlap.

SPIRE attests the node and the process, and issues short-lived identity documents with no prior secret. Keycloak does not attest workloads: it trusts a credential someone had to place there, except in the federated authentication case just mentioned, where the one attesting is Kubernetes. In exchange, Keycloak federates people, issues tokens with business data and governs consent, things SPIFFE does not do.

The practical rule: SPIFFE for service-to-service identity inside the perimeter, Keycloak for anything that has a person behind it or crosses the perimeter. The SPIFFE detail is in the close of the chain of trust series.

Part 4. MCP, and the two missing standards

This part has the most consequences and is the worst documented outside the primary sources.

What the specification requires

From revision 2026-07-28, with the normative words as they appear:

The MCP server must implement RFC 9728 protected resource metadata, and the client must use it to discover the authorisation server.
The client must implement RFC 8707 resource indicators, including the resource parameter in both the authorisation request and the token request, and must send it regardless of whether the authorisation server supports it.
The MCP server must validate that the access token was issued specifically for it as the audience, and must not accept or relay any other token.
Authorisation servers and clients should support client identifier metadata documents. RFC 7591 dynamic registration becomes optional and is declared deprecated, retained for compatibility.

The revision also adds the issuer in the authorisation response, from RFC 9207, and a stateless model with state handlers where it is expressly stated that holding a state handler is not equivalent to being authenticated.

What Keycloak supports

From its own MCP integration guide:

Standard	Keycloak
OAuth 2.1 and authorisation server metadata	Supported
RFC 9207, issuer in the response	Supported
RFC 7591, dynamic registration	Supported
Client identifier metadata document	Supported, experimental, after enabling the feature
RFC 8707, resource indicators	Not supported
RFC 9728, protected resource metadata	Out of scope, it belongs to the MCP server

The conformance declared by the project itself is “partially supported, without resource indicators” for the last three MCP revisions.

Two practical conclusions:

The protected resource metadata document is published by the MCP server or a proxy in front of it, never by Keycloak. That is correct and there is nothing to fix, but you have to remember to implement it, because the client needs it to discover where to go.
Without resource indicators, the audience is bounded with scopes. The official route is to use the scope parameter instead of resource, and to hang from that client scope an audience mapper with the MCP server URL as the included custom audience.

It is a workaround, it works, and it has to be documented in the design because anyone coming after will look for the standard parameter and will not find it.

Part 5. The confused deputy

With the above on the table, the classic mistake becomes obvious on its own. A token with a generic audience turns any compromised service into a master key: whoever obtains the token a user presented to the dashboard can present it to the gateway, and from there to the MCP server.

The MCP specification is explicit in three sentences worth quoting verbatim in any architecture document:

MCP servers must not accept tokens that were not explicitly issued for them.
If the MCP server calls upstream APIs, it must not forward the token it received from the client.
MCP proxies that use static client identifiers must obtain user consent for each dynamically registered client before forwarding to third-party authorisation servers.

Translated into the specific figure this track has been dealing with: the gateway must not forward the user’s JWT to third-party MCP servers. It has to obtain a different token with the right audience, or act as an OAuth client in its own right. By default, as already documented in the MCP gateway post, the behaviour is to use the server’s own credential, which is the correct one; there are two modes that forward the caller’s token and both have a name in the specification.

Part 6. Two hours of task against five minutes of token

The specification pushes towards short-lived tokens and mandatory refresh token rotation in public clients. A two-hour agentic task comfortably exceeds the life of an access token and may also exceed session idle time if nobody refreshes.

Three real options, with their cost:

Offline tokens. They survive session expiry, and expire through their own idle timeout or through a maximum if that limit is enabled. The risk is obvious: a near-permanent credential, hard to inventory. And the MCP specification adds that MCP servers should not include that scope either in the authentication header or in the supported scopes of their metadata.

Automatic refresh in the client. It is the right answer when there is a client that can implement it, with refresh token rotation and secure storage. For an agent with no person behind it, it is better to renew via client credentials.

Long-lived virtual keys in the gateway. It works, and it is what most people end up doing. The price has to be written down in the risk analysis: identity ends up decoupled from the provider, so deactivating someone in Keycloak does not invalidate their key, and traceability back to the real person depends on the discipline with which the keys were created. The levers that compensate are expiry, automatic rotation and a mandatory alias, covered in the virtual keys post.

A note of honesty about the default values. The numbers going around (five minutes of access token, thirty of session idle time, ten hours of maximum) do not appear published in the administration guide and I did not locate them in the code. Treat them as a console observation and confirm them in your own realm before quoting them in a document.

Part 7. Sizing and operating

On Kubernetes

The operator exposes three custom resources: the server, the realm import and the OIDC client. Probes and metrics live on management port 9000, with the health and readiness paths, and metrics in Prometheus format provided the corresponding option was enabled in the build phase. That port is not exposed publicly.

For high availability there are three shapes. A single cluster spread across zones, with no additional infrastructure. First-generation multi-cluster, with an external load balancer and an external Infinispan at each site. And second-generation multi-cluster, introduced in 26.7.0 and still in preview, which connects clusters without external Infinispan, using the synchronously replicated database as the source of truth, with less cache and more database load.

The official sizing figures

From the CPU and memory sizing concepts guide, which is still published:

1 vCPU for every 15 password logins per second, tested up to 300 per second.
1 vCPU for every 120 client credential grants per second, tested up to 2,000.
1 vCPU for every 120 token refresh requests per second, tested up to 435.
Leave a 150 % CPU headroom for spikes, startup and failover.
1,250 MB of memory per pod with realm data and 10,000 cached sessions. Keycloak allocates 70 % of the limit to the heap and around 300 MB outside it, so the limit comes from subtracting those 300 MB from the expected usage and dividing by 0.7.

The stated assumptions behind those figures include Argon2 with five iterations, sessions in the database and the default cache of 10,000 entries.

The difference between 15 and 120 is the whole story of Keycloak’s performance: the cost is in password hashing. Since 25.0, the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. Before it was PBKDF2, whose iterations were multiplied by ten in version 24. In FIPS mode PBKDF2 is still used.

From that comes an architectural consequence for an AI platform: machine workloads, which go through client credentials, cost eight times less than a human login. Sizing by number of services is cheap; sizing by morning login peaks is the expensive part.

The benchmark

From the 26.4 performance report, published on 1 October 2025: up to 2,000 logins per second and 10,000 refreshes per second with three pods, 74 vCPU in total and 8 GB per pod, over an Aurora PostgreSQL database, with almost linear vertical scaling.

Two useful findings from that report. With 20 ms of round-trip time between sites, the 99th percentile rises from 51 to 130 ms, and the project advises against deploying across different regions. And raising the cache from 10,000 to 200,000 entries brought the database CPU peak down from 77.77 % to 63.77 %, in exchange for somewhat longer garbage collection pauses.

Security

The urgent matter, if the deployment is below 26.7.2:

CVE-2026-18963, score 9.1, critical. A bypass of the email action token in the credential reset flow allowed resetting any user’s password with no prior authentication, administrators included. Fixed in 26.7.2, released on 19 August 2026. The temporary mitigation, if you cannot upgrade, is to disable password recovery in all realms. No known exploitation was recorded as of the end of August.

In 26.7.3 there are another nineteen, among them the missing hostname verification in the TLS certificate towards LDAP with a score of 8.8, two tenant restriction bypasses in token exchange with Microsoft and Google brokers, and two access control bypasses in organisations.

Official hardening comes down to four decisions:

Strict hostname, which is the default value since the second generation of that configuration. The server does not infer the URL from headers, which prevents poisoning email links and redirects.
Administration console on a different hostname, and network-restricted.
Proxy headers declared explicitly, trusting only a controlled reverse proxy.
Separate management port, not exposed.

Part 8. The mapping to the ENS

The codes and titles below are verified against the official text of Royal Decree 311/2022 published in the BOE, Annex II.

Code	Literal title	What a central IdP contributes
op.acc.1	Identificación	A unique identifier per entity, user or process that gains access
op.acc.2	Requisitos de acceso	Permissions are granted by role and group, not by loose credential
op.acc.3	Segregación de funciones y tareas	It supports it with roles, but the control is organisational
op.acc.4	Proceso de gestión de derechos de acceso	Least privilege and a specific remote access policy
op.acc.5	Mecanismo de autenticación (usuarios externos)	Flows and factors configurable per flow
op.acc.6	Mecanismo de autenticación (usuarios de la organización)	Reinforcements required already at the low level
op.exp.8	Registro de la actividad	It contributes part of the events, not all of them
op.exp.10	Protección de claves criptográficas	Management and rotation of the realm signing keys

Two nuances an auditor is going to look for.

The first, about op.acc.1: the text requires every entity that accesses the system, whether user or process, to have a unique identifier. That includes agents. A virtual key shared between several agents does not comply, and that is exactly what usually happens when JWT authentication falls outside the budget.

The second, about op.exp.8: the control requires recording the identifier of the user or entity associated with the event, the date and time, what information it is performed on, the type and the result. Keycloak contributes the authentication events. The usage events are contributed by the gateway and the MCP servers, and there the two already documented gaps reappear: adding and removing MCP servers generates no audit log and the trace user is set by the client.

The correspondence between control and product in this table is my own interpretation and must be validated by the certification body. The codes and titles are not: those are literal from the BOE.

Part 9. The configuration to leave written down

Three concrete pieces, so as not to stay at the level of concepts.

Hardened startup, as container arguments:

kc.sh start --optimized \
--hostname=https://sso.ejemplo.es \
--hostname-admin=https://sso-admin.interna.ejemplo.es \
--proxy-headers=xforwarded \
--health-enabled=true \
--metrics-enabled=true \
--http-management-port=9000 \
--cache=ispn --cache-stack=jdbc-ping \
--db=postgres

The metrics, health, cache and database options are build options: they go into the image, and passing them here forces a rebuild if they do not match.

The client scope with audience, which is the workaround forced by the lack of resource indicators. Created with the administration tool:

# one scope per protected resource
kcadm.sh create client-scopes -r plataforma \
 -s name=mcp-inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

# the mapper that sets the token audience
kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocol=openid-connect \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

The client asks for that scope and receives a token whose audience is only that MCP server. The MCP server validates the audience and rejects any other token, as the specification requires.

Realm import via GitOps, with the operator’s custom resource, so that the configuration does not live only in the console:

apiVersion: k8s.keycloak.org/v2beta1
kind: KeycloakRealmImport
metadata:
 name: plataforma
 namespace: identidad
spec:
 keycloakCRName: sso
 realm:
 realm: plataforma
 enabled: true
 # without this, the recovery email is attack surface
 resetPasswordAllowed: false
 bruteForceProtected: true
 sslRequired: all

Part 10. What Keycloak does not solve

Fine-grained authorisation over resources. The question “can this person see this particular trace” is a relation between objects, not a role. That problem is solved by Zanzibar-style engines, with OpenFGA and its object, relation and user tuples, or a policy language such as Cedar. Keycloak gives roles and scopes, and with those you do not model a graph of relations.

Machine secrets. Storage credentials, provider keys, database passwords. That is a secrets manager, covered in stack hardening.

Cryptographic workload identity with attestation. That is SPIFFE and SPIRE. In Keycloak it exists as an identity provider in preview, which is a different thing.

Authorisation inside the model. Which MCP tool an agent may invoke, and with what arguments. The specification offers scopes, the insufficient scope error and step-up elevation, but the per-tool and per-argument decision is taken by the MCP server or a policy engine, not by the identity provider.

Antipatterns

Forwarding the user’s token from the gateway to third-party MCP servers. Explicitly forbidden by the specification.
Static client identifier with dynamic registration and no per-client consent, which is the exact recipe for the confused deputy.
General scopes, of the full-access kind, and publishing the entire catalogue of supported scopes in the metadata.
Treating a state handler as authentication, which the new revision expressly forbids.
Following OAuth discovery towards any URL, which opens the door to requests towards internal addresses. It also applies to the authorisation server when it downloads a client metadata document.
Trust headers badly exposed in chat front ends. The Open WebUI documentation warns that an incorrect configuration allows authenticating as any user.
Starting in development mode in production, with the hostname not resolved strictly.
A single shared static key as the only control between the gateway and the engine.

Startup checklist

Check the version. Below 26.7.2 there is a critical password reset bug.
Build the image with the build phase done and start with the optimised flag.
Strict hostname, administration console on another hostname, separate management port and not exposed.
Decide between organisations and one realm per tenant before creating the second tenant.
One client scope per resource, with an audience mapper, because resource indicators are not supported.
In the gateway, define the two audience and issuer variables. Without them none of that is validated.
For workloads inside the cluster, look at the Kubernetes identity provider before handing out client secrets.
Size by login peaks, not by number of services: the difference is eightfold.
Export the Keycloak events to the same place the gateway’s go, because the activity logging control needs them together.
Write into the design what the system does with a two-hour task, before someone solves the problem with an offline token.

Closing

Keycloak is a mature piece that solves one bounded problem well: knowing who the caller is, and saying so verifiably in a document that expires. In a sovereign inference platform that is worth a lot, because the alternative is an inventory of static keys spread across ten services that no audit can walk through.

What is not advisable is asking more of it. Fine-grained authorisation lives in each service, workload identity with attestation lives in SPIFFE, and the decision of which tool an agent may call lives in the MCP server.

And there is one concrete gap worth noting down, because it does not close on its own: the MCP specification marks two standards as mandatory, and of the two that would fall to an authorisation server, resource indicators are not implemented. The workaround with scopes and an audience mapper works, but it is a workaround, and whoever comes after will look for the standard parameter. Leaving it written in the design saves an afternoon.

Sources

Keycloak 26.7.3, 31 de agosto de 2026, y descargas.
Configuración del servidor, caché, bases de datos y nombre de host.
Integración con Model Context Protocol, intercambio de tokens y autenticación federada de clientes, 27 de enero de 2026.
Dimensionado de CPU y memoria e introducción a alta disponibilidad.
Bancos de pruebas de rendimiento de Keycloak 26.4, 1 de octubre de 2025.
Especificación de MCP, revisión 2026-07-28, autorización y buenas prácticas de seguridad.
KEP-3331, configuración estructurada de autenticación.
Real Decreto 311/2022, Anexo II.
Registro de CVE-2026-18963 en la base de datos de Red Hat y notas de la versión 26.7.3 en GitHub.
CNCF, proyecto Keycloak.

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

Fri, 11 Sep 2026 01:45:00 +0200

Sixth post in the operational track on the control layer. The previous five treat the gateway as the piece the tokens pass through: the pair with Langfuse, day 2, virtual keys, humans and agents and prefix routing. This one is about the other door. Verified against LiteLLM 1.102.0, commit of 10 September 2026.

TL;DR

The MCP gateway is not a module, it is a complete second surface. A native MCP endpoint with four route spellings, a proxy mode with three fixed tools, four REST routes, some thirty-five administration routes and a long set of OAuth discovery endpoints. It still lives under _experimental and it is nearly 20,000 lines.

I correct what I said in the previous post: access is NOT open by default. In 1.102.0, a key with no declared permissions and no team resolves zero servers. What does open things up is a per-server flag, allow_all_keys, and that flag overrides the key’s explicit scope unless you enable a setting that is off by default.

Permissions are enforced on execution, not just on the listing. Calling a tool that did not appear in tools/list returns 403. There are three chained checks before the call, including one on the allowed arguments. The same predicate governs listing and calling, by declared design.

The hierarchy is built out of intersections, with two escapes and a fail-open. The key’s access groups are additive on top of the key and team ceiling. The organisation ceiling replaces rather than intersects when there is nothing below it. And on an indeterminate error while resolving tool permissions, the code returns “unrestricted”. It is a deliberate decision and it has to be declared in the risk analysis.

Per-tool cost is worth zero and that is why its metric disappears. There are exactly two Prometheus metrics for MCP, and the spend one is only incremented if the cost is greater than zero. With the default configuration, that series never exists. There is no listing metric, no latency metric, no error metric, no server health metric.

The cost that does matter is not attributed to MCP anywhere. It is the tokens of the tool definitions, resent on every turn inside the prompt_tokens of the model call. Five typical MCP servers add up to 58 tools and some 55,000 tokens. The tool’s spend row, with its zero cost, has nothing to do with that.

And the gateway’s semantic filter does not act on the MCP door. It only applies when it is LiteLLM calling the model. A client connected to /mcp receives the whole catalogue, filter enabled or not. And even if it did act, filtering per request rewrites the cached prefix of the prompt, which is the opposite of what Anthropic and OpenAI do with their own tool searches.

The good observability ships switched off. Without LITELLM_OTEL_V2, a tool call comes out as a generic span with the model set to MCP: name. With it enabled there is a dedicated span and correct attributes, and the arguments and the result stay hidden unless you opt in. And in neither case is trace context propagated towards the upstream MCP server: the trace is cut at the gateway.

Tool arguments are written, and they are written in the clear. They go in full into the metadata column of the spend table, without passing through the switch that governs prompt storage nor through the redaction function, which only knows about messages and response. Whoever enabled redaction for GDPR probably believes they are covered.

The 1.83 branch concentrated four serious vulnerabilities, and one of them belongs to the MCP gateway itself. The endpoint for testing a connection before saving accepted the stdio transport’s command, args and environment in the request body. A low-privilege key got command execution on the proxy host.

And the specification moved underneath. The revision in force since July 2026 removed protocol sessions, the initialisation handshake and the session header. LiteLLM 1.102.0 advertises the June 2025 revision and its version enumeration does not even contemplate the two later ones.

You are here: the door nobody chose

The gateway was chosen for the reasons in the switchboard post: licence, fit inside the cluster, maturity. What was evaluated was the inference traffic.

Later, somebody connects an agent, the agent needs tools, and the same process that is already authenticated and deployed turns out to speak MCP too. The decision to turn it into the tool door is almost never taken: it is inherited. And that door has neither the same controls, nor the same observability, nor the same cost model.

When MCP grows covered the identity of your own MCP servers. MCP from the inside covered the protocol and its instrumentation. This post covers the piece in between.

The analogy: the switchboard that also hands out keys

An office building’s telephone switchboard. Its original job is to route calls: it decides who can call outside, how much each department spends, and it leaves a record of every conference call.

Over time, the switchboard operator is also given the key cabinet. They are already at reception, they already know who everybody is, it looks like the natural place. And from that moment there are two services at the same desk with very different rules.

Calls are priced by the minute and show up on the bill. Keys have no tariff, so on the department’s bill they come out at zero, and somebody might conclude that the key cabinet costs nothing. It does cost: it costs the operator’s time, it costs having to read the whole key catalogue every time somebody asks what is available, and it costs that nobody checks that the label on a key still says what it said the day its loan was authorised.

The analogy ends at the interesting spot. When somebody takes a key, the record says which key it was and what they said they wanted it for. That record is simultaneously the evidence of the action and a piece of data that perhaps should not be stored in the clear.

The real routes

The first thing that surprises you when you look at the code is how many spellings get you to the same place.

The native endpoint

The MCP sub-application is mounted at /mcp and inside it has four mounts: the root as a wildcard, /mcp, /{name}/mcp and /sse.

And there is also a hand-declared route for bare /mcp on the main application, with a comment that explains why: the mount cannot match its own bare prefix, and the resulting 307 redirect breaks MCP clients sitting behind a proxy that terminates TLS. It is the kind of detail you discover in production on a Friday.

/mcp/sse holds a surprise. It does not use the classic SSE transport: it uses the same HTTP session manager in stateless mode. The SseServerTransport object is constructed and is not referenced on any other line of the file. It is dead code, and the /mcp/sse/messages endpoint that this object advertises does not exist as a working route.

The /{name}/mcp route resolves in a documented order: server alias, comma-separated list with a maximum of 16 elements, tool set, access group. If nothing matches, 404.

And /mcp/proxy exposes a fixed surface of three tools. It is the progressive disclosure pattern, and it will come up again further down because it is the most important mitigation in this post.

The REST routes

Four, under the /mcp-rest prefix:

Method	Route	Authentication
GET	`/mcp-rest/tools/list`	Virtual key
POST	`/mcp-rest/tools/call`	Virtual key
POST	`/mcp-rest/test/connection`	Key plus administrator role
POST	`/mcp-rest/test/tools/list`	Key plus administrator role

The variants without -rest that appear in the internal types have no router registering them. The /mcp/* wildcard absorbs them as JSON-RPC.

The last two are the stars of the security section.

Administration and discovery

Some thirty-five routes under /v1/mcp, almost all of them with virtual key authentication. Three details worth knowing:

GET /v1/mcp/registry.json carries no authentication dependency. It returns 404 if the public registry is not enabled, which is the real protection.

POST /v1/mcp/server requires the proxy administrator role. POST /v1/mcp/server/register does not: it is the proposal path, which requires a key with a team, explicitly forbids the stdio transport and leaves the entry pending review.

And on top of that there is a long set of OAuth .well-known endpoints, with dynamic client registration, authorisation, token, revocation and introspection, plus their per-server variants.

Configuration and transport

The mcp_servers schema in config.yaml has fifty-odd fields, all of them optional. A warning for anyone coming from earlier versions: spec_version no longer exists in 1.102.0.

The default transport when omitted is http, that is, streamable HTTP. The other two values are sse and stdio. The string streamable_http is not valid in the configuration file, although it is accepted as a synonym when importing connector files.

Tool prefixing uses the format {server}{separator}{tool} with - as the default separator, configurable through an environment variable read at import time. The prefix chosen is the alias, or the name, or the identifier, in that order. There is an optional short mode that derives three characters from the hash of the server identifier and resolves collisions by rehashing.

Access control, and the correction to the previous post

In the humans and agents post I closed the list of traps with this one: “access to MCP servers is open by default if no level defines a list”. Verified in 1.102.0, that is false.

The function that resolves the allowed servers does the opposite. If the team declares nothing, it inherits from the key; if the key declares nothing either, the base set is empty and the scope is marked as unbounded with zero servers. There is no branch that expands to “the whole registry” other than the administration role one.

What does open things up, and this is the important part, is something else.

`allow_all_keys`

It is a per-server flag, in configuration, in the API and in the database, defaulting to false. When set to true, that server joins the set of open servers and any key can see it.

The detail to know: by default it overrides the key’s explicit scope. A key that declares access only to server A will also see server B if B has the flag. The key’s scope is only respected if you enable general_settings.mcp_allow_all_keys_respects_mcp_scope, which is off by default.

There is a second opening case: anonymous subjects, with no user identifier, role or key, additionally receive the servers configured with authentication delegation towards the upstream and the pass-through ones. It is consistent with the design of those two modes, and it is a surface you have to know about.

`require_key_mcp_access_defined`

It lives in general_settings, defaults to false, and needs no licence. What it inverts is only the empty-key inheritance towards the team: with the flag on, a key with no list of its own gets an empty set instead of inheriting the team’s.

It does not affect keys that do declare servers, which still intersect with the team. And it does not close off the access group grants, which are additive and are the escape route you have to audit separately.

There is a sibling, require_end_user_mcp_access_defined, with the same shape and applied to the end user.

The hierarchy

The function’s docstring is normative and says that all the rules are intersections. In detail:

Key and team: intersection when both declare. Inheritance in one direction when one is empty.
Key access groups: additive union on top of the previous result.
End user: intersection if it declares anything.
Agent: intersection.
Internal user: ceiling that only narrows.
Organisation: ceiling that intersects if there are restrictions below, and that replaces by becoming the ceiling when there are none.

For subjects with no key, admitted by SSO or by the gateway, the model changes to a per-source union instead of an intersection.

The fail-open

This is the point you have to take to the risk analysis and not leave in a technical note.

On an indeterminate failure while resolving the tool permissions of a key or a JWT, the code returns “unrestricted”, that is, all tools allowed. The organisation ceiling is skipped in the same situation. For subjects with no key the behaviour is the opposite, fail-closed.

There is a well-resolved intermediate case: when a permission is named but cannot be read, a specific error is raised and it is denied in both cases.

The doctrine is documented in the class docstring, which is appreciated. And it remains a behaviour that, in a system under the ENS, you have to declare in writing instead of discovering it during an audit.

What is done well

Permissions are enforced on the call, not just on the listing. The execution path chains three checks before contacting the server: allowed or forbidden tools, per key and team tool permission, and validation of the arguments against the allowed ones. All three return 403.

Listing and calling share the same predicate, and the docstring declares it as an explicit invariant. A client that calls a tool directly when it did not appear in its listing receives 403, it does not execute. The same over REST and over the responses API route.

The x-mcp-servers and x-mcp-access-groups headers only narrow, and they fail closed if they do not resolve.

Header forwarding towards the upstream is an explicit per-server allowlist, with a centralised decision about when to strip the caller’s Authorization, and a protection against cross-forwarding: in a multi-server listing, the global header is withheld if more than one server would consume it. That is RFC 9700 applied.

The bill that never shows up

Here is, in my opinion, the most important thing in this post, and it has nothing to do with security.

Every tool the gateway aggregates takes up space in the model’s context. Name, description and parameter schema travel in the request’s tools block, on every turn, and they are paid for as input tokens of the real model.

The figures published by Anthropic in November 2025 are the best public reference there is. Five connected MCP servers add up to 58 tools and some 55,000 tokens. The breakdown: GitHub, 35 tools and some 26,000 tokens. Slack, 11 tools and some 21,000. Sentry, five and some 3,000. Grafana, five and some 3,000. Splunk, two and some 2,000.

With tool search enabled, that set drops from some 72,000 to some 8,700 tokens, and the usable context goes from 122,800 to 191,300.

The second cost of a large catalogue is not economic and is not billed: it is that the model chooses worse. It has its own section further down, because of the two it is the one that decides whether the agent is any good.

How to measure it with what is already there

The gateway stores enough data, even if it does not cross-reference it for you.

In the listing’s spend row, with call type list_mcp_tools, it enriches the metadata with allowed_server_count, tool_count_total, per_server_tool_counts and per_server_list_outcomes.

In the model call rows, the proxy_server_request column keeps the request body with the tools block inside it.

The practical method: pair the listing rows with the LLM rows of the same session by session_id, read the total number of tools from the first, and pass the tools array of the second through litellm.token_counter(model=..., tools=...), which accepts that argument. Multiply by the number of turns, because the block is resent on all of them.

For the relative cost, a quicker comparison: mean prompt_tokens of keys with MCP enabled against those without, same model and same period.

Tool selection

The previous section treats the catalogue as a bill. This one treats it as what determines whether the agent gets it right. This is ground where the talk runs well ahead of the measurement, so it is worth separating what has been measured from what gets repeated.

How many tools a model can take

Anthropic’s official figure is that Claude’s ability to pick the right tool degrades when going from 30 to 50 available tools. The same documentation gives the inverse criterion, which is the more useful one: the full catalogue with nothing else is the right option with fewer than ten tools, when all of them are used on every request, or when the definitions add up to fewer than a hundred tokens.

There is also an administrative ceiling before the semantic one. VS Code with Copilot cuts off at 128 tools per request and returns an error when you exceed it.

The full curve gets cited far more than it has been measured. The soundest primary data I found is from May 2025: varying the number of candidates from 1 to 11,100 over a registry of more than 4,400 servers, below some 30 tools the success rate is above 90 %, between 31 and 70 intermittent failures appear, and past the hundred mark the degradation becomes dominated by retrieval. It is a qualitative heat map, not a numeric table.

The figure that circulates most (78 % with ten tools, 40 % with a hundred, 13.6 % with seven hundred) comes from a conference talk with no paper, no repository and no published methodology. Do not take it to a client presentation.

And there is a warning that affects everything else. An audit of 496 tasks from four tool calling benchmarks found 18.5 % misalignment between the label and reality. In the BFCL v4 subset, 80 % of the failures assigned to the agent came from fragile state comparisons. In one of the benchmarks with an automatic judge, 23 identical re-runs gave a range from 57.9 % to 76.8 %, almost 19 points of spread, enough to reorder the ranking. Any absolute figure in this section deserves a generous margin of error.

How it fails

The work with the cleanest taxonomy evaluates 36 servers, 220 tools and 1,000 tasks against twenty frontier models, with an average of 15.2 tools exposed per task of which 4.1 are relevant.

Failure mode	Share
Not using a tool when one was needed	10.5 %
Choosing the wrong tool	9.0 %
Malformed parameters	6.9 %
Not recovering from an error	10.3 %
Cognitive failures, not tool failures	63.3 %

The interesting part is in the per-model breakdown: the dominant failure mode changes with the model, not with the size of the catalogue. In one of them, 40.1 % of the failures are simply not calling any tool. In others the malformed parameter dominates. In another, the wrong tool. There is no canonical failure to attack.

On tools that do the same thing, which is the real case of a gateway with fifteen servers, there is a good measurement. A study with ten groups of five functionally equivalent tools and a thousand query pairs puts the selection bias between 0.3 and 0.4, that is, you would have to redistribute between 30 and 40 % of the probability mass for equivalent tools to be chosen equally. The same work shows that a biased continued training takes the selection of a specific endpoint from 0.6 % to 12.8 %.

That study contributes another figure that reorders priorities: shuffling a tool’s description moves the selection distribution substantially, whereas changing only the name has minimal and inconsistent effects. Applied to the gateway’s server-tool prefixing: it costs tokens and probably does not change which tool the model goes to. There is no study isolating the effect of name spacing, and it is a real gap in the literature.

If the description is what carries weight, the state of the estate is the problem. An audit of 856 tools from 103 MCP servers found undeclared limitations in 89.8 %, missing usage guidance in 89.3 % and opaque parameters in 84.3 %. Only 2.9 % were free of problems across all their components.

The optimum is not a number

The May 2026 work I cited above proposes a chance-corrected metric and uses it to size the list adaptively. Over 370 functions it reaches 90.3 % coverage while seeing an average of 7.4 tools, against 90.8 % seeing a fixed 50.

But the same paper publishes its own counterexample, and it is honest to cite it: over another benchmark of 3,251 tools, the fixed K of 5 wins in aggregate, 64.7 % against 61.9 %. The adaptive advantage concentrates in the hard queries, where the right tool sits between sixth and twentieth by similarity: there it finds 16.7 % of the cases where the fixed K finds 0 %.

The March 2026 work over 121 tools puts the plateau at K=3, with 97.1 % that does not improve going up to K=5. With the caveat that it measures retrieval against labels, not model accuracy.

And a July 2026 work gives the formulation that seems to me the right one: ranking by score is inconsistent with optimal acquisition when costs are heterogeneous, so the optimal K is a function of cost, not a constant. Trimming from 7 to 4.4 exposed tools while keeping success is possible; fixing a universal number is not.

Filtering can make the result worse

This is the part that appears in no product blog. A June 2026 benchmark with three registry sizes (25, 100 and 250 tools), six filtering methods, seven models and 26,460 runs:

Strategy	Success	Visible tools	Tokens
Expose the whole catalogue	32.1 %	125.00	56,062
Keyword, top-5	22.1 %	4.80	3,200
Keyword, top-10	22.4 %	9.54	5,356
State-aware	24.0 %	25.95	13,522
Full causal path	24.0 %	26.24	13,697

Four of the five filtering strategies perform worse than not filtering. The fifth, which does win comfortably, exposes 0.99 tools per step and requires hand-written precondition and effect contracts for every tool, plus a search of the causal path to the goal state. It is a planner with the goal given, not a retrieval filter, and it is not comparable with anything a gateway offers.

There is more evidence in the same direction. A work over a corpus of 43,000 tools measured that the agent’s completion rate drops when using retrieved sets against hand-annotated sets: the filter introduces its own recall ceiling. And top-K breaks the tasks that need several tools: on a benchmark of 16,464 endpoints, a fine-tuned retriever reaches a Recall@3 of 68.6 % but only 39.7 % of the queries have their complete tool set within the first three. Almost 29 points of gap between “the one I needed is there” and “all the ones I needed are there”.

Filtering is not free on security either. A March 2026 work attacks the retrieval layer by injecting adversarial tools that saturate the top-k: with injection rates of 1.2 % it manages to dominate the top-k between 91 % and 97 % of the time. It is an attack surface that exposing the whole catalogue does not have.

The most elegant counterpoint comes from Anthropic’s own numbers. With the same catalogue of more than fifty tools, the gain from tool search drops from 25 points on one model to 8.6 on the next, because the unfiltered baseline rises from 49 % to 79.5 %. The margin left by filtering shrank to a third in one generation of models. Any architecture decision taken here has a short shelf life.

The prompt cache trap

Here is what makes a homemade top-K filter a bad idea even if it worked.

Tool definitions do not go just anywhere in the prompt: they go at the beginning. Anthropic documents the order of the cacheable prefix as tools, then system, then messages, with a hierarchy where each level rests on the previous one. And the invalidation rule is explicit: modifying tool definitions, be it names, descriptions or parameters, invalidates the entire cache, and a change at one level invalidates that level and all subsequent ones.

OpenAI says the same and adds a detail: the prefix must remain unchanged, and order counts. The same set of tools served in a different order is a cache miss.

With vLLM serving an open model there is no tool block of its own. The tools parameter is serialised by the chat template inside the first system message, behind the system prompt text. For automatic prefix caching the effect is identical, because it is still near the absolute start of the prompt.

The consequence is the one that matters. A filter that returns a different set of tools on the next turn does not just invalidate the tool block: it invalidates everything behind it, which is the system prompt and the whole conversation, which is exactly the part that grows with the turns. At list prices, a prefix token goes from costing 0.1 times when read from cache to 1.25 times when written. An agent with sixty thousand tokens of stable prefix goes from some six thousand equivalent tokens per turn to some seventy-five thousand, plus the prefill that gets paid again in latency.

The two big providers have separately arrived at the same solution, and it is the opposite of filtering in the prefix. Anthropic excludes deferred tools from the prefix and, when the model discovers one through search, injects it inline inside the conversation: “the prefix stays intact, so prompt caching is preserved”. OpenAI: “all tools are loaded at the end of the model’s context window (…) this allows the cache to be preserved from one request to the next”. Anthropic’s operational recommendation is the stable core pattern: leave the three to five most used tools undeferred.

The closest academic work, from January 2026, states the problem without measuring it and recommends exactly that, keeping a fixed set of reusable general-purpose functions and implementing dynamic capability through code generation.

The July 2026 MCP specification picks up both halves of the matter. Servers “should return the tools from tools/list in a deterministic order to allow caching in the client and improve the hit rate of the model’s prompt cache”. And it adds mandatory ttlMs and cacheScope in listing results, so that the client can freeze the catalogue for a known window instead of re-querying it and propagating any variation to the prompt.

Nobody has publicly measured the cost in euros of this effect. The only attempt I found is a self-published audit of ten thousand turns that puts it at 2.4 % to 3.5 % of cache misses, with non-reproducible methodology. It is a gap in the literature, and the order of magnitude in the previous paragraph comes from applying list prices, not from a measurement.

And now, LiteLLM

With all of the above in mind, the gateway’s levers read differently.

The semantic filter does not act on the MCP endpoint. It is registered as a generic LiteLLM callback, and its only dispatcher is the proxy’s precall hook, which additionally discards anything that is not completion, acompletion or aresponses. The tools/list path does not invoke that hook at any point; there is a comment in the code that admits it in passing, when explaining why the listing’s JWT signature is done separately. An external MCP client connected to /mcp receives the whole catalogue, filter enabled or not.

Lever	Serves external MCP clients	Breaks the prompt cache
Per-server `allowed_tools`	Yes	No, it is static
Name and description overrides	Yes	No, it is static
Selection by route or header	Yes	No, if it is stable per agent
`/mcp/proxy` and virtual tools	Yes	No, the surface is fixed
Semantic filter	No	Yes, it rewrites the block per request

When it does act, the filter has details you need to know before enabling it. It embeds only each tool’s description, with the name as a fallback if it is missing; the parameter schema does not go in. Of the query it embeds only the last user message: on turn twelve of an agent conversation, that might be “yes, go ahead”. The tool embeddings are cached and indexed at start-up; the query’s is computed on every request, with a synchronous call inside a coroutine.

Its failure modes are almost all open, which is the reasonable thing: if the embedding model does not respond, if nothing passes the threshold, if the names do not match the catalogue, it returns all the tools. The exception is context overflow, which fails closed with a 400, and which if it happens while building the index during start-up is memoised and makes all subsequent requests fail until you restart. The defaults are text-embedding-3-small, top_k of 10 and a threshold of 0.3, under litellm_settings.mcp_semantic_tool_filter.

/mcp/proxy does serve external clients, and that is where its real value lies. It replaces the catalogue with three tools with opaque identifiers. How search_tools searches deserves a line, because it is not what you would assume: by default it is neither a regular expression nor a semantic match, it is a keyword counter, how many tokens of the query appear as a substring in the concatenated name and description. It only moves to embeddings if you configure a model in mcp_tool_search. The result cap is hard-coded at five in this mode, and the client cannot ask for more. In exchange, call_tool goes through the same permission checks as a normal call plus a validation of the argument schema that the normal route does not do at that point.

There is a related variant: with mcp_tool_search_enabled in the key’s permissions, the catalogue is replaced by four virtual tools, and there the search one does expose top_k to the client.

The catalogue order is not deterministic, and that breaks the client’s cache without anybody filtering anything. The list of allowed servers comes from iterating a Python set of strings, so with the random hash seed the order differs between processes, between uvicorn workers and between restarts. There is no sorting anywhere in the listing path, and there is no tool cache: every tools/list queries the upstreams live, so the order within each server also depends on what the upstream returns that time. The gateway does not offer today the guarantee that the July specification recommends. The fix, sorting the catalogue before serving it, fits on one line.

Nor is there any pagination towards the client. The catalogue is returned whole, always.

And the lever almost nobody uses is the static one. Every MCP server accepts tool_name_to_display_name and tool_name_to_description, and those overrides really do replace what is sent to the client in tools/list, not just what the administrator sees in the interface. The tools/call routing translates from the new name to the real one before any permission check. With that you can trim a four-hundred-token description written by a third party, rename an ambiguous tool so it does not compete with a similar one, and do it without touching the prompt cache, because it is static and holds for all turns.

It is also the rug pull mitigation left pending from the security section: if the description the model sees is the one the operator wrote, whatever the upstream server changes stops reaching the context. Two limits. They are configured through the database or the management API, not through config.yaml. And the reverse name translation is by exact equality and is not disambiguated between servers, so two servers with the same display name route unstably, made worse by the non-deterministic order of the previous paragraph.

Alongside that, per-server allowed_tools remains the blunt and effective instrument: it filters the listing the client sees, reduces tokens, and is static.

The ladder, in order

Trim. Per-server allowed_tools, plus a per-key allowlist. Cheap, static, does not break the cache and reduces the permission surface at the same time.
Rewrite the descriptions with the overrides. That is where the evidence is, because the description carries weight and the name almost none, and along the way it pins the contract against the rug pull.
Split by route. Each agent to /{server}/mcp or to a tool set, never to the aggregated endpoint. Stable per agent, so the cache survives.
Sort the catalogue before serving it if you control the deployment, while the gateway does not do it.
/mcp/proxy when the catalogue goes past thirty or fifty tools. It is progressive disclosure and it works with external clients, which is more than you can say for the semantic filter.
The semantic filter, only when the one calling the model is LiteLLM. With a generous top_k, knowing that it embeds only the last user message and that context overflow is a closed and sticky failure.
What not to do: a home-grown top-K filter that rewrites the tools array on every turn. It costs the whole prefix and, according to four of the five measured strategies, probably makes selection worse.

The declared cost, which is worth zero

The calculation has four precedence steps: cost set by a post-call hook, per-tool cost in tool_name_to_cost_per_query, server default cost, and 0.0.

Without configuring mcp_server_cost_info, all MCP traffic is accounted at zero and consumes neither key nor team budget.

There are exactly two Prometheus metrics for MCP:

litellm_mcp_tool_calls_total
litellm_mcp_tool_call_spend_metric

Both with the same eight labels: tool name, server name, key hash, key alias, team, team alias, user and end user.

And the second is only incremented if the cost is greater than zero. With the default configuration, that time series never appears in Prometheus. It is not that it is worth zero: it is that it does not exist.

Nor is there a tools/list metric, or MCP latency, or MCP errors, or upstream server health. For that you are left with the generic request metrics.

In the reports, /spend/calculate has no MCP branch: it accepts a model and messages, or a completion response, and it always calls the LLM cost calculation. The per-team report does include MCP calls, grouped by the model column, which for these rows is worth MCP: tool_name. That is: every tool appears as if it were a model, with zero input and output tokens. Only the sessions endpoint separates the two call types explicitly.

Observability, and the trace that gets cut

There are two OpenTelemetry stacks in LiteLLM and which one runs depends on an environment variable.

Without LITELLM_OTEL_V2=true, the old logger runs, and it has no MCP handling: the only appearance of MCP in the whole file is the metadata key in a list. The tool call comes out as a generic litellm_request span with the model set to MCP: name.

With v2 enabled there is a dedicated span, with a dedicated role, client kind and a parent in the proxy’s request span. The span name is tools/call tool-name. The attributes are correct and follow the conventions:

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`mcp.method.name`	`tools/call`
`mcp.session.id`	MCP session identifier
`gen_ai.tool.name`	Tool name
`gen_ai.tool.call.arguments`	Only with content capture enabled
`gen_ai.tool.call.result`	Only with content capture enabled
`server.address`, `server.port`	From the upstream URL, redacted
`litellm.cost.total`	The cost that is probably worth zero

Content capture is off by default. Without it you see neither the arguments nor the result in the trace.

A note for anyone working with semantic conventions: mcp.tool.name does not exist. The MCP registry has exactly four attributes (mcp.method.name, mcp.session.id, mcp.protocol.version and mcp.resource.uri), all four in development, and the tool name goes in gen_ai.tool.name. LiteLLM gets it right. And the GenAI conventions were extracted into their own repository which as of today has no release at all, so there is no version number to cite.

In Langfuse, the vendor mapper returns an empty dictionary for any span that is not an LLM call. An MCP call appears as a raw span with its attributes, with no native input, output or model fields.

The trace is cut at the gateway

Incoming context is read. The traceparent the client puts in params._meta, following the specification’s propagation proposal, is extracted and turned into a link on the span, never into its parent. The reason is documented in the code: parenting to the remote trace would leave the span hanging off a trace whose root never reaches the backend. The client’s baggage is discarded on purpose to avoid forging identity attributes.

Outgoing context is not propagated. There is no propagator injection call anywhere in the tree. The headers that go out towards the upstream MCP server are built with authentication and the authorised extra headers, and nothing else.

The operational consequence: there is no end-to-end trace. If the upstream MCP server is instrumented, its spans live in another trace and there is no way to correlate them automatically.

Auditing and the conflict with privacy

A tool call leaves a normal spend row, with call type call_mcp_tool, spend equal to the cost per query, model MCP: name, zero tokens in the three token columns, a dedicated mcp_namespaced_tool_name column, and the usual key, team, user and session fields.

And in the metadata column, in JSON, the complete structure of the call: name, arguments, result, server, namespaced name, MCP session identifier, authentication mode and server resource with the URL redacted.

The problem

That structure is written into the metadata column without passing through the privacy switch. The assignment is direct and is not conditioned by the function that governs whether prompts and responses are stored, which does control the messages and response columns.

And the redaction function does not cover it either: it only touches messages and response. A search for MCP in the redaction file returns nothing.

The practical result: an operator who enables turn_off_message_logging for GDPR keeps sending the full arguments of every tool to every logging destination they have configured. To the SIEM, to Langfuse, to S3.

If an argument carries a national ID number, an address, a case number or a medical record, it has been exported.

There is an aggravating factor that does not depend on LiteLLM. The revision of the specification in force introduces an extension that allows mirroring tool arguments into HTTP headers visible to load balancers, proxies and web application firewalls. The specification recommends not marking sensitive parameters that way, and it is a recommendation, and the upstream server decides.

The fit with the ENS

The conflict is real and has a familiar shape: activity logging (op.exp.8) requires the call to be traced and non-repudiable; protection of personal data (mp.info.1) requires that log not to become a secondary repository of personal data with a different purpose. The tool argument is simultaneously the evidence of the action and the data. Redacting it destroys non-repudiation; storing it in the clear creates the repository.

The defensible way out you can build today: a hash of the argument in the audit log, the argument in the clear with encryption (mp.info.3), separate retention and its own access control. The hash sustains non-repudiation and the clear text is only opened under procedure.

There are two other measures this material touches directly. Change management (op.exp.5) because registering an MCP server and, above all, changing the description of an already approved tool are changes to the security configuration. And maintenance and updates (op.exp.4) because, as you can see in the next section, an old version of the gateway is a non-compliance with a name and a number.

Two honesty warnings. The codes follow the numbering of Annex II of RD 311/2022 and you have to check them against the text in force before citing them in a compliance document. And on NIS2: as of the date of this post the directive is still not transposed in Spain, with the draft bill on cybersecurity coordination and governance going through the process and a reasoned opinion from the Commission over the delay. What is enforceable today in the public sector and its suppliers is the ENS.

What is not logged

An administrative gap: registering, modifying and deleting MCP servers generates no audit log. There are three TODO comments in the management code that say so, one of them with an if is_audit_logging_enabled(): pass. In the same deletion block there are two more TODOs about not cleaning up orphaned permissions on keys and teams.

Security

The vulnerability you need to know about

CVE-2026-42271, published on 8 May 2026. Affects LiteLLM from 1.74.2 up to versions prior to 1.83.7. CVSS 3.1 of 8.8.

The POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list endpoints, which exist to preview an MCP server before saving it, accepted the stdio transport’s command, args and env in the request body. A low-privilege key executed arbitrary commands on the proxy host. It also affects Red Hat OpenShift AI on several branches.

It is the story of this post condensed into one line: the gateway you put in to contain MCP was the hole.

It does not come alone. The 1.83 branch concentrates three more serious ones:

CVE	Component	CVSS	Fixed in
`CVE-2026-42208`	SQL injection in key validation, with no prior authentication	9.8	1.83.7
`CVE-2026-35030`	OIDC userinfo cache keyed on `token[:20]`, collision and impersonation	9.1	1.83.0
`CVE-2026-40217`	Remote execution through bytecode rewriting in the guardrails test endpoint	8.8	no version recorded

Plus two privilege escalations fixed in 1.83.10 and 1.83.14.

The operational conclusion is short: a deployment below 1.83.14 is exposed to remote execution or to an unauthenticated SQL injection. The current branch is 1.102.0.

And a note for context: the whole MCP ecosystem has the same pattern. The unauthenticated remote execution in MCP Inspector, the command injection in mcp-remote when connecting to an untrusted server, DNS rebinding disabled by default in the Python and TypeScript SDKs until late 2025, the directory escapes in the reference servers. Connecting to an untrusted MCP server compromises the client, not just the other way round.

The rug pull nobody covers

Documented since April 2025: a server can change a tool’s description after the client has approved it. The description is what the model reads to decide when and how to use it, so changing it is changing the behaviour without asking for permission again.

The specification in force does not address it. It defines list change notification and cache freshness fields, which are recency mechanisms, not integrity ones. The closest normative warning is that clients must treat tool annotations as untrusted unless they come from trusted servers.

LiteLLM does not cover it either. The only “pinning” in the code is of the configuration’s server identifier, so that it does not change when you edit the YAML. There is no hash or comparison of the description or the input schema between successive listings. A search for related terms in the MCP module: nothing functional.

The control exists outside the gateway. mcp-scan, from the same researchers who documented the attack, implements description hash pinning precisely to detect this.

In ENS terms it is an unauthorised change to the security configuration that today goes undetected. And the home-made fix fits in a cron job: store the hash of the description and schema of every approved tool, compare against tools/list periodically, alert on the difference.

Three more holes in the code

No URL validation towards the upstream. A new server’s url field has no validator. The SSRF protection utilities exist in the tree and are used in OAuth discovery and in downloading OpenAPI specifications, but not in the egress of tools/call. A server pointing at a private range or at the cloud metadata IP is accepted as is.

The stdio transport is local execution on the proxy host. There is a command allowlist: npx, uvx, python, python3, node, docker, deno, extendable through an environment variable. The code’s own comment admits the residual risk: the permitted runtimes can execute code through arguments. With docker and npx in there, it is de facto arbitrary execution for any proxy administrator.

Two server modes skip the gateway’s authentication. Pass-through and authentication delegation to the upstream completely omit virtual key validation. They are well hardened (fail-closed if the target is mixed or unresolvable, strict comparison against truthy values, explicit blocking of the client credentials flow so as not to lend out the proxy’s service account), and the result is still that those servers have no gateway access control.

The limit that truncates in silence

The tools/list pagination stops and returns what it has accumulated in three cases: repeated cursor, one thousand pages, or expiry of a global deadline. The docstring declares it: a slow or faulty upstream produces a partial catalogue instead of an error.

The first two log a warning. The third, the deadline, exits without logging anything at all. A slow server makes part of its tools disappear from the catalogue with no trace left behind.

Correction on the slot leak

In the previous post I took issue 34534 at face value, according to which every MCP tool call acquires a max_parallel_requests slot and does not release it. Tracing the 1.102.0 code, that diagnosis no longer holds: the release path exists, on both success and failure, and the limiter is registered as a litellm callback and not just as a proxy hook.

With one honest caveat: I found no regression test covering the full cycle for MCP, and the definitive confirmation is a real proxy with max_parallel_requests: 2 and five MCP calls in a row. If anybody reproduces it today, I would like to know.

The specification moved underneath

The MCP revision in force is 2026-07-28, published on 28 July 2026, and in between there was another one, 2025-11-25. The changes in the latest one are not cosmetic:

Protocol sessions and the session header are removed. Cross-request state becomes an explicit identifier in the tool arguments.
The initialisation handshake is removed. Every request carries version and capabilities in _meta. MCP becomes stateless.
server/discover appears as a mandatory call.
Elicitation changes shape: out go server-initiated requests, in comes a multi-round pattern where the server returns an “input required” result and the client retries with the answers.
ping, logging/setLevel and the roots change notification are removed. SSE stream resumption is removed.
Standard HTTP headers are mandatory on POST, which lets a gateway or a web application firewall decide without parsing the JSON.
On authorisation: mandatory issuer validation in the client, and dynamic client registration is deprecated in favour of client identifier metadata documents.
Roots, sampling and logging are deprecated, with a minimum window of twelve months.

LiteLLM 1.102.0 advertises 2025-06-18. Its version enumeration has three values and none of them is later than that date. The literal is sent in the initialize against the upstream.

One nuance that is not entirely bad: the version the gateway advertises to its clients is not hard-coded in LiteLLM, the SDK negotiates it. And there is partial implementation of the intermediate revision, because the elicitation and sampling handlers reference it in their docstrings. That is: partial implementation of one revision while advertising an earlier one.

In practical interoperability it does not hurt yet, because the client ecosystem is still mostly on the 2025 revisions. It hurts in planning: July’s stateless redesign invalidates the gateway’s session model, and that is work ahead.

Two ecosystem notes for anyone choosing. There are clients that speak remote streamable HTTP natively (Claude and Claude Code, ChatGPT with OAuth and dynamic registration, VS Code with Copilot, Cursor, Zed, Kiro) and others that still demand stdio or a local bridge with mcp-remote, among them Windsurf, Cline and Continue.

Tool execution from the proxy itself

Besides serving MCP outwards, the proxy can consume it itself. It is enabled with an mcp block in the body, with server_url equal to the litellm_proxy sentinel, and it works both on /v1/responses and on /chat/completions.

The flow: it separates the gateway’s tools, lists them, filters by allowed, deduplicates, translates them into OpenAI function format and concatenates them. It calls the model. If there are tool calls and auto-execution is permitted, it runs them against the MCP servers. It builds the follow-up and makes one second call.

Two limits you need to know before designing on top of this.

The loop is a single hop. There is no multi-step iteration. The calls from the first response are executed and the follow-up is returned. A plan that chains tools requires the client to call again.

Auto-execution fails closed and as a block. It requires all MCP references to have approval set to “never”; a single one requiring approval disables auto-execution for the whole request.

And a third one, which is the one that bites: the multi-turn state of /v1/responses is rehydrated from the spend table. The query is literal, it looks up the session of the previous request_id and retrieves all the rows of that session ordered by time.

With disable_spend_logs: true nothing is written, the query comes out empty and the history is lost in silence, with no 4xx error. The code itself acknowledges it: it does not expect retries because deployments that do not write spend logs have nothing to wait for. It is a coupling between a performance option and a conversation feature that nobody expects to find.

Health and day-to-day operation

Health checks exist, they are on demand and they are not periodic. GET /v1/mcp/server/health accepts repeated identifiers and returns per-server status. The implementation opens a client and runs an empty operation inside the session, with a ten-second timeout.

It skips the check entirely if the server requires per-user authentication, and those always return unknown status.

The surprising part: the status, last check and error columns have existed in the database since a May 2025 migration, and the check function returns an object without writing to the database. Status is recalculated on every request. There is no periodic MCP health job.

When an upstream server is down, the behaviour is to degrade and not block. The listing fans out in parallel and each failing server contributes an empty list plus a result classified into six categories. Those results reach the client in the response’s _meta under a vendor key, without filtering the upstream’s text or URL. The reason for the design is in the code: an empty contribution with no signal makes a broken upstream indistinguishable from a healthy server with no tools.

There is no cache of the previous list. A server that is down disappears from that turn’s tools.

On registering and deregistering without restarting, there are two worlds. Database servers can be created, edited and deleted through the API, and every operation reloads the registry with an atomic swap. Propagation between replicas goes through Redis publication on the configuration change channel, if there is Redis, and through polling every 30 seconds if there is not.

Servers declared in config.yaml are loaded once, at start-up. There is no re-read. Changing them requires a restart.

And there is a precedence trap between the two worlds, with two different warnings in the code. A database row with the same identifier as a configuration entry hides the configuration one entirely. And the other way round, a configuration identifier that matches the name or alias of a database server captures its permissions, because identifiers are resolved before names. Both cases emit a log warning, once.

A hardened `config.yaml`

general_settings:
 # Access is not open by default, but this closes the empty-key
 # inheritance towards the team, which is the one that surprises people.
 require_key_mcp_access_defined: true
 require_end_user_mcp_access_defined: true
 # Without this, allow_all_keys overrides the key's explicit scope.
 mcp_allow_all_keys_respects_mcp_scope: true

litellm_settings:
 # Does not cover MCP arguments. See the auditing section.
 turn_off_message_logging: true
 callbacks: ["langfuse_otel"]

mcp_servers:
 inventario:
 url: https://mcp-inventario.interna.svc/mcp
 transport: http
 auth_type: bearer_token
 auth_value: os.environ/MCP_INVENTARIO_TOKEN
 # Explicit allowlist. Also enforced on tools/call.
 allowed_tools:
 - buscar_articulo
 - consultar_stock
 # Never true on a server with data.
 allow_all_keys: false
 access_groups: ["operaciones"]
 timeout: 20
 max_concurrent_requests: 8
 mcp_info:
 description: "Inventario de almacen, solo lectura"
 # Without this, MCP spend is zero and its metric does not exist.
 mcp_server_cost_info:
 default_cost_per_query: 0.0005
 tool_name_to_cost_per_query:
 consultar_stock: 0.0002

And in the process environment:

# Without this there is no MCP span, only a generic litellm_request.
LITELLM_OTEL_V2=true
# Full block hashes, not truncated, if correlating
# with engine events.
LITELLM_MCP_CLIENT_TIMEOUT=45
LITELLM_MCP_TOOL_LISTING_TIMEOUT=20

For clients, the rule is not to use the aggregated endpoint. Each agent points at /{server}/mcp or at a tool set, or at the three-tool proxy mode if the catalogue is large.

Checklist

Check the version. Below 1.83.14 there is remote execution and an unauthenticated SQL injection with an assigned CVE.
Inventory which servers have allow_all_keys and enable the setting that makes it respect the key’s scope.
Audit the access groups separately, because they are additive on top of the key and team ceiling.
Declare in writing the fail-open behaviour on tool permission resolution errors.
Measure the real cost: number of tools per server from the listing row, and tools block tokens from the stored request body.
Choose a catalogue strategy before connecting the third server: three-tool proxy mode, semantic filter, or selection by route. Not the aggregated endpoint.
Declare mcp_server_cost_info even with a token value, or the spend metric will not exist.
Enable the OTel v2 logger, and decide deliberately whether to enable content capture.
Assume the trace is cut at the gateway and plan correlation with the upstream by other means.
Treat the spend table’s metadata column as a personal data repository: encryption, its own retention and separate access control.
Set up the rug pull check, because nobody does it: hash of the description and schema of every approved tool, periodic comparison against the listing.
Restrict egress towards MCP servers at the network level, because the gateway does not validate the destination URL.
Do not enable the stdio transport unless you need it, and if you do, review what is on the command allowlist.
If you use /v1/responses with multi-turn state, do not enable disable_spend_logs.

Traps and things that are not what they seem

MCP access is not open by default in 1.102.0. I got that wrong in the previous post. What opens things up is per-server allow_all_keys.
allow_all_keys overrides the key’s explicit scope unless you enable a setting that ships disabled.
require_key_mcp_access_defined only closes key-to-team inheritance. Access group grants remain additive.
On an indeterminate failure resolving tool permissions, the code fails open for keys and JWTs.
The default cost of an MCP call is 0.0, and the spend metric is only emitted if the cost is greater than zero, so the series does not exist.
MCP calls appear in the per-team report as if they were models, named MCP: tool and with zero tokens.
The OTel logger with MCP support is off by default. Without the environment variable there is no MCP span.
Trace context is not propagated towards the upstream MCP server. There is no end-to-end trace.
mcp.tool.name does not exist in the semantic conventions. The name goes in gen_ai.tool.name.
Message redaction does not cover tool arguments. They are written in the clear into the spend table.
Registering, modifying and deleting MCP servers generates no audit log. There are three TODOs in the code.
The semantic tool filter is not applied on tools/list. Only on /chat/completions and /v1/responses.
The catalogue order is not deterministic, because it comes from iterating a set of strings. It breaks the client’s prompt cache even though nobody filters anything.
search_tools in proxy mode counts keywords by default, it uses neither regular expressions nor embeddings, and its cap of five results is hard-coded.
Description overrides really do replace what the client sees, and they are the only catalogue lever that does not break the cache. They are only configurable through the database or the API.
spec_version no longer exists in the configuration schema.
/mcp/sse does not use the SSE transport, and the messages endpoint its object advertises is dead code.
Listing pagination truncates in silence when the deadline expires, without leaving a single log line.
A database row hides a configuration entry with the same identifier, and the other way round a configuration identifier can capture the permissions of a database server.
Servers declared in config.yaml require a restart. Only the database ones reload hot.
The health check does not write its result to the database even though the columns exist, and there is no periodic job.
The gateway does not validate the upstream MCP server’s URL. The anti-SSRF utilities exist and are not used on that path.
docker and npx are on the stdio command allowlist.
The /v1/responses tool loop is a single hop, and disable_spend_logs: true breaks its multi-turn behaviour with no error.
LiteLLM advertises the June 2025 revision of the protocol, two behind the one in force.

Closing

There is a thread running through the six posts in this track that is more visible here than anywhere else. The gateway is chosen for one job and ends up doing five, and each new job arrives with its own defaults, which are almost never the ones you would have chosen.

In the case of MCP the asymmetry is twofold. On one side, the part that looks dangerous is better built than you would assume: access control is enforced on execution and not just on the listing, the hierarchy is built out of intersections, header forwarding is an allowlist, and default access is closed, contrary to what I myself wrote two days ago. On the other, the part nobody looks at is worse: tool arguments come out in the clear bypassing the privacy switch, spend is worth zero and its metric never gets to exist, the trace is cut, and a change to the description of an approved tool goes undetected by anybody in the whole chain, neither the specification nor the gateway.

The architecture decision that comes out of this is that the MCP door has to be treated as what it is, an interconnection of systems with third parties, and not as one more function of the LLM proxy. That means its own inventory, formal registration, change control over every tool’s description, network isolation towards the upstream and a log designed knowing that it is going to contain personal data.

And it means accepting that the cost you really pay for connecting fifteen MCP servers is in no column of the spend table. It is spread across the prompt_tokens of every model call, on every turn, and across the accuracy you lose when the catalogue is too large for the model to choose well. That number can be computed with what the gateway already stores. The first step is wanting to compute it.

Sources

LiteLLM, MCP Gateway: https://docs.litellm.ai/docs/mcp.
LiteLLM, código: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, auth/user_api_key_auth_mcp.py, cost_calculator.py, faults/list_outcomes.py, litellm/proxy/management_endpoints/mcp_management_endpoints.py, litellm/proxy/spend_tracking/spend_tracking_utils.py, litellm/litellm_core_utils/redact_messages.py, litellm/integrations/otel/, litellm/responses/litellm_completion_transformation/session_handler.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en llamadas MCP: https://github.com/BerriAI/litellm/issues/34534.
NVD, CVE-2026-42271, ejecución de comandos por los endpoints de prueba de MCP: https://nvd.nist.gov/vuln/detail/CVE-2026-42271.
NVD, CVE-2026-42208, inyección SQL sin autenticación en la validación de clave: https://nvd.nist.gov/vuln/detail/CVE-2026-42208.
NVD, CVE-2026-35030, colisión de clave de caché de userinfo OIDC: https://nvd.nist.gov/vuln/detail/CVE-2026-35030.
NVD, CVE-2025-49596, ejecución remota en MCP Inspector: https://nvd.nist.gov/vuln/detail/CVE-2025-49596.
NVD, CVE-2025-6514, inyección de comandos en mcp-remote: https://nvd.nist.gov/vuln/detail/CVE-2025-6514.
NVD, CVE-2025-66416, protección de DNS rebinding desactivada por defecto en el SDK de Python: https://nvd.nist.gov/vuln/detail/CVE-2025-66416.
Invariant Labs, MCP Security Notification: Tool Poisoning Attacks (tool poisoning, rug pull, shadowing): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks.
Invariant Labs, Introducing mcp-scan (fijado de herramientas por hash): https://invariantlabs.ai/blog/introducing-mcp-scan.
Model Context Protocol, changelog de la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/changelog.
Model Context Protocol, changelog de la revisión 2025-11-25: https://modelcontextprotocol.io/specification/2025-11-25/changelog.
Model Context Protocol, Security Best Practices (confused deputy, indicadores de recurso, validación de audiencia): https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.
NSA, Model Context Protocol: Security Design Considerations for AI-Driven Automation, mayo de 2026: https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDF.
CISA y agencias aliadas, Careful Adoption of Agentic AI Services, mayo de 2026: https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services.
Anthropic, Advanced tool use (coste en tokens de las definiciones, búsqueda de herramientas, precisión de selección): https://www.anthropic.com/engineering/advanced-tool-use.
Anthropic, Code execution with MCP: https://www.anthropic.com/engineering/code-execution-with-mcp.
Anthropic, Prompt caching (jerarquía tools → system → messages e invalidación): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.
Anthropic, Tool search tool (herramientas diferidas fuera del prefijo, umbral de 30 a 50 herramientas): https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, Prompt caching (el orden de las herramientas cuenta): https://developers.openai.com/api/docs/guides/prompt-caching.
OpenAI, Tool search (las herramientas descubiertas se cargan al final del contexto): https://developers.openai.com/api/docs/guides/tools-tool-search.
vLLM, Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html.
Microsoft, Use tools in chat (límite de 128 herramientas por petición en VS Code): https://code.visualstudio.com/docs/copilot/agents/agent-tools.
RAG-MCP, prueba de estrés de 1 a 11.100 servidores candidatos, mayo de 2025: https://arxiv.org/abs/2505.03275.
Benchmarking the Benchmarks, desalineación del 18,5 % entre etiqueta y realidad, junio de 2026: https://arxiv.org/html/2607.02577v1.
MCP-Atlas, taxonomía de fallos sobre 220 herramientas y veinte modelos, febrero de 2026: https://arxiv.org/html/2602.00933v3.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307.
MCP Tool Descriptions Are Smelly, auditoría de 856 herramientas de 103 servidores, febrero de 2026: https://arxiv.org/html/2602.14878v1.
ToolMenuBench, seis estrategias de filtrado y 26.460 ejecuciones, junio de 2026: https://arxiv.org/html/2606.15508.
Retrieval Models Aren’t Tool-Savvy (ToolRet), corpus de 43.000 herramientas, marzo de 2025: https://arxiv.org/abs/2503.01763.
Tools Are Not Islands, brecha entre Recall@3 y conjunto completo, julio de 2026: https://arxiv.org/html/2607.25718.
ToolFlood, saturación adversaria de la capa de recuperación, marzo de 2026: https://arxiv.org/html/2603.13950.
Scores Are Not Decisions, el K óptimo como función del coste, julio de 2026: https://arxiv.org/html/2607.27083v1.
Don’t Break the Cache, caché de prompts en tareas agénticas de horizonte largo, enero de 2026: https://arxiv.org/html/2601.06007v2.
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer, mayo de 2026: https://arxiv.org/html/2605.24660v1.
Semantic Tool Discovery for LLMs, marzo de 2026: https://arxiv.org/abs/2603.20313.
OpenTelemetry, convenciones semánticas de GenAI y MCP: https://github.com/open-telemetry/semantic-conventions-genai.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
IBM ContextForge, gateway y registro MCP: https://github.com/IBM/mcp-context-forge.
Docker MCP Gateway: https://github.com/docker/mcp-gateway.
agentgateway: https://github.com/agentgateway/agentgateway.

Model chain of trust (4/4): who serves the model and which machine you trust

Sun, 26 Jul 2026 17:00:00 +0200

The three previous articles closed questions about the artefact: what contract the served model exposes, where its bytes come from and why you trust them. What remains is the one almost nobody asks until an auditor asks it: nothing proves that the process currently returning tokens on port 8000 is the inference engine and not something that slipped into the cluster and inherited its token. And a second one, worse: whether the machine it runs on is trustworthy, and against whom.

These are different problems with different projects. Workload identity is solved by SPIFFE/SPIRE, a CNCF graduated project since 2022. Environment isolation is covered by Kata Containers and Confidential Containers, the latter promoted to CNCF incubating on 22 July 2026, four days before this article was published. The first is cheap and almost always worth it. The second is expensive, more than the vendor’s press release suggests, and is only justified against a specific threat model.

TL;DR

User identity ≠ workload identity. OIDC says which person is behind a request; it does not say which process serves it. ServiceAccount tokens, static keys and gateway API keys are bearer credentials: whoever holds them, is them.
SPIFFE defines a URI identifier (spiffe://domain/path) and a verifiable document (SVID, X.509 or JWT) delivered by a Workload API that does not require the workload to hold any prior secret. SPIRE implements it with node and workload attestation; defaults: default_x509_svid_ttl 1 h, default_jwt_svid_ttl 5 min, ca_ttl 24 h.
Identity is only worth something if something enforces it: Istio/Envoy via SDS, Cilium with mutual authentication (with serious reservations) or an ext_authz with OPA in front of the gateway.
Three rungs of isolation: container with a shared kernel → Kata Containers (its own kernel and VM; v3.32.0 from June 2026) → Confidential Containers (hardware TEE, hypervisor outside the trust base, remote attestation with Trustee per RFC 9334).
The overhead of confidential mode runs from ~0 % to ~28 % depending on what you measure: GPU-only in CC with large batches sits around 4-8 %, while the full stack (a CVM with Intel TDX plus an H100 in CC), measured independently in 2026, gives +21.8 % to +27.8 % on TTFT and −17.7 % to −21.1 % on throughput. Honest heuristic: reserve 15-25 % extra capacity.
MIG and vGPU are forbidden in CC mode according to NVIDIA’s reference architecture, and every GPU on a node goes in the same mode.
Closing criterion: the order is contract → digest → signature verified at admission → workload identity → TEE.

The analogy: the biosafety laboratory

A laboratory handling pathogens solves two problems much like ours, separately.

The first is who gets in. The company badge is not enough: at the airlock there is a specific credential, short-lived and automatically renewed (if lost, it expires on its own within an hour), issued only after independently checking that the applicant is where they say they are and is who they say they are. That credential is the SVID; the prior check is attestation.

The second is which room you work in, and here there are containment levels. On the open bench the air is shared and anything aerosolised affects everyone: the normal container, with a kernel shared by all the neighbours. One rung up, the biological safety cabinet, with a physical barrier and its own airflow: Kata Containers. At the very top, the maximum containment laboratory, where maintenance staff change filters without ever seeing the sample: the TEE, where the operator maintains the machine but cannot read its memory. And before opening the airlock somebody verifies that the integrity of the suit and of the room is as expected: remote attestation with conditional secret release.

The uncomfortable lesson: the maximum containment level is a cost decision before it is one more degree of security. Nobody builds a BSL-4 to grow yeast. The useful question is who it isolates from, not how much.

Part A — Workload identity

Six services talking to each other and none of them knows who the other is

In a serious inference cluster, like the seven layers one, the gateway talks to the serving engine, the engine queries the vector database, the trace collector receives spans from everyone and the agents with MCP invoke tools that in turn call the gateway. Many east-west connections and, in most deployments, none of them authenticated.

User identity says which person is behind the request and is resolved with OIDC, as when putting Keycloak in front of MCP. Workload identity says which process issues it: there is no human on the other side, no browser, no consent, and the process is born and dies in seconds. Its three usual substitutes fail for different reasons:

Mechanism	Where it fails
Projected `ServiceAccount` token	SA granularity, not workload granularity: two pods with the same SA are indistinguishable. It does not cross the cluster boundary and does not federate. It is a bearer token on the filesystem
Static key	It does not rotate, it gets shared over Slack, it does not distinguish issuer from bearer; once leaked, it is valid until somebody notices
Gateway API key	It authenticates the client, not the process. LiteLLM’s virtual keys are for quota and budget, not for proving identity

There is no cryptographic link between credential and process: copy the credential, copy the identity. In a multi-tenant cluster like the H100 cluster one, real isolation then depends on network topology and not on identity.

SPIFFE: the standard

SPIFFE and its implementation SPIRE graduated together in the CNCF on 20 September 2022. As of July 2026 they are settled infrastructure, not a bet.

The SPIFFE ID is a URI of the form spiffe://<trust-domain>/<workload-identifier>, for example spiffe://inferencia.example/ns/inferencia/sa/vllm: a name, not a credential. The trust domain is the root of trust (organisation, environment or site), and the guidance recommends separating workloads from different sites or security environments into different domains.

The SVID is the credential that proves that ID. The X509-SVID carries the SPIFFE ID in the URI-type SAN, not in the CN, which the specification discourages as a source of identity, and it is the preferred format. The JWT-SVID exists for when there are L7 proxies terminating TLS in the middle, with the replay risk the documentation warns about:

Certificate:
Issuer: C=ES, O=SPIFFE
Validity
Not Before: Jul 26 08:00:00 2026 GMT
Not After : Jul 26 09:00:00 2026 GMT
Subject: C=ES, O=SPIRE, CN=vllm.inferencia
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
URI:spiffe://inferencia.example/ns/inferencia/sa/vllm

One hour of validity, and what authenticates is the SAN URI: the CN is decorative.

The Workload API is the elegant piece: a local UNIX socket from which the workload obtains its SVID, its private key and the trust bundle, without presenting any secret. The documentation is explicit: “the Workload API does not require that a calling workload have any knowledge of its own identity, or possess any authentication token”. That solves the cold start of all identity cryptography: the secret you would need in order to obtain the first secret.

SPIRE: how that document is issued

SPIRE has a server (the authority that signs) and one agent per node (which exposes the Workload API), and it chains two checks.

Node attestation verifies that the agent runs where it says it does. In Kubernetes the reference attestor is k8s_psat, which validates a projected token with an audience against the API server; on bare metal there are TPM-based attestors. A note of honesty: SPIRE v1.15.1, from 28 May 2026, is a security patch over an incorrect PKCS7 validation in the azure_imds attestor that allowed attested documents to be forged and a virtual machine to be impersonated.

Workload attestation answers the opening question. The k8s attestor receives the PID of the process that opens the socket, infers the pod from its cgroup membership and queries the kubelet for the metadata. From there come the selectors: namespace, service account, pod name and UID, labels, owner, node and, above all, container and image by tag or by digest. From SPIRE v1.15.0 (19 May 2026) Sigstore support stopped being experimental, enabling selectors by signature verification status, certificate subject and issuer, and transparency log. That closes the series’ circle: you can require that the engine’s SVID go only to the container whose image carries a valid cosign signature from the expected issuer, which is what was verified in article 3/4. Provenance goes from being a one-off admission check to a precondition of identity at runtime.

spire-server entry create \
 -parentID spiffe://inferencia.example/spire/agent/k8s_psat/prod/nodo-gpu-a \
 -spiffeID spiffe://inferencia.example/ns/inferencia/sa/vllm \
 -selector k8s:ns:inferencia \
 -selector k8s:sa:vllm \
 -selector k8s:container-name:vllm \
 -selector k8s:container-image:vllm/vllm-openai@sha256:aa11bb22cc33 \
 -x509SVIDTTL 3600 \
 -federatesWith spiffe://sede-b.example

The selectors are conjunctive: a pod in another namespace, with another SA or with another image does not obtain that SVID even if it shares the node.

The default TTLs have three consequences. Rotation is continuous, not a one-off event: code that loads the certificate once at startup will break exactly one hour later. Server downtime has a clock: twenty minutes are invisible, two hours bring down the cluster’s authenticated communications, and the scaling guide admits that “a single SPIRE Server instance represents a single point of failure”, with the datastore as the bottleneck and indicative figures running from two replicas of 1 CPU and 1 GB for 10 agents to eight of 16 CPU and 16 GB for 5,000 agents and 10,000 workloads. And tag-based selectors are fragile: the runtime may report one tag or another depending on the node and the moment, so the correct selector is the digest, as in article 2/4.

Federation between trust domains

Two sites with independent clusters, or an external partner exposing a reranker, should not share an authority. SPIFFE federation makes each domain publish a bundle endpoint with its public trust material and the other poll it periodically: the https_web profile authenticates it with web PKI and https_spiffe with an X509-SVID from the domain itself, enabling automatic rotation and revocation of the root. The relationship is one-directional, bundles from different domains must never be merged and refresh is by polling with a default suggestion of five minutes. In practice, site A’s gateway accepts requests from site B’s engine without sharing a CA or an identity database, and cutting the relationship means deleting a bundle, not revoking certificates.

From identity to effective authorisation

An SVID blocks nothing by itself; somebody has to compare the presented ID against a policy.

Istio + SPIRE is the best-worn integration: Istio detects a UNIX socket implementing Envoy’s SDS API and the proxy obtains its identities from there instead of from istiod, mounted with the SPIFFE CSI Driver (recommended over hostMount). Two conditions break deployments: SPIRE’s trust domain and Istio’s must match exactly, and SPIRE only issues to previously registered workloads, including Istio’s own components. With that, policy is written in terms of identity and not IP:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: vllm-solo-desde-gateway
 namespace: inferencia
spec:
 selector:
 matchLabels:
 app: vllm
 action: ALLOW
 rules:
 - from:
 - source:
 principals: ["inferencia.example/ns/inferencia/sa/gateway"]
 to:
 - operation:
 methods: ["POST"]
 paths: ["/v1/chat/completions", "/v1/completions"]

It is a layer on top of the NetworkPolicy from layered hardening: there we closed communication pairs by topology; here we also require who calls and on what path.

Cilium offers mutual authentication backed by SPIRE, tempting if you already use eBPF for the data network. It deserves criticism: it is still marked beta in the stable documentation, and The New Stack published a reasoned critique arguing that the design does not preserve mTLS properties over the life of the connection (it uses the handshake only to authenticate and discards the session keys) and that the identity model rests on eventually consistent per-node IP-identity caches, which can allow traffic that policy should deny.

The inference gateway and LiteLLM do not speak SPIFFE natively as of this article, and saying so is better than suggesting an integration that does not exist. The pattern that does work is the one the project itself documents: an Envoy in front that terminates mTLS with an X509-SVID or validates a JWT-SVID and delegates to OPA via ext_authz. The gateway carries on with its own job (quota, budget, routing), as we saw when choosing an OSS gateway and in the L7 router; workload identity is resolved one hop earlier.

AI agents and MCP: the missing piece

An agent invoking tools over MCP is a shared secret with legs: it is handed a static credential, the MCP server cannot know whether the caller is the legitimate agent or any process holding the same string, and the blast radius is the entire set of exposed tools. The threat model for agent isolation warned that a granted permission is a usable permission; with bearer credentials it is also transferable.

SPIFFE provides what is missing: the agent presents a short-lived SVID obtained through attestation and the MCP server authorises against the SPIFFE ID. In 2026 this stopped being theory: there is work at the IETF on dynamic OAuth client registration based on trusted SPIFFE issuers, and in June 2026 Google published a SPIFFE-based Agent Identity in its IAM. None of it is fully standardised.

Two honest caveats. SPIFFE answers “who”, not “why”: to know whether the action is the one the user asked for you need traceability (MCP with OTel) and runtime detection with Tetragon. And both identities must be composed: the agent proves its own with an SVID and propagates the delegating user’s in the token, authorising on the pair. Collapsing them into one is how you end up with agents able to do, in the system’s name, things no user could.

Part B — Isolation of the execution environment

### Kata Containers: the virtual machine sandbox

Kata replaces runc with a runtime that boots a lightweight virtual machine per pod, with its own kernel and a kata-agent inside; for Kubernetes it is transparent via RuntimeClass. As of July 2026 the stable branch is 3.32.x (3.32.0 is from June 2026, with Rust 1.94, Go 1.25.11, QEMU 11.0.1, guest kernel 6.18.35 and containerd 2.3), and in April 2026 the 4.0.0 preview was published, making the Rust runtime (runtime-rs) the default and leaving the Go one deprecated until 5.0.0, because of memory footprint.

For GPUs the route is VFIO passthrough. The NVIDIA GPU Operator automates it with sandboxWorkloads.enabled=true and sandboxWorkloads.mode=kata, installing the VFIO Manager, the Sandbox Device Plugin, the Confidential Computing Manager and the Kata Manager. It requires virtualisation and ACS in the BIOS, IOMMU, kata-deploy 3.29.0 or later, containerd (not CRI-O), the KubeletPodResourcesGet feature gate (on by default from Kubernetes 1.34) and removing the NVIDIA drivers from the host: the GPU is handed over whole to the VM and the guest manages it.

apiVersion: v1
kind: Pod
metadata:
 name: vllm-kata
 namespace: inferencia
spec:
 runtimeClassName: kata-qemu-nvidia-gpu
 containers:
 - name: vllm
 image: vllm/vllm-openai@sha256:aa11bb22cc33
 resources:
 limits:
 nvidia.com/pgpu: "1"

Two measurable costs. Memory: the VM reserves its own and you have to declare it as overhead in the RuntimeClass so the scheduler does not overcommit the node. And startup: according to the SESAME'24 study on serverless confidential containers, cold start goes from around 6 s with runc to around 7 s with Kata, and warm start from 1 s to 2 s. Acceptable for an engine that takes minutes to load weights; unacceptable for ephemeral functions.

Confidential Containers: the TEE

Confidential Containers (CoCo) takes the hypervisor and the platform operator out of the trust base. The CNCF promoted it to incubating on 22 July 2026, with more than 150 active contributors, 26 repositories and over 1,200 merged pull requests, and with Microsoft Azure, Intel, AMD, IBM, NVIDIA, Alibaba and Red Hat behind it. A serious project, but incubating means it is not graduated and that its integration surface changes between versions.

The components are the CoCo pods (unmodified containers run in a TEE via Kata) and Trustee: KBS (Key Broker Service, which releases secrets conditionally), AS (Attestation Service, which validates the hardware evidence) and RVPS (Reference Value Provider Service, which holds the reference values). Inside the guest, the Attestation Agent collects the evidence and the Confidential Data Hub consumes the secrets.

The flow is a clean instance of the RATS architecture from RFC 9334: the agent inside the TEE is the attester, the Attestation Service the verifier and the Confidential Data Hub the relying party. The evidence is a hardware-signed report with the boot measurements; the AS validates it against the RVPS values; and only if it matches does the KBS release the key that decrypts the image layers or the weights. Without valid attestation there is no key, and without a key there is no model: that is what stops the operator extracting it.

A mapping warning: the confidential-containers/operator repository was archived in February 2026 and deployment moved to the Helm charts and the trustee-operator. As of July 2026 the latest chart release is v0.21.0, aligned with kata-deploy 3.31.0, and Trustee v0.20.0, with TLS 1.3 and post-quantum cryptography, external plugins for the KBS and multi-GPU support via Intel Trust Authority and NVIDIA NVSwitch.

apiVersion: confidentialcontainers.org/v1alpha1
kind: TrusteeConfig
metadata:
 name: trusteeconfig
 namespace: operators
spec:
 profileType: Restrictive
 httpsSpec:
 tlsSecretName: trustee-tls-cert

The Permissive profile exists for development and should not leave it: it is the mode in which attestation does not block.

The hardware: CPU and GPU

On the CPU side the two relevant TEEs are AMD SEV-SNP and Intel TDX; NVIDIA’s reference architecture sets EPYC Milan/Genoa and Xeon Emerald/Granite Rapids as the baseline. Host support is no longer the hard part.

On the GPU side, the technical description of the design of the first confidential GPUs explains what CC mode on an H100 buys you. Memory is split into a Compute Protected Region that hardware firewalls stop both the CPU over PCIe and other GPUs over NVLink from accessing. Everything crossing the CPU-GPU boundary passes through bounce buffers encrypted with AES-GCM-256, and command buffers and CUDA kernels are encrypted and signed before crossing the bus. There is a chain of trust from GPU boot with a signed attestation report, and only NVIDIA-signed firmware runs in CC mode, validated against NRAS or locally in isolated environments. And the performance counters are disabled in hardware as a side-channel mitigation: that is the telemetry much of observability with DCGM depends on.

What it does not protect: the package’s HBM memory is not encrypted (it is considered resistant to interposers, which is a claim about physical difficulty and not a cryptographic guarantee); nothing against denial of service, because the operator who cannot read you can still switch you off; nothing against timing or access-pattern side channels; and nothing against a bug in your own code inside the enclave or prompt injection, which remains the job of guardrails.

On Blackwell, NVIDIA announces (2 July 2026) NVLink encryption that extends confidential computing to up to 8 GPUs, non-existent on Hopper and a condition for serving a large model with tensor parallelism. The reference architecture lists H100, H200, RTX Pro 6000 Blackwell Server Edition and B200 in single-GPU passthrough, and H100/H200 in PPCIe mode and B200 for multi-GPU.

The overhead numbers, with source and with judgement

Here it pays to distrust everybody, papers included. The published spread runs from 0 % to 28 % because the studies do not measure the same thing.

Source (date)	What it measures	Result	Nature
NVIDIA, Blackwell blog (Jul. 2026)	B200 with CC enabled	−1.0 % to −7.5 % throughput; per-token latency under 8 %; “up to 98 % of native performance”	Vendor claim
Corvex, HGX B200 with TDX (2026)	“Verified” deployment, encrypted NVSwitch	“Near-native performance”, with no figures of its own	Commercial claim
arXiv 2409.03992 (2024)	H100 GPU only in CC, Llama-3.1 8B/70B	−0.36 % to 6.85 % throughput; TTFT up to +19 %; overhead → 0 as the model grows	Independent, partial
arXiv 2509.18886 (Sep. 2025)	CPU TEE (TDX/SGX) and GPU TEE (H100), Llama2 7B/13B/70B	CPU TEE: under 10 % throughput and 20 % latency. GPU TEE: 4-8 %	Independent
arXiv 2607.19353 (May 2026)	Full stack: H100 in CC inside a CVM with TDX, Mistral-7B and Qwen3-30B-A3B under load	TTFT +21.8 % and +27.8 %; throughput −17.7 % and −21.1 %; closed-loop, 11.5-20.2 %	Independent, complete

The reconciliation: the penalty is in the data path, not in the GPU’s compute. NVIDIA itself quantifies the bottleneck: the effective bandwidth of the CPU-GPU interconnect in CC mode is limited by the CPU’s encryption throughput, “around 4 GB/s”. From there, three rules. The higher the compute/I-O ratio, the lower the overhead: a 70B with large batches and long sequences amortises the toll almost entirely, and a 7B with small batches and short prompts pays it in full. Adding the CPU TEE brings its own cost, and that is where most of the difference between 4-8 % and 20 % lies. And model loading and cold start are the worst cases, with tens of gigabytes crossing an encrypted bus at 4 GB/s, which makes everything discussed in from disk to HBM and in speeding up cold start more important.

The recommendation from the authors of the 2026 study, reserving between 15 % and 25 % extra capacity, is the figure I would take to a capacity planning exercise, and not the vendor’s “98 % of native performance”. Both can be true at once; only one is prudent for sizing.

When each rung is worth it

Scenario	Kata	CoCo (TEE)	Reason
Tenants that do not trust each other	Yes	Depends	Kata removes the shared kernel, a real escape vector
Unaudited third-party code or models	Yes	Optional	A shared kernel is too much surface for someone else’s code
Model intellectual property against the infrastructure operator	Not enough	Yes	The canonical case: third-party hosting, cloud, a partner providing the iron
Regulated third-party data on infrastructure you do not control	Not enough	Yes	See defence and healthcare
Sovereign on-premise, one tenant, trusted operator	Useful	Expensive over-engineering	It protects you from an adversary you do not have
Low latency with small batches	Yes	Bad idea	The worst point on the bus-encryption cost/benefit curve

The nuance that saves the most money: the two axes are independent. Kata without a TEE is cheap and delivers most of the isolation between tenants; CoCo only adds something if your threat model includes the operator. If you build a sovereign factory with your own iron and your own staff and you answer “no” to that question, the TEE is a cost with no return.

Operational traps

MIG and vGPU against confidential mode. NVIDIA’s GA 1.0.0 reference architecture is blunt: MIG and vGPU are forbidden in CC mode, and mixed-mode nodes are not supported; every GPU on a host goes in CC or none does. The forums show the typical contradiction of a moving area: in February 2026 a moderator states that MIG+CC is not supported and in April a user cites the MIG page suggesting that Hopper and Blackwell already allow it. As of this article the reference architecture overrides the product page: if your multi-tenancy rested on partitioning the GPU with MIG, enabling CC sends you back to “one whole GPU per tenant”.

Startup grows non-linearly. SESAME'24 breaks down where: SEV memory provisioning adds around one second per 2 GB of guest (SEV pins every page in advance, and an inference VM allocates a lot), the OVMF firmware about three seconds and layer decryption another three to five. Scaling from 0 to 16 instances goes from 16 s with runc to 190 s: irrelevant for a stable Deployment, a design change for aggressive autoscaling with KEDA.

Version coupling is brutal. Kata, kata-deploy, GPU Operator, containerd, QEMU with its own patches, guest kernel, NVIDIA driver and Trustee are a matrix that has to be treated as a unit: the reference architecture fixes Kubernetes 1.32 or later, Kata 3.29, GPU Operator 26.3.1 or later, containerd 2.2.2 or later and QEMU 10.1. Add it to the weight the stack’s operators already carry.

Attestation fails after a firmware update, and that is normal. The RVPS reference values describe a specific boot state. When you patch the SEV-SNP firmware, the microcode or the GPU firmware, the TCB version changes, the evidence stops matching and the KBS does not release keys, so the pods do not start. Since these updates usually respond to a security bulletin, the typical scenario is “we applied the critical patch on a Friday and on Saturday inference would not start”. The correct procedure reverses the order: update the reference values first with an overlap window, and patch the iron afterwards.

The attestation service is a single point of failure by design. If the KBS does not answer, no new confidential pod gets its key; those already started survive, but any rescheduling or scaling does not. The same dependency as the SPIRE server and the same treatment: real high availability, its own alert and rehearsed degradation.

Closing the series: the complete chain

Article 1/4 set the contract: what the endpoint exposes and what control plane governs it reproducibly. In 2/4 the subject was the provenance of the bytes: which registry they come from, by immutable digest and not by a moving tag. 3/4 added the cryptographic proof of what they are, with signatures, build attestations and an AIBOM verified at admission. This 4/4 closes with who runs them and where: an SVID that proves the serving process’s identity and an isolation rung chosen for the adversary you actually have. If one of the four fails, the other three are worth less than they look: an impeccably signed model served by an unidentified process on a machine a third party can dump is still a problem.

With a limited budget, the order follows cost per unit of risk removed. First, digests everywhere: almost free, and it removes a whole family of substitution attacks. Second, signature verified at admission, low cost and the first thing an auditor asks to see. Third, a declarative contract and control plane, which make everything else auditable. Fourth, workload identity with SPIFFE/SPIRE: expensive to operate, but it is the only thing that turns east-west traffic into something authorisable, and it becomes mandatory as soon as agents with MCP arrive. And last, the TEE, only if the infrastructure operator is in your threat model; Kata without a TEE can jump ahead to fourth place if there are tenants that do not trust each other.

Mapping the series to ENS, ISO/IEC 42001 and the EU AI Act

The detail is in technical controls ENS × ISO 42001 × EU AI Act; this table is the summary by link.

Link	ENS measure (RD 311/2022)	ISO/IEC 42001 (Annex A)	EU AI Act
1/4 Contract and control plane	`op.exp.2`; `op.mon.1`	A.6 life cycle	Art. 12 (record-keeping); Art. 13 (transparency)
2/4 Registry and distribution by digest	`op.ext.3`	A.10 third parties	Art. 11 and Annex IV (technical documentation)
3/4 Signature, attestations and AIBOM	`op.exp.6`; `op.ext.3`	A.6.2 and A.7 data and traceability	Art. 15(5), data and model poisoning
4/4-A Workload identity	`op.acc.1/2/5`; `op.exp.11`; `mp.com.2-3`	A.9 use of the AI system	Art. 15(5), unauthorised third parties
4/4-B Isolation and TEE	`mp.info.3`; `op.exp.2`; `mp.com.4`	A.6 life cycle controls	Art. 15(4) robustness; Art. 15(5)

Art. 15(5) carries the most weight: it requires high-risk systems to be “resilient against attempts by unauthorised third parties to alter their use, outputs or performance”, and it enumerates data and model poisoning and confidentiality attacks. Links 3 and 4 are its technical implementation. For the management framework, see ISO/IEC 42001 as an AIMS and the AI Act mapping.

For an inference factory

Start workload identity with the gateway↔engine pair and with the agents, not with the whole cluster. Registering all forty workloads on day one is the most reliable way to abandon the project: register two entries, set up the Istio policy that requires the gateway’s principal and live with it for a couple of weeks to learn what breaks when the certificate rotates. Then the agents with MCP, where static credentials are the entire threat model.

Separate the Kata decision from the TEE decision, and take them in that order. Kata is a RuntimeClass change with a second of startup and some memory. CoCo is a redesign: you lose MIG and the GPU’s performance counters, you gain ten seconds or more of startup, you pay 15-25 % of capacity and you add two dependencies whose failure prevents workloads from starting. Evaluate the second with a written threat model, not with a slide.

If you go to CoCo, rehearse patch day before you need it. The failure that will take your inference down will not be an attack, it will be a firmware update that misaligns the RVPS reference values. Write the runbook (update the RVPS with overlap, patch, retire the old value), rehearse it on one node and monitor the attestation success rate as a first-class metric, just as for SPIRE: alert on attestation failures before they turn into 401s at the gateway.

That closes the series: from the contract to the artefact, from the artefact to its proof, and from the proof to the process and the machine. What lies below is no longer a chain of trust but daily operation: seeing what each process does with Tetragon and measuring whether what is served is still good with evals. Trust is established once; vigilance is continuous.

Sources

CNCF, SPIFFE and SPIRE Projects Graduate from CNCF Incubator (20 sep. 2022) — https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/
SPIFFE, SPIFFE Concepts (SPIFFE ID, trust domain, SVID, Workload API) — https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/
SPIFFE, SPIFFE Federation specification — https://spiffe.io/docs/latest/spiffe-specs/spiffe_federation/
SPIFFE, Scaling SPIRE (dimensionamiento y punto único de fallo) — https://spiffe.io/docs/latest/planning/scaling_spire/
spiffe/spire, Releases (v1.15.0 de 19 may. 2026; v1.15.1 de 28 may. 2026, parche de azure_imds) — https://github.com/spiffe/spire/releases
spiffe/spire, Kubernetes Workload Attestor plugin (selectores, kubelet, cgroups, Sigstore) — https://github.com/spiffe/spire/blob/main/doc/plugin_agent_workloadattestor_k8s.md
spiffe/spire, SPIRE Server configuration reference (default_x509_svid_ttl, default_jwt_svid_ttl, ca_ttl) — https://github.com/spiffe/spire/blob/main/doc/spire_server.md
Istio, SPIRE integration (Envoy SDS, SPIFFE CSI Driver, trust domain) — https://istio.io/latest/docs/ops/integrations/spire/
SPIFFE, OPA Authorization with Envoy and JWT-SVIDs — https://spiffe.io/docs/latest/microservices/envoy-jwt-opa/readme/
The New Stack, How Cilium’s Mutual Authentication Can Compromise Security — https://thenewstack.io/how-ciliums-mutual-authentication-can-compromise-security/
Riptides, Bringing SPIFFE to OAuth for MCP — https://riptides.io/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads/
CNCF, Confidential Containers becomes a CNCF incubating project (22 jul. 2026) — https://www.cncf.io/blog/2026/07/22/confidential-containers-becomes-a-cncf-incubating-project/
Confidential Containers, Attestation with Trustee (KBS, AS, RVPS, CDH) — https://confidentialcontainers.org/docs/attestation/
Confidential Containers, Deploy Trustee in Kubernetes (11 feb. 2026) — https://confidentialcontainers.org/blog/2026/02/11/deploy-trustee-in-kubernetes/
confidential-containers/trustee, Releases (v0.20.0: TLS 1.3, PQC, multi-GPU ITA, NVSwitch) — https://github.com/confidential-containers/trustee/releases
IETF, RFC 9334 — Remote ATtestation procedureS (RATS) Architecture — https://www.rfc-editor.org/info/rfc9334/
Kata Containers, Kata Containers 4.0.0 Preview (28 abr. 2026, runtime-rs por defecto) — https://katacontainers.io/blog/release-4-0-0-preview/
kata-containers, Release 3.32.0 — https://github.com/kata-containers/kata-containers/releases/tag/3.32.0
NVIDIA, Deploy with Kata Containers — GPU Operator — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html
NVIDIA, Deploying Proprietary Models Securely with Confidential Computing on Self-Hosted Kubernetes (GA 1.0.0; MIG y vGPU prohibidos en CC) — https://docs.nvidia.com/enterprise-reference-architectures/deploying-proprietary-models-confidential-compute-self-hosted-kubernetes/latest/reference-implementations.html
CACM, Creating the First Confidential GPUs (CPR, bounce buffers, contadores deshabilitados, HBM no cifrada) — https://cacm.acm.org/practice/creating-the-first-confidential-gpus/
NVIDIA, Confidential Computing on H100 GPUs for Secure and Trustworthy AI (límite de ~4 GB/s, NRAS) — https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/
NVIDIA, Hardware-Rooted AI Security That Won’t Slow You Down (2 jul. 2026; Blackwell, NVLink cifrado hasta 8 GPU) — https://developer.nvidia.com/blog/hardware-rooted-ai-security-that-wont-slow-you-down
Corvex, Confidential Computing Meets NVIDIA HGX B200 (claim comercial) — https://www.corvex.ai/blog/confidential-computing-meets-nvidia-hgxtm-b200-secure-ai-without-the-performance-trade-off
arXiv 2409.03992, Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study — https://arxiv.org/html/2409.03992v1
arXiv 2509.18886, Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs (23 sep. 2025) — https://arxiv.org/abs/2509.18886
arXiv 2607.19353, Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX (may. 2026; heurística del 15-25 %) — https://arxiv.org/html/2607.19353
Segarra et al., Serverless Confidential Containers: Challenges and Opportunities (SESAME'24) — https://carlossegarra.com/assets/papers/sesame24-serverlesscoco.pdf
EU Artificial Intelligence Act, Article 15: Accuracy, Robustness and Cybersecurity — https://artificialintelligenceact.eu/article/15/
EU Artificial Intelligence Act, Article 12: Record-Keeping — https://artificialintelligenceact.eu/article/12/

Storage in the AI era (3/4): security

Mon, 22 Jun 2026 10:00:00 +0200

Data is the most valuable and most durable asset of an AI system. Models get retrained; weights get replaced; but training datasets, the customer data feeding RAG and the weights of a frontier model represent years of investment and, often, regulated information. In the two previous articles we discussed how to move that data fast. This third one is about how to protect it: encryption, integrity, access control, sovereignty and the threat that forces us to act today even though it looks distant, quantum computing.

Encryption at rest: SED, AES-XTS and the transition to FIPS 140-3

The first line of defence for any persistent data is encryption at rest. In enterprise storage the dominant pattern is the SED (Self-Encrypting Drive), a drive that implements the Trusted Computing Group’s Opal, Ruby and Enterprise standards and integrates a 256-bit AES-XTS encryption engine in the controller. The operational advantage is twofold: they encrypt at interface speed, with no perceptible performance penalty and without stealing CPU cycles, and they enable instant cryptographic erase.

The mechanism behind that erase is the two-tier key model of TCG Opal 2.0. Data is actually encrypted against a data encryption key (DEK) that never leaves the drive; that DEK, in turn, is protected by an authentication key derived from the user’s passphrase. Destroying or regenerating the DEK instantly renders terabytes of data unreadable without having to overwrite them: the cryptographic erase. For an organisation decommissioning 245 TB drives, this property is not a luxury, it is the only practical way to guarantee erasure.

SED hardware encryption and software encryption with AES-NI are not mutually exclusive, but they do have different profiles. Hardware acceleration keeps a measurable edge, on the order of 2 times on 4K random reads and writes compared with pure software, although the gap narrows on modern CPUs with AES-NI and ARMv8. The rule of thumb: SED for the baseline at no performance cost, and software encryption on top when you need key domain separation or end-to-end encryption.

The Achilles heel of encryption is never the algorithm, it is key management. Impeccable AES-256 protects nothing if the key is badly held. That is why serious storage encryption relies on an external key manager that speaks KMIP (Key Management Interoperability Protocol) to the arrays, separating custody of the key from the medium that encrypts. Keys are rotated periodically, backed up with their own availability guarantees (losing the key means losing the data as definitively as a fire would) and their access is audited. In regulated environments those keys live in certified hardware security modules (HSM). For an architect, the design decision is not “do we encrypt?”, the answer is always yes, but “who holds the keys and how are they governed?”, a question that, as we will see, sits at the heart of data sovereignty.

The point no architect should ignore in 2026 is the transition to FIPS 140-3. The CMVP stopped accepting new FIPS 140-2 validations in April 2022, and on 21 September 2026 it will move all remaining 140-2 certificates to the “historical” list. From that date, US federal agencies must not include historical modules in new procurements; existing systems can keep operating, but the compliance clock is running. And there is a logistical side effect: validation times have grown from 367 to 542 days, 42 % more, which strains vendor roadmaps. Anyone planning storage purchases on a multi-year horizon should verify 140-3 certification, not 140-2, for every cryptographic module. The reference framework for the overall design remains NIST SP 800-209, which covers encryption, isolation, authentication and authorisation for block, file and object storage.

Encryption in transit: the weak point of AI fabrics

Data is also exposed in motion, and here the high-performance fabrics of AI introduce specific tensions. For NVMe-oF over TCP there is in-band authentication, and for stronger protection the NVMe working group proposes IPsec over RoCEv2. Modern NICs (ConnectX-6 Dx, BlueField-2) secure NVMe-oF connections with hardware-accelerated IPsec and TLS, so encryption in transit does not throttle bandwidth.

The problem is InfiniBand. IPsec is not implemented over InfiniBand, since it is not an IP protocol, which leaves that fabric, ubiquitous in AI clusters, without one of the usual confidentiality tools. Nor does IPsec isolate well the RDMA connections of different users sharing the same interface. Academic research has documented attacks that exploit RDMA flaws in NVMe-oF storage applications. The conclusion for an architect is uncomfortable but important: in-transit security on an AI fabric is not inherited from the good habits of IP networks, and it demands explicit design of segmentation and isolation.

The threat arriving from the future: post-quantum cryptography

No storage security topic has changed its urgency as much as post-quantum cryptography (PQC), and the reason is an attack pattern that does not need a quantum computer to begin: "harvest now, decrypt later" (HNDL). Well-funded state actors are collecting encrypted data today with the intention of decrypting it once they have quantum capability. For any data whose confidentiality must last a decade, such as medical records, industrial secrets or proprietary training datasets, the threat is present, not future.

In August 2024 NIST finalised the first three PQC standards, and by mid-2026 they are the foundation on which the entire migration is built:

Standard	Algorithm	Function	Origin
FIPS 203	ML-KEM	Key encapsulation	CRYSTALS-Kyber
FIPS 204	ML-DSA	Digital signature (default)	CRYSTALS-Dilithium
FIPS 205	SLH-DSA	Backup signature (hash-based)	SPHINCS+

In March 2025 NIST additionally selected HQC, a code-based encapsulation mechanism, as a backup for ML-KEM with different mathematical assumptions. The NSA, for its part, set a demanding timetable in CNSA 2.0: by 2030 all software and firmware deployed in national security systems must use CNSA 2.0 signatures, with full enforcement expected around 2031-2033 and complete quantum resistance in 2035.

The pressure has intensified from the quantum hardware side. Between May 2025 and early 2026, several papers cut the estimates of qubits needed to break RSA-2048 from around 20 million to under a million, and even to figures on the order of 100,000 with new architectures. “Q-Day” around 2030 is increasingly cited by analysts, NIST and the NSA. It is not a certainty, but the safety margin has narrowed enough to make inaction imprudent.

The storage industry is already moving. NetApp announced PQC for data at rest in 2025, compliant with the NIST algorithms and integrated both at rest and in flight. Western Digital began integrating NIST-approved PQC into its Ultrastar drives in 2026, one of the first deployments in production infrastructure. Cohesity, Commvault and Quantum also have products along these lines. The dominant pattern in 2026 is not wholesale replacement but hybrid approaches that combine classical and post-quantum cryptography: they protect against a failure of the new algorithms and allow a gradual migration. For an architect, today’s concrete action is the cryptographic inventory, or crypto-agility: knowing which algorithms protect which data and planning their rotation.

Security specific to AI data: integrity and poisoning

Beyond classical encryption, AI brings a threat class of its own that falls directly on the data layer: data poisoning. An adversary manipulates training or fine-tuning data to corrupt what the model learns. Its distinctive trait is stealth: a poisoned model behaves normally for long periods before exhibiting the malicious behaviour, which makes it very hard to detect after the fact. The OWASP Top 10 for LLM applications of 2025 formally recognises “data and model poisoning” as an integrity attack category, with especially high risk when external sources are ingested.

The defence has moved towards provenance and lineage. Organisations are adopting rigorous documentation of every data source, a “digital chain of custody”, and the concept of the ML-BOM (Machine Learning Bill of Materials), analogous to the software SBOM. Frameworks such as the FINOS AI governance framework explicitly catalogue data poisoning among their risks. For storage, this means the system must be able to guarantee the integrity and immutability of training data and to record its lineage verifiably. Integrity stops being a desirable property and becomes a security control.

Model theft: when the asset to protect is the weights

There is one asset the traditional storage security conversation did not contemplate: model weights. Training a frontier model costs tens or hundreds of millions, and the result, a file of a few TB, concentrates all that value. Weight theft (model exfiltration) is consequently a first-order threat, and its attack surface is precisely the storage layer where checkpoints and final models live.

Protection combines several of the layers already described, applied with a specific focus. Encryption at rest with strict key control stops an attacker who reaches the drives from walking away with a usable model. Least-privilege access control limits who can read the weight directories, and auditing logs every access so anomalous exfiltration can be spotted; a process that suddenly reads 4 TB from a model directory is a signal. And confidential computing, which we will see further down, protects the weights even while they are being loaded into the GPU for inference. The lesson for the architect is that the trained model deserves, at minimum, the same defences as customer data, because its loss can be irreversible and its value greater.

Immutability and ransomware: the second phase of the attack

Ransomware remains the highest economic impact threat against storage, and its recent evolution points straight at backups. The reference defence is WORM immutability (Write-Once-Read-Many). In the object world, S3 Object Lock prevents modifying or deleting data even for an attacker with administrative credentials, neutralising the second phase of the attack, the encryption of the victim’s data. It has two modes: Governance, which can be overridden with specific permissions, and Compliance, which nobody can override, not even the root account. Object Lock creates a logical air gap that, for many use cases, makes tapes and physical air gaps redundant.

The numbers justify the investment. Ransom payments reached a record USD 1.1 billion in 2023; the average total cost of a ransomware attack was around USD 5.13 million in 2024, with an average recovery cost, excluding the ransom, of 1.53 million in 2025. Two out of three organisations suffered ransomware in the past year according to Sophos.

For an inference factory, ransomware has an angle of its own: the most valuable target is not only customer data but the model repository. Encrypting or deleting the weights the factory serves leaves it inoperative immediately, with a direct business impact, the service goes down, on top of the recovery cost. The defence is the same as for any critical asset, applied as a priority: immutable WORM copies of the model repository and of the vector indexes, so that a clean version of each model can always be restored even if the hot copy is compromised. A model that took months and millions to produce, or whose weights cannot be regenerated, deserves the strictest immutability protection available.

The storage-side response has been to integrate detection into the array itself. Dell PowerProtect Cyber Recovery offers an isolated, immutable vault with an automated air gap, and its CyberSense component uses AI and full content analysis, not just metadata, to detect ransomware corruption, with very high accuracy figures announced by the vendor. NetApp integrates Autonomous Ransomware Protection into the storage, creating immutable snapshots in real time by analysing data patterns at the storage layer. It is wise to treat the accuracy figures (99 % and above) as vendor claims rather than independent benchmarks, but the underlying trend is real: the storage array has stopped being a passive component and become an actor in the defence.

A useful framework for prioritising these investments is the annualised loss expectancy, which relates the impact of an incident to its frequency:

$$\text{ALE} = \text{SLE} \times \text{ARO}$$

where $ \text{SLE} $ is the expected loss per incident and $ \text{ARO} $ the expected annual frequency. Immutability and fast recovery act by reducing $ \text{SLE} $: even if the attack happens, the loss and the recovery time are bounded.

Access control and the cost of a breach

Encryption protects the data; access control decides who touches it. IBM’s Cost of a Data Breach 2025 report supplies the reference figure: the global average cost of a breach fell to USD 4.44 million, 9 % less and the first drop in five years, driven by faster containment thanks to AI-powered defences, with the average time to identify and contain falling to 241 days, the lowest in nine years. The United States, by contrast, set a record of 10.22 million. The largest cost component is still detection and escalation.

Mature architectures combine Zero Trust with RBAC as the base, ABAC attributes for fine grain and microsegmentation to prevent lateral movement inside the network. In an AI environment, where training data moves between data lake, pipeline and GPU cluster, microsegmentation and continuous contextual authorisation stop being optional.

Data sovereignty: the regulatory dimension

For a European company, AI storage security is inseparable from data sovereignty. The EU Data Act, applicable since 12 September 2025, mandates data portability and interoperability to eliminate vendor lock-in, and in its Chapter VII requires cloud providers operating in the EU to put in place technical, legal and organisational measures to prevent non-EU government access to non-personal data stored in the EU where such access would be unlawful under European law. This creates a documented tension with the US CLOUD Act, which empowers US authorities to claim data from providers under their jurisdiction regardless of where it is stored.

The technical answer gaining traction in 2026 is client-side encryption with full key ownership: whoever controls the keys controls the data, regardless of where it physically resides. “Whoever allows plaintext in the cloud cedes control” has become the motto of real sovereignty. For a European AI architect, data residency and key ownership are design decisions as important as performance.

The EU AI Act, furthermore, applies in phases that touch data governance: prohibited practices are enforceable since February 2025, obligations for general-purpose models since August 2025, and sanctioning powers since August 2026, although the provisional Digital Omnibus agreement of May 2026 postponed the deadline for Annex III high-risk systems to December 2027, a point worth re-checking given how recent it is.

Confidential computing: protecting data in use

Encryption at rest and in transit leaves a gap: data in use, decrypted in memory while it is processed. Confidential computing closes it with trusted execution environments (TEE). On the CPU, Intel TDX and AMD SEV-SNP launch confidential virtual machines with encrypted memory. The novelty for AI is on the GPU.

NVIDIA introduced confidential computing on the H100 (Hopper) with a CC mode and a root of trust in silicon: the CPU’s confidential VM exchanges encrypted data with an enclave in the GPU, extending the chain of trust from CPU to GPU. The Blackwell architecture (B200, GB200) has made the qualitative leap: it is the industry’s first GPU with TEE-I/O capability, with inline protection over NVLink and NVSwitch that removes the PCIe I/O bottlenecks of earlier generations. And what matters most for performance: thanks to AI-specific encryption engines and hardware-accelerated encrypted HBM access, the HGX B200 keeps its roughly 2 times advantage in training and 2.5 times in inference over the HGX H200 even with confidential computing fully enabled. This enables confidential training, inference and federated learning without the prohibitive penalty that previously made encrypting data in use unworkable, protecting both sensitive data and the model weights themselves from theft.

Security in a multi-tenant inference factory

Confidential computing takes on special meaning in an inference factory, above all if it serves several customers or teams on the same infrastructure. Here the attack surface is not only the disk at rest but the data in use of each request: the prompts, the context documents, the responses and, crucially, the KV cache. And the KV cache introduces a risk specific to inference that is worth naming: if a cache layer is shared between tenants to save compute, reusing prefixes as we saw in the performance article, badly isolated reuse can leak one customer’s content to another. The design rule is that prefix caching should only be shared within a single trust domain; between different tenants, KV cache isolation, whether in memory, in CXL or on NVMe, is a security control, not an optional optimisation.

The same principle extends to the factory’s other planes. The model repository must apply per-tenant access control so that nobody loads or extracts someone else’s model. The RAG vector database must segment indexes per customer, because an embedding retrieved from the wrong corpus is both a quality failure and a data leak. And telemetry, which records prompts and responses, is one of the most sensitive repositories in the whole installation, subject to the same encryption, retention and residency requirements as customer data. A well-designed inference factory treats each of these planes as an isolation domain, not as a resource shared for convenience.

An important technical nuance of confidential computing is attestation. A secure enclave is worthless if you cannot prove the data is really executing inside it and not in a compromised environment pretending to be one. Attestation is the cryptographic mechanism by which the hardware, the root of trust in the silicon of the GPU or the CPU, issues a verifiable proof of its state and identity before keys or data are entrusted to it. In a confidential AI architecture, the chain of trust extends from the CPU’s confidential VM to the GPU enclave, and every link is attested. For the architect, the operational question is who verifies those attestations and where the keys that are only released after a valid attestation are held: that is where confidential computing connects back to the key management from the start of the article.

A checklist for the architect

Gathering all of the above into actionable practice helps avoid leaving layers uncovered. At rest: SED with AES-XTS-256 as the baseline, external key management via KMIP with HSM, and verification of FIPS 140-3 certification on every module, anticipating September 2026. In transit: TLS 1.3 and IPsec wherever possible, with particular attention to isolation on InfiniBand fabrics, where the IP tools do not apply. Against the quantum threat: cryptographic inventory, crypto-agility and adoption of hybrid classical-PQC schemes for long-lived data, without waiting for Q-Day. Against ransomware: WORM immutability with S3 Object Lock in Compliance mode for critical copies, detection built into the array and tested recovery. On integrity: verifiable lineage and ML-BOM for training data, with reinforced protection of model weights. In the inference factory, additionally: isolation of the KV cache and of prefix caching between tenants, per-customer segmentation of the model repository and of the RAG vector indexes, and treatment of prompt and response telemetry as sensitive data. On governance: Zero Trust access control, microsegmentation and auditing. And on sovereignty: key ownership and data residency aligned with the EU Data Act and the EU AI Act. None of these layers substitutes for another; real security emerges from their combination.

Takeaways

AI storage security is a layered system that has to be designed all at once: encryption at rest with SED and migration to FIPS 140-3; encryption in transit with particular care on InfiniBand fabrics; crypto-agility and hybrid approaches to get ahead of quantum risk and harvest now, decrypt later; data integrity and lineage against poisoning; WORM immutability against ransomware; Zero Trust access control; data sovereignty through key ownership; and confidential computing to protect data and models in use. The good news is that the array is no longer passive: it defends. The bad news is that the most serious threat, the quantum one, demands action before it becomes visible.

The last article in the series closes with the property without which none of the previous ones matter: availability.

Sources

NIST, FIPS 140-3 Transition Effort — https://csrc.nist.gov/projects/fips-140-3-transition-effort
SafeLogic, What Happens on September 21, 2026 — https://www.safelogic.com/blog/what-happens-on-september-21-2026
NIST, Security Guidelines for Storage Infrastructure (SP 800-209) — https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-209.pdf
NIST, NIST Releases First 3 Finalized Post-Quantum Encryption Standards — https://www.nist.gov/news-events/news/2024/08/nist-releases-first-3-finalized-post-quantum-encryption-standards
PostQuantum.com, NSA Unveils CNSA 2.0 — https://postquantum.com/quantum-policy/nsa-cnsa-2-0-pqc/
Blocks & Files, NetApp boosts storage security with post-quantum encryption — https://www.blocksandfiles.com/security/2025/04/29/netapp-boosts-storage-security-with-post-quantum-encryption/1610355
The Quantum Insider, Western Digital Adds Post-Quantum Cryptography to Hard Drives — https://thequantuminsider.com/2026/05/19/western-digital-adds-post-quantum-cryptography-to-hard-drives/
FINOS AI Governance Framework, Data Poisoning (RI-9) — https://air-governance-framework.finos.org/risks/ri-9_data-poisoning.html
Object First, S3 Object Lock for Ransomware Protection — https://objectfirst.com/guides/immutability/s3-object-lock-for-ransomware-protection/
StorageReview, Dell enhances PowerProtect portfolio for cyber resilience — https://www.storagereview.com/news/dell-technologies-enhances-powerprotect-portfolio-for-improved-cyber-resilience
NetApp, Autonomous Ransomware Protection — https://www.netapp.com/cyber-resilience/autonomous-ransomware-protection/
IBM, Cost of a Data Breach Report 2025 — https://www.ibm.com/reports/data-breach
European Commission, Data Act — https://digital-strategy.ec.europa.eu/en/policies/data-act
artificialintelligenceact.eu, Implementation Timeline — https://artificialintelligenceact.eu/implementation-timeline/
NVIDIA, Confidential Computing on H100 GPUs — https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/
Corvex, Confidential Computing Meets NVIDIA HGX B200 — https://www.corvex.ai/blog/confidential-computing-meets-nvidia-hgxtm-b200-secure-ai-without-the-performance-trade-off

How we use our infrastructure to meet defence standards with AI

Thu, 18 Jun 2026 11:00:00 +0200

In defence, the challenge of putting AI into operation is not the model: it is security and compliance. The rules that apply, the allied interoperability standards, the data-centric security rules, the accreditation frameworks and the principles of responsible use, impose concrete requirements on access control, classification, confidentiality, traceability, supply chain and sovereignty. A demo on a laptop connected to the internet ignores all of that; an operable platform has to satisfy it, and has to be able to prove it to an accreditation body.

This article is not a tour of technologies. It is the opposite: we take the requirements imposed by the standards of the sector and, for each one, we show how our infrastructure is used to meet it. The infrastructure is the same real platform we use in other sectors, two RKE2 clusters (Kubernetes v1.35) with Cilium as the network, GitOps with Flux, Keycloak as the identity server, inference on our own GPUs and, in pre-production, scheduling with Volcano, adapted to an environment with stricter security requirements. Every component is here because it solves a regulatory requirement, not because it looks good on a diagram. And, like all infrastructure, it is measured by efficiency and cost, which is what makes compliance sustainable.

It is worth starting with an idea many people are unaware of: standards for this already exist in the sector, and nothing has to be invented. There are allied interoperability standards, there is a defence-specific DevSecOps reference model on Kubernetes, there is a security framework (Zero Trust) and a supplier accreditation scheme, and there is a body of principles for the responsible use of military AI. Let us first lay out that map, and then walk the requirements one by one.

The map of standards in the sector

Four families of standards intersect in a defence AI platform, and understanding what each one covers avoids reinventing wheels.

Allied interoperability. In a coalition, systems have to understand each other. Federated Mission Networking (FMN) defines how mission networks are federated across nations through successive “spirals”; the MIP (Multilateral Interoperability Programme) and its JC3IEDM data model standardise command and control; Link-16 and formats such as NVG (NATO Vector Graphics) cover tactical and situational exchange. They are, in defence, the equivalent of what FHIR and DICOM are in healthcare: the common language without which AI is an isolated system that cannot be integrated into the operation.

Data-centric security: STANAG 4774 and 4778. This is the most distinctive piece of the sector. STANAG 4774 defines the syntax of a confidentiality label (policy, classification, security categories), and STANAG 4778 defines the binding mechanism that ties that label to the data throughout its whole life cycle and between the parties that share it. Instead of protecting only the perimeter or the system, the information object itself is protected: the classification travels with the data and the access decision is taken against the label. Both STANAGs are in the process of ratification by the nations and are moving from recommended practice to a requirement for mission-critical software.

Defence DevSecOps on Kubernetes. To build and deploy defence software there is a very concrete reference model: the DoD Enterprise DevSecOps Reference Design on Kubernetes, embodied in platforms such as Platform One and its Big Bang product line (declarative continuous delivery of hardened packages on a Kubernetes cluster), supported by Iron Bank, the repository of hardened container images compliant with the DISA STIGs, continuously scanned. The associated philosophy is cATO (Continuous Authority To Operate): accreditation stops being a one-off milestone and becomes a state maintained through continuous monitoring. For defence, it is the equivalent of the Kubernetes-native standard that MONAI represents in medical imaging: an agreed way to package and operate software with guarantees about the substrate we already use.

Security and accreditation frameworks. The DoD Zero Trust Reference Architecture (v2.0) sets the guiding principle, do not trust by default, always verify, with adoption targets towards 2027. Supplier accreditation is governed by CMMC 2.0 (whose Phase 2, with mandatory third-party certification for level 2, arrives in November 2026) over the 110 controls of NIST SP 800-171. In Spain and the EU, the Esquema Nacional de Seguridad (the Spanish National Security Framework) at high category and the CCN-STIC guides of the Centro Criptológico Nacional (the Spanish National Cryptologic Centre) play an equivalent role for systems handling sensitive or classified information.

Responsible AI in defence. NATO adopted six Principles of Responsible Use (PRU) for AI in defence: lawfulness, responsibility and accountability, explainability and traceability, reliability, governability and bias mitigation, reinforced by its AI Strategy revised in 2024 and by the DARB (Data and AI Review Board). An important legal nuance: the EU AI Act expressly excludes (art. 2.3) AI systems intended exclusively for military, defence or national security purposes. That is, in defence the responsible AI framework is not the AI Act, but NATO’s PRU, national rules and, as a management system, ISO/IEC 42001.

With this map on the table, let us walk the requirements.

Requirement: coalition interoperability (FMN, MIP/JC3IEDM)

In an allied operation, a system that does not integrate is useless however good it is. The requirement, derived from FMN, from the command and control data model (MIP/JC3IEDM) and from the tactical formats (Link-16, NVG), is that AI must consume and produce information in the common formats of the coalition, not in a dialect of its own.

How we meet it: just as in other sectors we put an interoperability layer at the boundary, here the platform integrates through the agreed gateways and formats, so that AI never talks directly to the mission systems but consumes and delivers information already normalised to the coalition standard. This keeps AI as a component that couples into the federated architecture, respecting its profiles and its data governance, instead of a silo that forces bespoke integrations at every deployment. Interoperability is also a precondition for traceability and labelling: only if the data enters and leaves in a known format can its confidentiality label be attached and preserved.

Requirement: do not trust by default, Zero Trust and verified access (DoD ZT RA, NIST 800-171)

The guiding principle of modern defence security is Zero Trust: no entity, user, service or node, is trusted because of its position in the network; every access is authenticated, authorised and continuously verified. NIST 800-171 details the access and identification controls that underpin it.

How we meet it: a single identity server, Keycloak, governs authorisation with OAuth 2.1 and OpenID Connect, issuing short-lived tokens and binding them to their recipient with audience validation (aud), so that a credential issued for one service is useless against another. Between components handling sensitive information we require mutual TLS (mTLS): it is not only the user or the service that is authenticated, but the workload itself taking part in the exchange. And the network operates in default-deny (next section), so that connectivity is not a right conferred by being inside the cluster, but an explicit permission. Workloads use service accounts with no permissions by default, and administrator access to the Kubernetes API is federated against Keycloak over OIDC with scoped roles: technical access too is authenticated, limited and audited. Zero Trust is not a product you buy; it is a property that emerges from strong identity, least privilege and continuous verification, and that is how the platform is built. Verification, moreover, does not stop at sign-in: tokens are short-lived and are renewed, policies are re-evaluated on every request, and the posture of each workload, signed image, applied security policy, determines whether it can run and communicate. A component that stops complying with its policy stops being able to operate; it is granted no grace period for having been inside.

Requirement: data-centric security and labelling (STANAG 4774/4778)

Unlike other sectors, in defence the classification travels with the data. The requirement, derived from STANAG 4774/4778, is that every information object carries its confidentiality label and that access decisions are taken against that label, not against the system hosting it.

How we meet it: authorisation decisions are not limited to roles, but incorporate attribute-based access control (ABAC) evaluating the classification label of the data against the clearance of the requester. The policy is expressed as code and evaluated at a central decision point (a policy engine of the OPA/Open Policy Agent kind) that the application layer and the gateways consult before serving a resource. The label, in accordance with STANAG 4774, travels with the object, and the binding under STANAG 4778 is preserved across the platform, in the object metadata in storage and in the annotations that travel with the request. For an AI agent, this means that its token gives it access to a classification level, and that data above that level is simply not handed to it, regardless of where it is stored.

This approach embodies the classic no-read-up rule: an entity with a given clearance does not access information above its level, and the decision is taken on the object’s label, not on assumptions made by the system. When a workflow requires moving information to a lower level, a declassification or sanitisation, it is an explicit, audited step, with its own policy, never a side effect of a query. The label is preserved in the object metadata in storage and in the annotations accompanying the request, so that the STANAG 4778 binding is not lost when crossing the components of the platform.

Requirement: segregation by mission and level (multitenancy, NIST 800-171, ENS)

Different missions, different classification levels and different compartments cannot share a common space without controls. The requirement: strong isolation, so that what belongs to one mission or level is not accessible from another.

How we meet it: each mission or compartment lives in its own Kubernetes namespace, with scoped RBAC, ResourceQuota and LimitRange, dedicated storage and databases per operator, not shared behind an application filter, and its own network segmentation. Identity is segregated by realm in Keycloak. The whole definition lives in Git and is reconciled by Flux (GitOps), so that segregation is declarative and auditable through change history. Where the classification level demands it, separation goes beyond the cluster: operation runs on physically distinct clusters or sites, and exchange between classification domains, when necessary, is done through controlled cross-domain solutions, not by opening a hole between namespaces. At the highest levels, each security domain can be an independent cluster with its own chain of trust, its own identities and its own governance, preventing a single control plane from spanning classifications that must never be mixed. The choice between a separate namespace, cluster or site is itself an accreditation decision: it is documented, justified according to the level and compartment, and reflected declaratively in the repository.

Requirement: confidentiality, encryption and exfiltration prevention (NIST 800-171, ENS, Zero Trust)

NIST 800-171 (system and communications protection families) and the ENS require confidentiality, encryption of communications and prevention of information leakage. In defence, exfiltration is not an abstract risk: it is the adversary’s objective.

How we meet it: the network is managed by Cilium (eBPF) with a default-deny model. A pod has no connectivity beyond what a policy grants it explicitly, flow by flow, which turns exfiltration into something blocked in the kernel. A component can only talk to strictly necessary destinations:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: inferencia-egress-minimo
 namespace: mision-alfa
spec:
 endpointSelector:
 matchLabels:
 app: serving-inferencia
 egressDeny:
 - toEntities: [world]  # no egress outside the cluster
 egress:
 - toEndpoints:
 - matchLabels:
 app: gateway-modelos
 - toEndpoints: # internal DNS, without this it does not resolve
 - matchLabels:
 k8s:io.kubernetes.pod.namespace: kube-system
 k8s-app: kube-dns
 toPorts:
 - ports: [{ port: "53", protocol: UDP }]

Two nuances separate a correct policy from a leaky one: you have to explicitly allow DNS and filter by endpoint identity, not by CIDR, because in an identity-based network filtering by IP ranges does not behave the way intuition suggests. On top of this we add flow observability with Hubble and runtime security with Tetragon (eBPF), which watches process execution, file access and connections inside the container, making it possible to detect and block anomalous behaviour; an unexpected process in a sensitive workload is exactly the signal you want to see in real time. Encryption in transit is covered by cert-manager (TLS at the edges) and Cilium’s transparent node-to-node encryption. Added to this are the Pod Security Standards in restricted profile (no privileges, no root) and disciplined management of secrets and keys, which do not live in plain manifests.

Requirement: verifiable software supply chain (Iron Bank/STIG, SBOM, CMMC)

The software supply chain is a first-order attack vector, and the sector treats it as such: images hardened in line with the STIGs, a component inventory (SBOM) and cryptographic verification of provenance. CMMC and NIST 800-171 require it in the supplier flow; Executive Order 14028 popularised the SBOM.

How we meet it: we start from hardened base images (in the spirit of Iron Bank and the DISA STIGs), we generate an SBOM (CycloneDX/SPDX) per artefact and we sign images and SBOMs with Sigstore/Cosign, a graduated CNCF project, with keyless signing and a transparency log. In the cluster, an admission control (Kyverno/policies) rejects any image that is not signed by a trusted identity, so software whose provenance cannot be verified never runs. The rule is expressed as code and verifies the signature of every image before allowing it to run:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: exigir-imagenes-firmadas
spec:
 validationFailureAction: Enforce
 rules:
 - name: verificar-firma-cosign
 match:
 any:
 - resources:
 kinds: [Pod]
 verifyImages:
 - imageReferences:
 - "registro.interno/*"
 attestors:
 - entries:
 - keys:
 publicKeys: |-
 -----BEGIN PUBLIC KEY-----
 ...trusted key...
 -----END PUBLIC KEY-----

An image without a valid signature is rejected at admission, before it runs. Since the whole deployment is GitOps, the chain, from the commit to the signed image to the policy that admits it, is traceable end to end. This satisfies the supply chain control of NIST 800-171 and fits the cATO model: verification is continuous, not a one-off stamp.

Requirement: sovereignty and disconnected or air-gapped operation (classified networks, DDIL)

Defence networks frequently operate isolated from the internet (air-gapped) or under DDIL conditions, degraded, intermittent or low-bandwidth connectivity at the tactical edge. And the data cannot leave to an external API under any circumstances. The requirement: the platform has to work without internet and without taking the data outside.

How we meet it: inference is served locally, on our own GPUs, with a high-performance serving engine (vLLM) and open-weight models that can be hosted entirely inside the perimeter. Everything the platform needs, container images, models, dependencies, is mirrored into internal registries and repositories, so that the GitOps deployment reconciles from internal sources and requires no internet access. A model gateway centralises access and reverses the burden: the default behaviour is local inference, and any egress is an exception that, besides having to be enabled explicitly, would collide with the default-deny network policies. Sovereignty does not rest on a promise, but on several mutually reinforcing controls, gateway, network policy and isolation, so that a configuration slip does not translate into a leak.

One point that cannot be overlooked in defence: no component should “phone home”. Many tools send telemetry, check licences or download updates against external servers by default, and in a classified network that is unacceptable. That is why default-deny is not only a measure against the adversary, but also against the software itself: if an image attempts an undeclared telemetry connection, the policy blocks it and Tetragon logs it. The absence of phone-home is verified, not trusted.

Requirement: resilience and operation at the tactical edge (DDIL)

Much defence operation happens under DDIL conditions, degraded, intermittent or low-bandwidth connectivity, and even in total isolation. The requirement: the platform must remain useful when the link to the rear falls, and reconcile when it returns.

How we meet it: the declarative GitOps model fits disconnected operation naturally. An edge site keeps its own copy of the desired state and reconciles against local registries and repositories, so it operates without depending on a permanent connection; when the link returns, it synchronises. Inference is served locally with models already hosted on the node, with no external calls. Traces and audit events are stored locally and forwarded (store-and-forward) when there is a link, without losing the record. The high-availability control plane and the separation into sites allow controlled degradation instead of collapsing as a block. Resilience is not an add-on: it is a property of the declarative design and of sovereign inference.

Requirement: robustness and protection of the AI system itself (PRU reliability)

NATO’s reliability principle requires AI systems to have explicit and well-tested capabilities, and to be secure against manipulation. This adds a requirement that other sectors treat with less urgency: protecting the model itself, its weights and its behaviour, as a sensitive asset.

How we meet it: model weights are treated as sensitive data, hosted inside the perimeter and with access controlled by the same identity and network policies as everything else. The model gateway is the point where input and output safeguards are applied (validation, limits, filtering) and where a model can be constrained or disconnected. The signed supply chain (SBOM and Cosign) extends to model artefacts, so that only what has verifiable provenance is executed. And traceability feeds continuous evaluation: measuring the model’s behaviour over time is what makes it possible to detect degradation or manipulation before it affects the operation. Robustness is built with the same controls, identity, network, supply chain, traces, now applied to the model as well. Added to this are safeguards against adversarial inputs, prompt injection, manipulated data, at the gateway, which validates and constrains what goes into and out of the model. In defence, assuming the adversary will try to manipulate the AI is not pessimism: it is the design’s starting assumption.

Requirement: reliable execution of mission workloads and efficiency (availability, sustainability)

Serving AI in-house is only sustainable if the GPU, a scarce and expensive resource, is used well, and certain mission workloads must run reliably and to completion. The default Kubernetes scheduler schedules pod by pod, which for distributed workloads produces half-started jobs that hold GPUs without making progress.

How we meet it: GPU allocation is governed by Volcano (a CNCF incubating project), today in pre-production, with gang scheduling (PodGroup with minAvailable: the job starts complete or does not start), per-mission queues with quota and priority, fair-share (DRF) to share out fairly, pre-emption so that an urgent workload can overtake a deferrable one, and topology awareness (NVLink/PCIe) to reduce fragmentation. This raises the effective utilisation of the GPU fleet, which is what makes sovereign inference viable. On the state of the art: Kubernetes 1.34 took DRA (Dynamic Resource Allocation) to general availability, accelerator allocation by attributes instead of by count, and Kueue covers job quota management; both combine with a gang scheduler. It is managed with metrics: effective versus allocated utilisation, fragmentation, throughput per GPU and GPU-hours per mission. At the edge, where the GPU is even scarcer, these same techniques, together with quantised and smaller models, are what make it possible to serve useful inference with the hardware available instead of depending on a datacenter that may not be reachable.

Requirement: explainability, traceability and governability (NATO PRU, NIST AU)

Three of NATO’s six Principles of Responsible Use bear directly on infrastructure: explainability and traceability, governability and responsibility/accountability. Plus the audit (AU) controls of NIST. The requirement: being able to reconstruct every AI decision, keep human control and be accountable.

How we meet it: every inference is recorded as an auditable trace with Langfuse, input, output, model and version, cost, latency and mission, stored in our own components locked behind network policies that prevent it from leaving. That trace provides the traceability required by the PRU and by the NIST AU controls, and is the basis for human oversight, which is exercised at the gateway (review, limits and, crucially, the possibility of intervention and disconnection demanded by the governability principle: an AI system in defence must be able to be deactivated or constrained by the operator). An audit log is only worth anything as evidence if it is intact and ordered in time: that is why events are kept in append-only storage and clocks are kept synchronised by NTP. Instrumentation converges on the OpenTelemetry GenAI semantic conventions.

Accountability also requires being able to attribute a decision: which model, which version, which input and which operator were involved. That is why each trace links the inference to the identity that requested it and to the exact version of the model artefact that answered, so that the chain “who asked what, with which system and with what result” can be reconstructed. Without that attribution, NATO’s responsibility and accountability principle remains a statement of intent.

Requirement: continuous accreditation and reproducible operation (cATO, CMMC, ENS)

The sector has replaced one-off accreditation with cATO: an authorisation that is maintained as long as it is continuously demonstrated that the controls remain in place. The ENS and CMMC likewise require configuration management and traceability of changes.

How we meet it: the entire platform state, deployments, network policies, quotas, RBAC, identity, admission policies, is described in Git and applied by Flux over GitOps. Every change is a revision with an author, a date and a reviewer; every environment is reproducible from its repository. Security policies are code (Kyverno/OPA) evaluated continuously, not a list someone reviews once a year, and compliance monitoring is permanent. This is what cATO looks like in practice: an audit stops being a snapshot and becomes a query against a state that is maintained and verified without interruption, with the ability to reconstruct exactly what was deployed on a given date. A practical benefit is that compliance evidence generates itself: the declarative state in Git, the admission records, the image signatures and the traces are, together, the body of evidence an accreditation needs, produced continuously instead of gathered by hand the week before the inspection.

The piece that ties it all together: ISO/IEC 42001 and NATO’s principles

Everything above consists of technical controls that satisfy concrete requirements. But a management system is needed to order, evaluate and improve them, with defined roles and responsibilities. In defence, that governance has two complementary references: NATO’s Principles of Responsible Use, which set the ethical and operational “what”, and ISO/IEC 42001, the certifiable standard for AI management systems, which supplies the “how” of continuous management. Given that the AI Act excludes defence, this combination, PRU plus ISO 42001, is the governance backbone for responsible, demonstrable military AI. It is, precisely, the content of the AI Management Systems course we will publish shortly.

A view of the sector as of June 2026

It is worth pinning down the snapshot of the moment, because several currents have converged and they condition any platform decision.

An “AI-first” agenda and AI on classified networks. In January 2026, the US Department of Defense set an AI agenda at “war speed”, and through the spring agreements were closed to deploy generative AI models on classified networks. The signal is clear: military AI is moving from pilot to operation, which raises the demands on the infrastructure that supports it.

The substrate is hardened Kubernetes. Platform One, Big Bang and Iron Bank consolidate the pattern: Kubernetes clusters with STIG-compliant images, a signed supply chain and cATO. Defence AI is deployed on that same cloud-native substrate, not on bespoke systems.

Zero Trust with a deadline. Zero Trust adoption targets sit around 2027, which turns strong identity, microsegmentation and continuous verification into requirements with a calendar, not aspirations.

Supplier accreditation is tightening. CMMC Phase 2, in November 2026, ends self-attestation for anyone handling controlled information: third-party certification against the 110 NIST 800-171 controls will be required. The software supply chain, SBOM and Sigstore signing, moves from good practice to a contractual condition.

Data-centric security, operational at last. With STANAG 4774/4778 advancing through ratification, confidentiality labelling bound to the data becomes a real requirement for mission software, and not just a recommendation.

Defence-specific AI governance. NATO’s Principles of Responsible Use and the DARB are maturing as the reference framework, occupying the space the AI Act leaves free by excluding defence. And GPU economics, together with the maturity of open-weight models, pushes compute towards in-house infrastructure, precisely where classified data and disconnected operation demand it.

Europe pushes its own autonomy. In parallel, the EU is strengthening its defence industrial and technological base, through the European Defence Fund and the European Defence Agency, with growing emphasis on technological sovereignty. For a European platform, this reinforces the bet on in-house infrastructure, locally hostable models and minimal dependence on external services: exactly the direction it is built in.

Where we stand in that picture: with the platform on Kubernetes, eBPF as the network and security substrate, a signed supply chain, GPU scheduling maturing in pre-production, sovereign and disconnectable inference by design, traceability in auditable format and the management framework (PRU plus ISO 42001) as the next step. We are not chasing fashion; we are on the line the sector has decided to advance along.

Summary: requirement by requirement

Each rule or standard in the sector imposes a requirement, and each requirement is satisfied by a concrete piece of the infrastructure. Zero Trust and verified access, with Keycloak, mTLS and default-deny. The data-centric security of STANAG 4774/4778, with ABAC over classification labels. Segregation by mission and level, with multitenancy and, where appropriate, physical separation and cross-domain. Confidentiality and leak prevention, with Cilium in default-deny and Tetragon. A verifiable supply chain, with STIG-hardened images, SBOM and Sigstore signing plus admission control. Sovereignty and disconnected operation, with local inference over open models and internal registries. Reliable execution and efficiency, with Volcano. The explainability, traceability and governability of NATO’s PRU, with auditable traces and human control at the gateway. Continuous accreditation (cATO) and reproducible operation, with GitOps and policy as code. And the governance that ties it all together, with NATO’s PRU and ISO 42001.

It is not a PowerPoint: it is infrastructure that operates, that complies and that, being infrastructure, is measured by efficiency and cost. That, the cost control of an AI platform, is what the next article is about.

Sources

When MCP grows up: giving it authentication with Keycloak

Thu, 18 Jun 2026 08:30:00 +0200

When a child is small, it is enough to make sure they do not hurt themselves. You put a gate on the stairs, you cover the sockets and little else. But the child grows. They start going out alone, handling money, making decisions that have consequences. And then raising them stops being “don’t let them fall” and turns into something far more demanding: teaching them to identify themselves, to ask permission, to know how far they can go and to account for what they do.

That is exactly what has happened with our MCP servers.

MCP is no longer a toy

Not long ago, a Model Context Protocol server was an experiment running on someone’s laptop so a model could read a couple of files or call an internal API. It did not matter who used it: it was us, locally, with nothing at stake.

That is over. MCP servers have become infrastructure pieces that AI uses to touch real systems. And, like any child who grows up, they have developed new needs they did not have before:

Multitenancy: there is no longer a single user but many, and one user’s data cannot be visible to another.
Cost control: every model call consumes tokens, compute and, sometimes, paid APIs. You have to measure, attribute and set limits.
Authentication and authorisation: knowing who is on the other side, what permissions they hold and which specific server their credential is good for.

These are three separate fronts. Today we focus on the third, which is the one that opens the door to the other two: authentication. Without knowing who is calling, there is no tenant to separate and no cost to attribute.

Why MCP authentication is a problem with a name of its own

The good news is that nothing has to be invented here. The MCP specification did not pull a security mechanism of its own out of a hat: it builds on OAuth, the same standard that has been behind half the internet’s “Sign in with…” for years. MCP defines how an MCP server acts as a protected resource and delegates credential issuance to an authorisation server.

And that authorisation server can be Keycloak, the open source identity manager from the Cloud Native Computing Foundation that you probably already run for the rest of your applications. The idea is not to add one more identity silo just because we are now talking to models: reuse the one that already governs your users.

The important nuance is that MCP is not a single frozen standard but a living specification with several versions, and each one asks for different things. As of today four coexist:

2025-11-25 (the latest)
2025-06-18
2025-03-26
2024-11-05 (the initial one, which did not even contemplate authorisation)

Just as with raising a child what they need at 6 is not what they need at 16, what Keycloak has to satisfy depends on the MCP version you work with. Let us look at it.

What MCP demands from an authorisation server (and what Keycloak provides)

The specification lists a series of OAuth standards the authorisation server must support, with different levels of requirement (MUST, SHOULD, MAY). This is the picture, cross-referenced with what Keycloak supports:

Standard	2025-11-25	2025-06-18	2025-03-26	Keycloak
OAuth 2.1 Authorization Framework (draft)	MUST	MUST	MUST	Supported
OAuth 2.0 Authorization Server Metadata (RFC 8414)	MUST	MUST	MUST	Supported
Resource Indicators for OAuth 2.0 (RFC 8707)	MUST	MUST	—	Not supported
Dynamic Client Registration (RFC 7591)	MAY	SHOULD	SHOULD	Supported
OAuth Client ID Metadata Document (draft)	SHOULD	—	—	Supported (experimental)

One clarification: MCP also adopts OAuth 2.0 Protected Resource Metadata (RFC 9728), but that standard is the MCP server’s business, not the authorisation server’s, so it does not appear in the Keycloak table.

If we take as our conformance criterion that Keycloak “supports” a version when it meets all its MUST and SHOULD requirements, the result is this:

MCP version	Conformance
`2025-03-26`	Supported
`2025-06-18`	Partial, without Resource Indicators (RFC 8707)
`2025-11-25`	Partial, without Resource Indicators (RFC 8707)

The only piece Keycloak lacks today is Resource Indicators for OAuth 2.0 (RFC 8707), the resource parameter that ties a token to a specific server. The Keycloak community already has support for it on the roadmap; in the meantime there is a perfectly valid workaround we look at below. Following the analogy, it is that subject the child still struggles with but can pass with a study trick until they fully mature.

Down to work: setting up Keycloak for your MCP version

For MCP 2025-03-26

Nothing special has to be configured. This version does not require the resource parameter, so Keycloak complies out of the box. The child is still small and the gate on the stairs is enough.

For MCP 2025-06-18 and 2025-11-25: tying the token to its audience

Here is where it gets interesting. For security, these versions require an access token to be bound to the audience it was issued for. In plain terms:

The MCP client must include the resource parameter (from RFC 8707) in the authorisation and token requests, with the value that identifies the MCP server it is going to use.
The MCP server must validate that the tokens it receives were issued specifically for it.

This prevents a stolen or reused token from getting into a server other than the intended one. The problem, as we said, is that Keycloak does not yet understand the resource parameter.

The solution until native RFC 8707 support arrives is to use OAuth’s scope parameter to achieve the same effect. Picture this situation:

The MCP server is at https://example.com/mcp.
It supports three scopes: mcp:tools, mcp:prompts and mcp:resources.
The client asks for a token with resource = https://example.com/mcp and a combination of those scopes.
We want Keycloak to issue a token whose aud (audience) is precisely https://example.com/mcp.

To achieve it, we configure Keycloak like this, repeating the pattern for each scope:

Create a client scope mcp:tools of type Optional and add an Audience mapper whose Included Custom Audience is https://example.com/mcp.
Create a client scope mcp:prompts of type Optional with its Audience mapper pointing at https://example.com/mcp.
Create a client scope mcp:resources of type Optional with its Audience mapper pointing at https://example.com/mcp.

The key is that the Included Custom Audience of each client scope is identical to the value of the request’s resource parameter and to the MCP server URL. With this, if the client asks for a token with those three scopes, Keycloak issues something like:

{
 "aud": "https://example.com/mcp",
 "scope": "mcp:resources mcp:tools mcp:prompts"
}

The MCP server can now check the aud and reject any token that was not meant for it. We have tied the credential to its recipient without waiting for RFC 8707.

If you use MCP Inspector

MCP Inspector is the official tool for debugging MCP servers, and it registers clients dynamically against Keycloak by running JavaScript from its backend. For it to work you have to adjust CORS in the client registration anonymous access policies:

Allowed Client Scopes: include the scopes your MCP server supports.
Allowed Registration Web Origins: include the web origin of the MCP Inspector backend.
Trusted Hosts: include the host or IP of the machine sending the dynamic registration request, that is, where your browser runs.

For MCP 2025-11-25: client registration with Client ID Metadata Document

The latest version adds a new chapter: how clients register. It contemplates three approaches, and you choose according to your scenario:

Client ID Metadata Documents: when client and server do not know each other beforehand (the most common case).
Pre-registration: when a prior relationship already exists.
Dynamic Client Registration: for backward compatibility or specific requirements.

Keycloak supports the first, OAuth Client ID Metadata Document (CIMD), although it is worth knowing that it is an experimental feature: it may bring breaking changes in future versions. To enable it, start Keycloak with the flag:

bin/kc.sh start --features=cimd

The idea behind CIMD is that the client_id stops being an opaque identifier and becomes a URL pointing at a document with the client’s metadata. Keycloak downloads that document and processes the request with what it finds there. For it to do so, you have to create a client policy profile and a policy that trigger it.

Profile. In Realm Settings → Client Policies → Profiles, create a profile (e.g. cimd-profile), add the client-id-metadata-document executor and configure it:

Allow http scheme: allows http for the URLs (client_id, client_uri, logo_uri, jwks_uri, etc.). Development only; in production OFF.
Trusted domains: wildcard patterns of accepted domains (e.g. *.example.org). If empty, all are denied.
Restrict same domain: if ON, it requires the client_id URL, the redirect_uri and the metadata URLs to all be under the same trusted domain.
Required properties: properties the metadata document must include.
Only Allow Confidential Client: if ON, only confidential clients are accepted (with jwks/jwks_uri and private_key_jwt or tls_client_auth authentication).

Policy. In Realm Settings → Client Policies → Policies, create a policy (e.g. cimd-policy), add the client-id-uri condition and configure it:

URI scheme: schemes to recognise in the client_id (in production, https only).
Trusted domains: domains accepted in the client_id host. If filled in, the condition is only true when the host matches; if left empty, it is always false.

Then associate the cimd-profile with this policy. From then on, when a request arrives with a client_id that is an https URL from a trusted domain, Keycloak downloads the metadata document and uses it to process the request.

Global executor settings. Some parameters are not in the console and are passed as SPI options at startup:

min-cache-time: minimum document cache time (300 s by default).
max-cache-time: maximum cache time (259200 s by default, 3 days).
upper-limit-metadata-bytes: maximum document size (5000 bytes by default).

bin/kc.sh start \
 --spi-client-policy-executor--client-id-metadata-document--min-cache-time=600 \
 --spi-client-policy-executor--client-id-metadata-document--max-cache-time=86400 \
 --spi-client-policy-executor--client-id-metadata-document--upper-limit-metadata-bytes=10000

A real case: VS Code desktop as an MCP client

Visual Studio Code is an MCP client that uses CIMD. When it connects to an MCP server that requires authorisation, it sends a client_id that is an https URL hosted on vscode.dev (e.g. https://vscode.dev/mcp-client), and Keycloak downloads its metadata from there. The detail to bear in mind is that VS Code is a public client using PKCE (no client secret) and, for the OAuth callback, it starts a local server with a redirect_uri of the form http://127.0.0.1:<port>/callback. Since that redirect is not in the vscode.dev domain, you have to leave Restrict same domain OFF.

Start with --features=cimd and configure:

Profile vscode-cimd-profile (client-id-metadata-document executor):

Allow http scheme: OFF
Trusted domains: vscode.dev, 127.0.0.1
Restrict same domain: OFF (the redirect is a localhost, not vscode.dev)
Only Allow Confidential Client: OFF (VS Code is public)

Policy vscode-cimd-policy (client-id-uri condition):

URI scheme: https
Trusted domains: vscode.dev
Associate the vscode-cimd-profile profile.

With this, when VS Code launches the request, Keycloak recognises the client_id as a vscode.dev URL, downloads the Client ID Metadata Document and completes the OAuth flow with the callback on localhost.

The education continues

We have taught our MCP to identify itself and to ask permission in a standard way, reusing the identity Keycloak already governs in the rest of the house. That is no small thing: with a token properly tied to its audience, everything else starts to become possible.

But, as with any child who grows up, this does not end here. The other two subjects we mentioned at the start are still pending, separating the tenants (multitenancy) and controlling what it spends (costs), and Keycloak still has some maturing to do in places, such as native Resource Indicators support. We will cover them in future posts. For now, our MCP already knows how to say who it is. And in security, that is the first day of school.

Sources

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

Tue, 09 Jun 2026 17:00:00 +0200

Operational companion to The contractor with the master key. That post explains the why and the where: the threat model, the five isolation families, which domain uses each. This one is the how, with commands. If you have not read it, read it first: here I take for granted what the blast radius is, why bwrap runs without root and what Tetragon watches. The procedure comes in two independent tracks, client and cluster, because, as the sibling post argues, the control extrapolates but the primitive gets rewritten.

TL;DR

Two reproducible procedures. Client (workstation): install ai-jail (it wraps bubblewrap), generate the per-project .ai-jail, audit with --dry-run, fix the allowlists with --bootstrap, use --lockdown for anything you do not trust, and leave the agent without git push permission. Cluster (RKE2 with Cilium + Tetragon): put the pod baseline in place (unprivileged securityContext, seccomp: RuntimeDefault, default-deny NetworkPolicy), put the untrusted agent’s pod in a microVM with runtimeClassName: kata, and deploy the Tetragon TracingPolicy files in two phases: observe with action: Post to raise the baseline, then promote to action: Sigkill on tcp_connect (egress) and security_file_open (secret paths). The golden rule of the Tetragon phase: adopt first, block later; never put a Sigkill into production without having seen the events in observation mode first.

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

ai-jail wraps the sandbox; on Linux it needs bubblewrap separately, on macOS it needs no extra dependency.

# ai-jail (macOS and Linux)
brew tap akitaonrails/tap && brew install ai-jail
# or, with cargo:
cargo install ai-jail
# or, with mise:
mise use -g ubi:akitaonrails/ai-jail

# bubblewrap on Linux (pick your distro)
sudo pacman -S bubblewrap # Arch
sudo apt install bubblewrap # Debian / Ubuntu
sudo dnf install bubblewrap # Fedora

Check the binary is there and that bwrap runs without root (it must not ask for sudo):

ai-jail --version
bwrap --ro-bind / / --unshare-all echo "bwrap ok without root"

If bwrap fails asking for privileges, your kernel has unprivileged user namespaces disabled; enable them (sysctl kernel.unprivileged_userns_clone=1 on older Debian/Ubuntu) before continuing.

A1 — The per-project .ai-jail file

On the first launch inside the project, ai-jail creates a .ai-jail (TOML) that is committable to the repo: any colleague who clones inherits the same policy.

cd ~/Projects/mi-app
ai-jail claude # creates .ai-jail and launches Claude Code inside the sandbox

The generated file:

# .ai-jail — sandbox configuration (commit it to the repo)
command = ["claude"]
rw_maps = [] # extra writable directories
ro_maps = [] # extra read-only directories

Before trusting the sandbox, audit it. --dry-run --verbose prints every mount point, every isolation flag and the full bwrap command, without executing anything:

ai-jail --dry-run --verbose claude

Read the output and confirm three things: that $HOME is mounted as a tmpfs (not the real one), that ~/.ssh, ~/.aws and ~/.gnupg do not appear among the mounts, and that the only writable directory is the project one. If you need an extra directory:

ai-jail --rw-map ~/Projects/shared-lib claude # extra, writable
ai-jail --map /opt/datasets claude # extra, read-only

Other agents, same binary:

ai-jail codex
ai-jail opencode
ai-jail bash # bare shell to debug the sandbox
ai-jail -- python script.py # any command

A2 — Permission allowlists with –bootstrap

--bootstrap generates each agent’s permission configuration, with sensible allow/deny/ask, and takes a backup before overwriting:

ai-jail --bootstrap

What it produces, in summary:

Agent	File	Base policy
Claude Code	`~/.claude/settings.json`	allow: `git status/diff/log`, `ls`, `grep`, `cargo`, `npm`, `python`, `docker compose` · ask: `git push`, `rm`, `docker run` · deny: `rm -rf`, `sudo`, `chmod 777`, `git push --force`
Codex	`~/.codex/config.toml`	`approval_policy = "on-request"`
OpenCode	`~/.config/opencode/opencode.json`	permissions for `bash`, `edit`, `write`

The operational key: git push is under ask, not allow, and git push --force under deny. The agent can commit, branch and rebase locally as much as it wants; none of that touches the remote. (If you use the Claude Code /sandbox, also set "allowUnsandboxedCommands": false to close the dangerouslyDisableSandbox escape hatch, which is opt-out out of the box.)

A3 — Lockdown for anything you do not trust

To audit third-party code or run an agent over a project you do not know, --lockdown goes further: project mounted read-only, GPU/Docker/display disabled, --rw-map/--map ignored, $HOME a pure tmpfs with no host dotfiles, network cut with --unshare-net and the environment wiped with --clearenv.

ai-jail --lockdown bash

It is the most restrictive sandbox possible short of a VM. Use it as your mental default for anything that is not your own code on your own machine.

A4 — The safety net: git with no push

It is not a flag, it is a property of the environment that changes the risk arithmetic. If the project is in git with a remote, and the agent does not have push permission, the worst case, that it corrupts every file in the project, is reverted with:

git checkout . # back to the last commit
# and if it touched .git (unlikely): delete the dir and re-clone

The remote was never touched. Sandbox for the filesystem + git for the code + manual push is already a reasonable level for daily use: ai-jail protects your data and the system, git protects the code, and the decision to publish is still yours.

Track B — Cluster (RKE2 with Cilium + Tetragon)

The untrusted agent, or the inference that executes generated code, runs as a pod. The same principle as on the client, different primitives. We assume a generic RKE2 cluster with Cilium as CNI and Tetragon already deployed (the eBPF agent DaemonSet on every node).

B0 — The pod baseline

Before any eBPF, the standard kit. Unprivileged securityContext, read-only root, default seccomp:

apiVersion: v1
kind: Pod
metadata:
 name: ai-agent
 namespace: agentes
 labels:
 app: ai-agent
spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 10001
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: agent
 image: registry.interno/ai-agent:pinned
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop: ["ALL"]
 volumeMounts:
 - { name: work, mountPath: /work }  # the only writable one
 volumes:
 - name: work
 emptyDir: {}

And the default egress cut, the cluster twin of --unshare-net. A default-deny egress NetworkPolicy in the namespace, opening only DNS and the bare essentials:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-egress
 namespace: agentes
spec:
 podSelector: {}
 policyTypes: ["Egress"]
 egress:
 - to:
 - namespaceSelector:
 matchLabels: { kubernetes.io/metadata.name: kube-system }
 ports:
 - { protocol: UDP, port: 53 }
 - { protocol: TCP, port: 53 }

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

For genuinely untrusted code, take it out of the shared kernel. With Kata deployed there is a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
 name: kata
handler: kata

And the pod asks for it with one line, runtimeClassName: kata, running in its own microVM with a dedicated kernel instead of sharing the node’s:

spec:
 runtimeClassName: kata  # ← the pod runs in a microVM, not on the node's kernel
 # ...the rest the same as B0

It is the cluster twin of isolation by construction: a kernel exploit inside the pod does not reach the node.

B2 — Tetragon, observation phase (Post)

Now the layer that separates a platform with runtime visibility from one without. Observe first, never kill from the outset. A TracingPolicyNamespaced, scoped to the namespace and to the agent’s label, that reports (does not kill) three things: process executions, network connections and opens of sensitive paths. action: Post only emits the event.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-observa
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- outbound connections ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchActions:
 - action: Post
 # --- opens of sensitive files ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets"
 - "/work/.git/config"
 matchActions:
 - action: Post

(Process executions need no kprobe: Tetragon emits process_exec/process_exit natively.) Deploy it and watch the events live from the node’s Tetragon pod:

kubectl apply -f agente-observa.yaml
# readable events, filtered by namespace:
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
 tetra getevents -o compact --namespace agentes

Leave this running for a typical working day of the agent. Note which destinations it really connects to (your internal registry, your HF mirror, your vLLM endpoint) and which paths it opens. That is your baseline: the list of what is legitimate. Without this step, a Sigkill kills good work and generates an availability incident, precisely what ENS asks you to avoid.

B3 — Tetragon, enforcement phase (Sigkill)

With the baseline in hand, promote to blocking. Two rules. The first: kill any connection whose destination is not on the allowlist — NotDAddr inverts the match: it fires for everything that is not those networks. The second: kill any attempt to open a secrets path.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-enforce
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- egress: kill everything that is NOT on the allowlist ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr"
 values:
 - "127.0.0.1"
 - "10.0.0.0/8" # internal cluster network
 - "172.16.10.20" # internal registry (example)
 matchActions:
 - action: Sigkill
 # --- reading secrets: kill the process ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets/kubernetes.io/serviceaccount/token"
 - "/work/.ssh"
 matchActions:
 - action: Sigkill

kubectl apply -f agente-enforce.yaml

Now the agent can do whatever it likes inside the pod, but the instant it tries to connect to a disallowed destination or read the service account token, Tetragon kills it in the kernel, before the packet leaves or the read returns bytes. It is the cluster twin of the curl blocklist and the unmounted ~/.ssh, but applied at runtime and over any binary, not just the ones you know about.

Operational warning. Enforcement with Sigkill requires a recent kernel with eBPF support for the action (5.10+ is safe). Deploy agente-enforce in a test namespace first, and keep agente-observa active in parallel: if the block fires, the Post event tells you exactly what triggered it. Adopt first, block later.

The client ↔ cluster equivalence table

The same vector, the two primitives. This is “extrapolating the technology” made explicit:

Threat vector	Client (workstation)	Cluster (RKE2)
`$HOME` / writable root	`$HOME` as ephemeral tmpfs (`bwrap`)	`readOnlyRootFilesystem: true` + `emptyDir`
Arbitrary egress	`curl`/`wget` blocklist · `--unshare-net`	default-deny NetworkPolicy + Tetragon `NotDAddr`→`Sigkill`
Reading secrets	`~/.ssh`/`~/.aws`/`~/.gnupg` not mounted	secrets outside the pod + Tetragon `security_file_open`→`Sigkill`
Kernel escape	Landlock (2nd VFS barrier)	`runtimeClassName: kata` (microVM, own kernel)
No escape hatch	process inside `bwrap`, no way out	no `privileged`, `drop ALL`, `allowPrivilegeEscalation:false`
Damage to the code	git remote with no `push` → `git checkout .`	GitOps + PR review, the agent does not apply to `main`
Visibility	`--dry-run --verbose` (static, pre-run)	Tetragon `tetra getevents` (dynamic, at runtime)

Checklist of gotchas

Do not put in a Sigkill without passing through Post. The observation baseline is not optional: it is what separates “blocking a C2” from “killing your own fine-tuning job”.
The .ai-jail gets committed; secrets do not. The TOML is policy, not credentials. Check you are not putting paths holding sensitive data into rw_maps.
readOnlyRootFilesystem breaks apps that write to /tmp. Mount an emptyDir at /tmp as well as the work one.
A NetworkPolicy with no DNS rule leaves the pod blind. Open port 53 to kube-system or nothing resolves.
Kata is not free. It adds startup latency and not every workload with special devices (GPU passthrough) fits; reserve it for the untrusted, not for everything.
The Claude Code /sandbox does not cover MCP or hooks unless you enable sandbox-runtime. If your agent uses MCP servers, assume they run with full permissions until you do.
NotDAddr with literal IPs ages badly. Document the allowlist and review it when the registry or the inference endpoint changes; consider stable internal CIDRs instead of loose IPs.

References

ai-jail (Fabio Akita), GPL-3.0: https://github.com/akitaonrails/ai-jail
bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io
Tetragon — TracingPolicy: https://tetragon.io/docs/concepts/tracing-policy/
Tetragon — enforcement (Sigkill/Override): https://tetragon.io/docs/concepts/enforcement/
Kata Containers — Kubernetes RuntimeClass: https://katacontainers.io
Kubernetes — Pod Security & seccomp: https://kubernetes.io/docs/tutorials/security/seccomp/
Kubernetes — Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Cilium: https://cilium.io

The contractor with the master key: isolating AI agents from workstation to cluster

Tue, 09 Jun 2026 16:00:00 +0200

First post of a pair on AI agent isolation. This one fixes the what and the where: the complete map of isolation primitives and which domain each belongs to. The sibling runbook fixes the how, with commands: ai-jail and bubblewrap on the client, Tetragon TracingPolicy and RuntimeClass on the cluster. If you are only going to read one, this one gives you the mental model; the other one gives you the files to copy and paste.

TL;DR

An AI agent that runs code needs access to your filesystem and your tools: compiler, linter, grep, make, cargo, npm. That is the minimum to be useful. The problem is that alongside that access travels the ability to read ~/.aws/credentials, exfiltrate your SSH keys or fire an rm -rf outside the project directory. And no malicious model is needed: a compromised dependency in an npm install is enough, because the well-meaning agent and the poisoned post-install script run with the same permissions. The answer is not to trust the LLM’s good intentions; it is to isolate in order to bound the blast radius. This post walks the five isolation families of 2026, from the process sandbox to the full VM, and sorts them into two columns: what applies on the client (the developer’s workstation: bubblewrap, ai-jail, sandbox-exec, Landlock, the native Claude Code and Codex sandboxes) and what applies on the cluster (where the agent or the inference runs on Kubernetes: namespaces+seccomp, gVisor, Firecracker/Kata microVMs and eBPF/Tetragon as the layer for live observation and enforcement). The thesis: the threat model is the same in both places; the tools are not. Policy extrapolates; the primitive gets rewritten.

The analogy: the contractor with the master key

You hire a builder for a renovation. He is competent and acts in good faith. But two things happen that you do not control. The first: he may misread the instruction and knock down the wrong partition wall. The second, worse: his toolbox may have been tampered with before he came through your door — somebody put something inside — and when he opens it in your living room, that something activates.

Nobody sensible gives him the master key to the whole building. You open the room he works in, you leave him the tools he needs, and you keep the study with the safe and the server room locked. If the renovation goes wrong, by error or by sabotage, the damage stays in that room.

An AI agent is that contractor. The sandbox is the key policy: you give it the project room and the tools, not the master key to the system. And here is the twist that justifies two posts: the builder works in two different buildings. One is your flat, the developer’s workstation, with your credentials, your ~/.ssh, the browser’s password vault. The other is the datacenter, the cluster where inference and autonomous agents serve customers, with data from several tenants at once. The key policy is identical in both: least privilege, bound the radius. But the lock on your flat’s door is not the same as the one at the datacenter. In the flat you fit a cylinder lock (bubblewrap). At the datacenter you put a guard watching every door and a separate wing of the building (Tetragon + microVM). Same principle, different hardware. That is extrapolating the technology, not copying it.

The threat model: what a runaway agent can do

Before choosing a lock it helps to enumerate the burglar. The attack surface of an agent that executes arbitrary bash breaks down into five concrete threats. Not all are defended by the same layer and, an uncomfortable fact that the security documentation itself acknowledges, no single layer covers them all.

Threat	What the agent does	Minimum layer that cuts it
Filesystem out of scope	Reads `.env`, `~/.ssh/id_rsa`, system secrets; modifies sources outside the project	Process sandbox (path allowlist)
Arbitrary network egress	Exfiltrates data, receives instructions from a remote C2, calls unauthorised APIs	Network blocking / NetworkPolicy / microVM
Kernel syscall surface	A kernel exploit from inside the container escalates to the host (shared kernel)	gVisor or microVM (dedicated kernel)
Cross-tenant leakage	One customer’s workload reads another’s data on a multi-tenant platform	microVM (de facto standard)
Secret exfiltration	Extracts tokens and environment variables via `/proc` or the environment	`--clearenv` / tmpfs `$HOME` / secrets outside the pod

There is a sixth threat that no sandbox solves: prompt injection. If an attacker manages to slip instructions into the agent’s context — a poisoned comment in the code, a malicious file the agent reads, an adversarial response from a tool — the agent will execute those instructions with whatever permissions the sandbox grants it. Isolation shrinks the impact radius of a successful injection; it does not prevent the injection. That is why the sandbox is a layer, not the solution: on top go input validation, tool-call allowlists and output auditing. The sentence to internalise: isolation does not make the agent trustworthy; it bounds what an untrustworthy agent can break.

Two domains, one policy

The builder works in two buildings. The distribution of tools reads better in two columns:

The client: isolating the agent on the workstation

Here the agent is a coding assistant — Claude Code, Codex, OpenCode, Cursor — that a developer launches on their machine. One process, one user, and right next to it the juiciest assets that exist: ~/.aws/credentials, ~/.ssh, ~/.gnupg, the browser’s password store. The matching tier is the lightest one: the process sandbox.

bubblewrap (Linux) and sandbox-exec (macOS). bubblewrap (bwrap) is the same sandbox Flatpak uses to isolate each desktop app: a ~50 KB binary, ~4,000 lines of C, maintained by the GNOME team and, the key property, it runs without root via CLONE_NEWUSER, creating namespaces with no elevated privileges. It mounts $HOME as an ephemeral tmpfs and only exposes the project directory, writable; the rest of the system becomes invisible. On macOS the equivalent is sandbox-exec with SBPL profiles: a legacy Apple API, officially deprecated and with no public replacement, but working today. Parity between the two is not exact — on macOS the GPU (Metal) and the display (Cocoa) are system services and sandbox-exec does not restrict them — but both protect what matters: access to the sensitive areas of the filesystem.

Landlock as a second barrier. bubblewrap isolates via namespaces and mounts; Landlock, a Linux Security Module available since kernel 5.13, restricts access at the VFS level, independently of namespaces. It does not replace bwrap: it complements it. It closes vectors that mount-based isolation does not cover on its own (escape routes via /proc, symlink tricks inside allowed mounts) and acts as a safety net should the namespace machinery have a bug. It is defence in depth inside the client itself, and it degrades cleanly to a no-op on kernels that do not support it.

Dev containers, when reproducibility is needed. A dev container (devcontainer.json, what Codespaces and Cursor use) is a Docker container with a configuration layer on top. It gives reasonable filesystem isolation and easy reset (destroy and recreate), but it shares the host kernel, the same limitation as any Docker, and it tends to be long-lived, accumulating state. For an agent running your own team’s code on your own machine it is a good repeatability story; it is not, on its own, the isolation layer for untrusted code.

What wraps all of this. A hand-written bash script works, but it does not scale to a team. The tools that package it:

ai-jail (Rust, GPL-3.0): wraps bwrap/sandbox-exec with per-project config in a .ai-jail file (TOML, committable to the repo, so the whole team inherits the same policy), auto-detection of GPU/Docker/display, a --lockdown mode (project read-only, network cut with --unshare-net, --clearenv), --dry-run for auditing, and --bootstrap to generate each agent’s permission allowlists. It is tool-agnostic: the same binary serves Claude, Codex, OpenCode or Crush. It also applies Landlock automatically on 5.13+ kernels as defence in depth.
The Claude Code /sandbox: since October 2025 Claude Code ships its own sandbox that uses, exactly, bubblewrap on Linux and sandbox-exec on macOS. Its Sandboxed Bash isolates shell commands, but not the file tools, the MCP servers or the hooks, which run with the process’s full permissions unless you enable the beta sandbox-runtime package, which wraps the whole process. There is one nuance worth knowing: if a command fails because of a restriction, the agent can retry with dangerouslyDisableSandbox — it is opt-out, not opt-in.
Codex CLI: three modes via --sandbox (read-only, workspace-write, danger-full-access); the recommended default is workspace-write. The philosophy is deliberate: Codex does not provide the isolation, it delegates it to the environment that wraps it. danger-full-access only makes sense inside a microVM.
Cursor: its cloud agents run in isolated VMs; /worktree creates a single-use isolated worktree per task, and /best-of-n launches several attempts in parallel in separate worktrees.

The cluster: isolating the agent in production

The second building is the datacenter. Here the “agent” can be an autonomous agent running with no human in front of it, or the inference service itself executing generated code, or a multi-tenant workload where one customer’s pod must not touch another’s data. The process is no longer one: they are pods in a Kubernetes cluster (RKE2/RKE3 on a typical sovereign platform). The primitives change in nature.

The pod baseline. First of all, the standard kit: Linux namespaces, seccomp (RuntimeDefault) to trim the syscall surface, cgroups for resource limits, an unprivileged securityContext (runAsNonRoot, readOnlyRootFilesystem, dropping all capabilities) and NetworkPolicy to cut egress. It is the cluster equivalent of the process sandbox allowlist. Necessary, but it shares the kernel with the host: insufficient for genuinely untrusted code.

gVisor (runsc). Google’s user-space kernel: it intercepts the workload’s syscalls before they reach the host kernel and serves them inside a Linux kernel reimplemented in Go (the Sentry). The surface exposed to host kernel vulnerabilities drops sharply, while keeping fast startup and a low footprint. It is the middle ground when the risk of a kernel escape is real but the overhead of a microVM is not acceptable.

Firecracker / Kata microVMs. The de facto standard for untrusted code in 2026. Firecracker (AWS’s VMM written in Rust, on top of KVM) gives each sandbox a dedicated Linux kernel: a kernel exploit inside the microVM does not reach the host by construction. It is what sits underneath Vercel Sandbox (GA January 2026) and E2B. On Kubernetes, Kata Containers brings that model to a RuntimeClass: you mark the untrusted agent’s pod with runtimeClassName: kata and it runs in its own microVM instead of sharing the node’s kernel. For multi-tenant with generated code, this is the baseline, not the luxury.

eBPF / Tetragon: the layer we already have. Here is the piece that separates a platform with runtime observability from one that only trusts its configuration. The previous layers are static: they define what the pod can do before it starts. Tetragon, Cilium’s runtime security component, based on eBPF, is dynamic: it observes, in the kernel and at minimal cost, every process execution, every network connection and every file open in every pod, and it can act inline. It does not replace the sandbox; it watches it from inside the kernel. Where bubblewrap on the client blocks curl with a command blocklist, Tetragon on the cluster hooks tcp_connect in the kernel and, if the destination is not allowed, kills the process with Sigkill before the packet leaves. Where the client hides ~/.ssh behind a tmpfs, Tetragon hooks security_file_open and reports, or kills, any attempt to read a sensitive mounted path. It is the guard patrolling the corridors while the microVMs are the walls. And it is exactly the kind of control that materialises the ENS monitoring and traceability measures (op.mon, op.exp) without instrumenting the application: visibility lives in the kernel, not in the agent’s code.

The landscape table

The five families, their relative isolation strength, their startup cost and the domain where they live:

Tier	Primitive	Isolation	Startup	Natural domain
Process sandbox	Seatbelt · bubblewrap	Baseline	~0 ms	Client (Claude Code default)
Dev container	Docker + seccomp	Moderate	seconds	Client / cluster (repeatability)
User-space kernel	gVisor (`runsc`)	Strong	ms	Cluster (medium multi-tenant)
microVM	Firecracker · Kata	Strongest (practical)	<1 s	Cluster (untrusted code)
Full VM	KVM · EC2	Maximum	30 s+	Cluster (external boundary, compliance)
Runtime enforcement	eBPF · Tetragon	Cross-cutting	always on	Cluster (watches+kills on top of any tier)

Tetragon gets its own row on purpose: it is not a rung on the ladder, it is a cross-cutting layer that operates on top of any of the others. It stacks with all of them.

A numerical note on why the “startup” column decides as much as the “isolation” column. A full VM wins on raw isolation but takes tens of seconds to provision; for an agent that needs a fresh environment per request or per session, that cost is prohibitive. A Firecracker microVM starts in under 1 second and a process sandbox in ~0 ms. That is why the dominant pattern in 2026 is not “the most isolated VM”, but a full VM as the external boundary plus a microVM as the per-request execution unit inside, the architecture of Vercel, AWS Lambda and E2B. On the client the arithmetic is the opposite: the developer launches the agent dozens of times a day interactively, and a startup measured in seconds would break the flow; hence the process sandbox, with its microsecond overhead, is the correct default.

Extrapolate, do not copy

The thesis of this pair of posts fits in one sentence: the threat model is invariant across domains; the primitive that implements it is not. The client and the cluster defend exactly the same five vectors — filesystem, network, kernel, multi-tenant, secrets — but with toolboxes that do not overlap. Every control has its twin on the other side:

$HOME as an ephemeral tmpfs (client) ↔ readOnlyRootFilesystem + emptyDir (cluster).
A curl/wget blocklist in bwrap (client) ↔ a TracingPolicy on tcp_connect in Tetragon plus NetworkPolicy (cluster).
--unshare-net in lockdown (client) ↔ default-deny NetworkPolicy (cluster).
No escape hatch, the process lives inside bwrap (client) ↔ no privileged, no hostPath, RuntimeClass kata (cluster).
~/.ssh and ~/.aws never mounted (client) ↔ secrets outside the pod plus Tetragon watching security_file_open (cluster).

The sibling runbook turns each of these equivalences into concrete files. What matters here is the method: when someone shows you an agent sandbox, be it the Claude Code /sandbox on a laptop or a microVM on a PaaS, the useful question is not “which tool does it use?”, but “which of the five vectors does it close, and which does it leave open?”. The tool gets replaced; the threat map stays.

What no sandbox solves

Three limits that Anthropic’s own documentation states, and which are worth keeping in view so as not to sell smoke:

Egress remains a risk in any sandbox that allows outbound connections. If the agent can open a connection, it can exfiltrate. That is why the client lockdown cuts the network and the cluster uses default-deny NetworkPolicy plus Tetragon: you do not trust “filtering well”, you trust “not letting anything out”.
Code modification remains possible in any sandbox with the project directory mounted writable. The remedy is not a sandbox-technical one, it is git: with the remote untouched and no push permission, the worst case is corrupting the local working copy — git checkout . and start again. The damage never reaches the remote.
No sandbox prevents a compromised prompt from reaching the API. Isolation bounds the impact of an injection; it does not prevent it. The complementary defences — input validation, tool-call allowlists, output auditing — are mandatory alongside isolation, not instead of it.

The operational conclusion: isolation shrinks the blast radius; defence in depth is what closes the circle. A process sandbox for trusted code on a known machine is appropriate and practically free. For an agent acting on user prompts, executing generated code or running multi-tenant, the acceptable minimum in 2026 is a microVM, with Tetragon watching from above. Pick the tier that matches your real threat, check which vector it leaves open, and stack complementary controls on top.

References

bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io · https://docs.rs/landlock
ai-jail (Fabio Akita): https://github.com/akitaonrails/ai-jail
gVisor: https://gvisor.dev
Firecracker: https://firecracker-microvm.github.io
Kata Containers: https://katacontainers.io
Tetragon (Cilium): https://tetragon.io
Vercel Sandbox — concepts: https://vercel.com/docs/vercel-sandbox/concepts
E2B: https://github.com/e2b-dev/E2B
Claude Code sandboxing: https://docs.claude.com/en/docs/claude-code
Codex CLI: https://github.com/openai/codex

Tetragon: Cilium's security cousin that sees every syscall in the kernel

Tue, 19 May 2026 05:00:00 +0200

TL;DR

Tetragon is the runtime security and observability engine that the Cilium project published as a companion to the CNI. Its job is not to route packets, Cilium is already there for that, but to observe what happens inside the node’s processes in real time: which binary runs in each pod, which files it opens, which syscalls it invokes, which capabilities it asks for, which network connections it establishes, which kernel modules get loaded. It does this by loading eBPF programs into the kernel’s hook points (kprobes, tracepoints, uprobes, LSM hooks) and filtering the relevant events inside the kernel itself with a declarative language expressed as a CRD (TracingPolicy and TracingPolicyNamespaced). The result is a stream of events enriched with Kubernetes metadata (pod, namespace, labels) that costs less than 1% of CPU and, the thing that sets Tetragon apart from the competition, can block actions inside the kernel, killing the process with SIGKILL or overwriting a syscall’s return value, before they finish executing, with no race conditions. Against Falco (which parses syscalls in userspace, 5-10% overhead, detection-only), Tetragon is “cheaper and with enforcement”; against the bare kernel, it is “a declarative layer your operations colleague can read”. This article is the extensive introduction you need to take it on seriously: architecture, all the hooks and selectors, the operating modes, a guide to use cases (exec auditing, sensitive file access, container escape, cryptomining, rootkit detection, network observability) and the traps you see in production.

This article is part 2 of the eBPF series. Part 1, eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack, covered basic eBPF, the networking hooks (XDP, TC, sock_ops), how Cilium implements the datapath and the BGP Control Plane v2 CRDs. Here we take those same eBPF hooks and use them for something different: observing and, if needed, stopping what the cluster’s processes do.

The analogy: auditd on steroids in eBPF

Anyone who has spent a few years administering Linux has used auditd. It is the classic kernel subsystem for auditing syscalls: you configure a rule with auditctl (for instance, “monitor any open on /etc/shadow”) and the kernel sends events to a userspace daemon that persists them. It works, but it has two limitations that weigh heavily on modern Kubernetes clusters:

No Kubernetes context. auditd reports processes by PID and UID. Knowing which pod, which namespace, which image, which labels, the information that actually matters when responding to an incident, requires correlating afterwards with data from cri-o or containerd. It is operationally miserable.
No granular enforcement. auditd can generate events, but it cannot take the decision to kill the offending process before the syscall finishes. You leave that to a higher layer that reads the events, processes them and kills the process… if it gets there in time. A race by design.

Tetragon is auditd on steroids: the same conceptual ideas, hooks on syscalls and events to userspace, but implemented with modern eBPF, with filtering inside the kernel so you do not pay the cost of waking the daemon for every irrelevant syscall, with Kubernetes metadata injected by an agent that knows the cluster, and with actions executed inside the kernel itself without waiting for userspace to decide. If the rule says “kill any process that opens /etc/shadow from the prod namespace”, the decision is taken in the kernel kprobe and SIGKILL is delivered before the open completes. There is no race; there is no window between detection and action.

What Tetragon is, architecturally

Tetragon is an agent deployed as a DaemonSet (one pod per node) and a set of CRDs that define the policies to apply. The agent has four responsibilities:

Load eBPF programs into the hook points the active TracingPolicies demand.
Maintain a cache of Kubernetes metadata (pods, namespaces, labels) by reading the API server, so it can enrich every event with the right context.
Collect the events the eBPF programs emit (via ring buffers) and serialise them.
Export the events to configurable destinations: stdout JSON (typical in sidecars or log-collection agents), gRPC streaming (to consume them from Hubble or another consumer), a file, or Fluentd/Loki/SIEM.

The eBPF programs are not written by the user. Tetragon generates the bytecode from the TracingPolicies: it reads the declarative policy, decides which hooks to attack, which arguments to read from the kernel, which filters to apply inline and which actions to execute. The user only writes YAML.

This separation of declarative policy → generated eBPF bytecode is what makes Tetragon usable. Writing eBPF programs by hand is a specialist’s job; writing a TracingPolicy is the job of an SRE with a good example in front of them.

The two CRDs: TracingPolicy and TracingPolicyNamespaced

Tetragon exposes exactly two main CRDs:

TracingPolicy (cluster-scoped, cilium.io/v1alpha1): applies to the whole cluster, every node, every pod. Suitable for platform policies (the whole cluster must be audited the same way): for example, “log every execve in every pod” or “kill any process that tries to load a kernel module”.
TracingPolicyNamespaced (namespaced, same group and version): defined inside a namespace and applied only to the pods of that namespace. Suitable for policies with per-tenant autonomy: for example, “in the prod-payments namespace, kill any outbound connect to an IP outside the corporate range”.

Both CRDs have exactly the same internal structure. The difference is one of scope. The distinction was introduced precisely to allow multi-tenancy: the central security team defines cluster-wide TracingPolicy objects and each tenant can add its own with TracingPolicyNamespaced without needing cluster-admin permissions.

Anatomy of a TracingPolicy

A policy is made up of:

Hook points: which kernel events to observe.
Arguments: which data to read when the hook fires.
Selectors: filters evaluated inside the kernel to discard irrelevant events and, optionally, execute actions when they match.

Supported hook points

The official documentation lists five families of hook points:

kprobes: hook a kernel function. Syscalls are a particular case (when syscall: true) because their ABI differs from that of internal functions. Typical examples: sys_open, sys_openat, sys_connect, tcp_connect, do_mount, commit_creds. It is the most versatile hook and the one used 80% of the time.
tracepoints: hook static tracepoints compiled into the kernel. More stable across kernel versions than kprobes (they do not depend on function names that can change). Examples: syscalls/sys_enter_openat, sched/sched_process_exec.
uprobes: hook functions in userspace libraries or binaries. They serve to observe runtime primitives such as libssl functions, libc, the Go runtime, the JVM.
USDT tracepoints (User Statically Defined Tracepoints): static tracepoints defined in userspace binaries (like those MySQL, PostgreSQL and OpenJDK expose). Useful for application observability.
lsmHooks (LSM, Linux Security Module): hooks of the LSM subsystem, where SELinux/AppArmor plug in. They allow security policies very similar to traditional MAC but programmable with eBPF. Example: file_open, inode_unlink, socket_bind.

Arguments

Every hook can read the arguments of the function it is attached to. The supported types cover the primitives (int, uint64, bool, string, char_buf) and higher abstractions (file, path, sock, linux_binprm, capability, bpf_attr, cred). The high-level types are pointers to kernel structures that Tetragon knows how to parse; instead of having to read an offset, you write type: file and Tetragon gives you the full path of the descriptor’s file.

There is an important capability detail: on kernels ≥ 5.4, Tetragon can read up to 327,360 bytes of an argument if the large buffers flag is enabled. That is the difference between being able to audit execve with all of its long argv complete versus truncating them at 256 bytes and losing context.

Selectors: filtering in the kernel

Selectors are what make Tetragon cheap. Without them, every syscall on the node would fire an event that would travel kernel → ring buffer → agent → processed → filtered → discarded. With selectors, the filtering happens inside the eBPF program itself, in the kernel, and only the events that matter reach userspace.

The available selectors include:

matchArgs: filters by an argument’s value. Operators: Equal, NotEqual, Prefix, Postfix, GreaterThan, LessThan, Mask, SPort (source port), DPort (dest port), Family (AF_INET vs AF_INET6), State (socket state).
matchPIDs: filters by PID; useful for targeted observation.
matchBinaries: filters by the binary executing the syscall (absolute path), with Operator: In, NotIn, Prefix. Essential for avoiding noise from legitimate system processes.
matchNamespaces: filters by Linux namespace (Pid, Mnt, Net, Ipc, Cgroup, User). It allows policies specific to processes in containers versus the host.
matchCapabilities: filters by the process’s effective capabilities. Block actions requiring CAP_SYS_ADMIN that run in pods that should not have them.
matchNamespaceChanges: detects namespace changes (typical of container escape).
matchCapabilityChanges: detects capability changes (privilege escalation).
matchActions: the actions executed when all the preceding matchers hit.

Actions: from a simple Post to Sigkill

When a selector matches, an action is executed. Tetragon defines several:

Post: emits an event to userspace (the observability case). It supports rateLimit to avoid flooding the agent if the condition fires a thousand times per second. The syntax accepts 5 for 5 seconds, 5m for 5 minutes, 1h for 1 hour.
Sigkill: sends SIGKILL to the offending process from inside the kernel, before the syscall completes. This is the only thing that guarantees enforcement without a race.
Override: overwrites the syscall’s return value. Useful for making the process believe the syscall failed (Override -EPERM) without killing it. A better experience for apps that can handle errors; worse for apps that assume success.
Signal: sends any arbitrary signal (not just SIGKILL).
NoPost: does not emit an event, useful when combined with another selector that does emit and you only want the action without duplicated telemetry.
FollowFD and UnfollowFD: mark a file descriptor to follow its lifecycle and enrich subsequent events with the original path. Useful for auditing “which process read this file after opening it”.
TrackSock and UntrackSock: the same for sockets.
GetUrl and DnsLookup: make HTTP requests or DNS resolutions from the kernel. Designed for integrations with external systems (security webhooks, IP reputation lookups).
NotifyEnforcer and CleanupEnforcerNotification: communication with Tetragon’s enforcement subsystem for complex actions.

Modes: detection vs enforcement

A policy can be declared in one of two explicit modes:

enforce: enforcement actions (Sigkill, Override, Signal) are active. This is production.
monitoring: enforcement actions are ignored; only Post events are emitted. This is the “let us see what would happen if this were switched on” mode, critical for testing policies without breaking applications.

Control is done with the spec.options[].name: policy-mode field and value: monitoring or enforce. It is the best practice: start in monitoring, collect events for days, tune the selectors until no false positives come out, and only then switch to enforce.

Full example: blocking writes to `/etc/passwd` in the prod namespace

A realistic policy, commented line by line:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-passwd-write
 namespace: prod
spec:
 kprobes:
 - call: "fd_install"
 syscall: false # kernel function, not a syscall
 args:
 - index: 0
 type: int  # file descriptor
 - index: 1
 type: "file" # struct file*
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values:
 - "/etc/passwd"
 matchActions:
 - action: Sigkill  # kills the process
 rateLimit: "1m" # max once per minute
 options:
 - name: policy-mode
 value: enforce  # enforcement mode active

fd_install runs every time a process obtains a new file descriptor; the second argument is the file’s file struct. Tetragon knows how to resolve it to its absolute path. The matchArgs compares that path with /etc/passwd. If it matches, Sigkill kills the process before the descriptor even becomes usable. rateLimit: 1m stops the agent from saturating if a malicious application tries it in a loop.

Common use cases

Now to real use. These are the six cases that show up in any serious Tetragon deployment in 2026.

1. Execution auditing (`execve`)

The most basic use case and, even so, the most valuable. Which binaries are running in each pod? In a container that is supposed to run only nginx, suddenly seeing an sh or a wget is almost always a red flag.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-execve
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm  # struct linux_binprm*
 selectors:
 - matchActions:
 - action: Post  # events only, no enforcement

With no filters: every execve in the cluster generates an event. With K8s metadata, the event includes pod, namespace, container, image, labels. You turn it into a stream of events towards your SIEM and set up rules: “alert if I see sh, bash, nc, curl, wget or python running in any pod of the prod-api namespace”.

An enforcement variant: instead of Post, use matchBinaries with Operator: NotIn and a whitelist, plus Sigkill if the binary is not on the list. A very rigid box, but effective in pods that are “single-binary” (like a Go microservice).

2. Sensitive file access

Detecting (or blocking) reads and writes on critical files: /etc/shadow, /etc/kubernetes/, Secret mounts, /var/run/docker.sock, /proc/*/cmdline.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: sensitive-file-access
spec:
 kprobes:
 - call: "security_file_open" # LSM-ish via kprobe
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/etc/shadow"
 - "/var/run/secrets/"
 - "/var/run/docker.sock"
 matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/kubelet" # legitimate kubelet access
 matchActions:
 - action: Post

The matchBinaries: NotIn is important: kubelet and other legitimate node agents access these paths constantly and would generate noise. We filter those out in the kernel.

In enforcement: swap Post for Override with argError: -1 (EPERM), so that the open fails but the offending process stays alive and produces the error so that tracing tools pick it up.

3. Unauthorised outbound network connections

Detecting outbound connections to destinations outside the corporate range. Useful for spotting data exfiltration or malware command-and-control.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-external-egress
 namespace: prod
spec:
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr" # destination NOT in these CIDRs
 values:
 - "10.0.0.0/8"
 - "192.168.0.0/16"
 - "172.16.0.0/12"
 matchActions:
 - action: Sigkill
 options:
 - name: policy-mode
 value: enforce

This kills any attempt at a TCP connection to an IP that is not in the corporate CIDRs, in the prod namespace. Cilium already does this with NetworkPolicy, but Tetragon has two complementary advantages:

It gives you the process that attempted the connection, not just “pod X tried to connect to Y”.
It also works for exotic protocols where NetworkPolicy is less expressive.

4. Container escape detection

Container escape is the operational nightmare: a process inside a container manages to break the isolation (via a kernel exploit, a badly set capability, a misconfigured hostPath mount) and gain access to the host. Three typical signals:

A namespace change on the process (it leaves the container’s pid namespace).
setns or unshare in non-init processes.
Access to /proc/1/root or /dev/ from a container.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-container-escape
spec:
 kprobes:
 - call: "__x64_sys_setns"
 syscall: true
 args:
 - index: 0
 type: int
 - index: 1
 type: int
 selectors:
 - matchNamespaces:
 - namespace: Pid
 operator: NotIn
 values: ["host_ns"] # only processes NOT in the host pid namespace
 matchActions:
 - action: Sigkill
 - call: "__x64_sys_unshare"
 syscall: true
 args:
 - index: 0
 type: int
 selectors:
 - matchNamespaceChanges:
 - unshare: true
 matchActions:
 - action: Post

__x64_sys_setns targeting the host namespace from a process in a container is practically always malicious (legitimate containers do not need this at runtime).

5. Cryptomining

Mining processes have fairly recognisable profiles:

Processes with names like xmrig, minerd, cgminer, or legitimate processes such as python running CPU-intensive scripts.
Outbound connections to known mining pools (a public list of IPs and domains).
Anomalous use of /dev/cpu_dma_latency to avoid throttling.

A combined policy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-cryptomining
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm
 selectors:
 - matchArgs:
 - index: 4
 operator: "Postfix"
 values:
 - "/xmrig"
 - "/minerd"
 - "/cgminer"
 matchActions:
 - action: Sigkill
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "DPort"
 values: ["3333", "5555", "7777", "14444"] # common pool ports
 matchActions:
 - action: Post  # log only

The double policy: kill binaries with the classic names (belt) and log connections to pool ports (braces), so you also get an alert when someone renames xmrig to nginx-helper or uses exotic ports.

6. Detecting rootkits and suspicious kernel modules

Modern rootkits load kernel modules to patch functions (hide processes, hide network connections, hide files). Detecting them:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: kernel-module-load
spec:
 kprobes:
 - call: "do_init_module"
 syscall: false
 args:
 - index: 0
 type: "string"
 selectors:
 - matchActions:
 - action: Post
 - call: "security_kernel_read_file"
 syscall: false
 args:
 - index: 0
 type: "file"
 - index: 1
 type: int
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values: ["READING_MODULE"]
 matchActions:
 - action: Post

In a “well configured” Kubernetes cluster no new kernel modules get loaded at runtime; any event here is highly suspicious. Combine with enforcement on machines where modules should be fixed: Sigkill for whoever tries to load one.

Bonus: detecting third-party modification of eBPF maps

As a 2025-2026 trend: loading malicious eBPF programs to hide presence. Tetragon can observe the bpf syscall:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-bpf-syscalls
spec:
 kprobes:
 - call: "__x64_sys_bpf"
 syscall: true
 args:
 - index: 0
 type: int  # bpf cmd
 - index: 1
 type: bpf_attr
 selectors:
 - matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/cilium-agent"
 - "/usr/bin/tetragon"
 - "/usr/bin/bpftool"
 matchActions:
 - action: Post

Any process that is not one of the legitimate agents loading eBPF programs: you want to know about it.

Comparison with Falco

Falco is the closest competitor: it is also runtime security for Kubernetes, also originally based on eBPF (and before that on kernel modules), also with declarative policies. Three years ago they were functionally similar. In 2026 the divergence is clear:

Dimension	Tetragon	Falco
Philosophy	Cilium-native, integrated	Standalone, generic
Filtering	In the kernel (eBPF)	Parsing in userspace
Typical overhead	<1% CPU	5-10% CPU
Enforcement	Yes, in-kernel (Sigkill, Override)	Not native (depends on plugins)
Race conditions	No (action atomic with the syscall)	Yes, in enforcement via plugins
False positive rate	Low (K8s context in the kernel)	Higher (parsing afterwards)
Detection latency	5-26 ms	~10 ms (more constant)
Ecosystem maturity	Young, growing	Mature, plenty of material
Community	Cilium / CNCF Incubating	CNCF Graduated
Integrations	Hubble native	Falcosidekick, many
CRDs per policy	TracingPolicy / Namespaced	No CRDs; rules in YAML

When to choose each one:

Tetragon if you already use Cilium, if you need enforcement in the kernel (not detection-only), if overhead matters to you (workloads with many syscalls), and if you value the Hubble integration. Container escape and cryptomining detection is where its advantage over Falco shows up most.
Falco if you want a tool independent of the CNI, if you need the catalogue of ready-made rules and the wide community, if your cluster is not Cilium, or if the integration with SIEMs and ready-made notifiers (Falcosidekick) saves you work.
Both if the organisation is large: Falco for breadth of detection, Tetragon for surgical enforcement on critical workloads. That is what you see most in companies that have spent years with Falco and add Tetragon for specific cases.

Hubble + Tetragon: unified observability

Hubble is Cilium’s traffic observability component: it shows L3-L7 flow logs with zero impact on latency. Tetragon exposes its events over gRPC with the same format and vocabulary as Hubble, which allows you to:

See them in the same UI (Hubble UI shows Tetragon events as one more “layer”).
Correlate network events (Hubble) with process events (Tetragon) on the same timeline.
Export them together to Loki/Tempo/SIEM as a single stream.

The key synergy: Hubble tells you “this pod made a TCP connection to 1.2.3.4:80”. Tetragon tells you “this pod ran curl 1.2.3.4 from a bash binary launched by pid 1234”. Together they give you the full story.

Deployment and operation

Helm

The canonical install with Helm:

helm repo add cilium https://helm.cilium.io
helm install tetragon cilium/tetragon \
 --namespace kube-system \
 --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
 --set tetragon.exportFileMaxSizeMB=50 \
 --set tetragon.exportFileRotationInterval=24h

Tetragon deploys its DaemonSet, its CRDs and a service for Hubble. By default, it exposes the events on the agent pod’s stdout (any cluster log aggregator picks them up).

The `tetra` CLI

Tetragon ships a CLI called tetra for interactive investigation:

# real-time stream of the node's events
tetra getevents -o compact --pods <pod-name>

# structured JSON events to process with jq
tetra getevents -o json --since 5m --namespace prod

# view loaded policies
tetra tracingpolicy list

It is the best tool for debugging policies in monitoring before moving them to enforce.

Exporting to a SIEM

Three usual routes:

stdout + log aggregator: the agent writes JSON to stdout, Fluent Bit/Vector picks it up and sends it to Splunk/Datadog/Elastic. Simple, works with any logging infrastructure.
gRPC streaming: for low-latency integrations. A gRPC consumer of your own or Hubble Relay.
File + rotation: for air-gapped environments or regulatory audits that require persistent logs with controlled rotation.

Performance

Published benchmarks consistently place Tetragon at <1% of the node’s CPU under real workloads, compared with 5-10% for Falco under the same workloads. The reason is the architectural separation: Tetragon filters in the kernel and only carries the events that actually matter to userspace; Falco carries every syscall to userspace and filters there. On clusters with thousands of pods making hundreds of thousands of syscalls per second, the difference shows up on the bill.

Common operational traps

Permanent `monitoring`

The biggest trap is never reaching enforce: starting well with policies in monitoring, collecting events, tuning selectors, and then never switching. The result: you have detection without prevention, exactly what Falco gave you without paying Tetragon’s complexity. If you are going to use Tetragon, plan the road to enforce for the critical policies.

Selectors that are too lax

A policy with a single matchActions: Post and no specific selectors generates events for every syscall of the chosen hook. On a serious node that means tens of thousands per second, which fill logs, saturate exporters and hide the signal in the noise. Always start with strict filters (matchBinaries, matchNamespaces, matchPIDs) and open up once you know what you are looking for.

A kernel that is too old

Tetragon needs modern eBPF features. Kernels < 5.4 do not have the large buffer support (needed for execve with complete argv). Kernels < 5.10 do not have many of the LSM hooks. Kernel 5.15+ is the recommended minimum for production and 6.1+ to have every feature.

Hooks on renamed kernel functions

kprobes are tied to kernel function names that can change between versions. A policy that uses __x64_sys_setns can fail silently on a kernel where the function is called __do_sys_setns. Solutions: use static tracepoints where they are available (more stable), or keep alternative policies with several call entries for compatibility.

`Sigkill` in critical namespaces

Applying Sigkill to processes in kube-system or cilium-system can break the cluster. Enforcement policies must explicitly exclude the platform namespaces with matchNamespaces Operator: NotIn, or limit the scope with TracingPolicyNamespaced to make sure they do not act on systems they should not.

Missing `rateLimit`

A policy without a rateLimit on Post can suffer a catastrophic fan-out if the condition is met millions of times in an instant (typical in attack loops or application bugs). The agent saturates, events are lost, logs overflow. Always put a sensible rateLimit on detection policies, especially on high-frequency hooks such as tcp_connect or execve.

What we have not covered (upcoming articles)

eBPF LSM hooks in depth: how they relate to SELinux/AppArmor and when Tetragon is the right tool versus classic MAC.
Hubble UI with a Tetragon overlay: configuring the UI to show process observability and network observability on the same timeline.
Integration with OPA/Kyverno: how Tetragon complements admission policy engines (Kyverno validates at admission; Tetragon validates at runtime).
Forensics with eBPF: combining Tetragon with tools such as Beyla or OpenTelemetry to trace the full chain of an incident from the initial connection to the final syscall.

References

Official documentation (May 2026):

Tetragon — official site — entry point.
Tetragon docs — Tracing Policy — conceptual reference.
Tetragon docs — Hook points — kprobes, tracepoints, uprobes, LSM, USDT.
Tetragon docs — Selectors — full filter reference.
Tetragon docs — Enforcement Mode.
Tetragon docs — Kubernetes Identity Aware Policies — TracingPolicyNamespaced.
Tetragon GitHub.

Comparisons and analysis:

Comparative Analysis of eBPF-Based Runtime Security Monitoring (SciTePress paper 2025) — a benchmark with independent numbers.
Best eBPF Security Solutions for Kubernetes (ARMO, 2026) — Falco vs Tetragon vs KubeArmor comparison.
Falco vs. Tetragon (Asim Mirza, Medium) — analysis with use cases.
Deep Dive into Tetragon (A Security Engineer) — a walk through the inside of the agent.
Tetragon Series, Part 2: Enforcing Sensitive File Access (Medium).

Ecosystem:

Cilium Hubble — network observability.
Falco — official site — the other big name in the field.
KubeArmor — the third option, with AppArmor + eBPF.

Cross-references:

Part 1 of the series: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack — the eBPF fundamentals we take as read here.
Kubernetes with Cilium BGP: services reachable without Ingress — the starting point of the Cilium ecosystem on this blog.

Security on lo0 — Tech Blog

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

TL;DR

You are here: the resource side

The analogy: the visa and the border checkpoint

Part 1. The split of obligations

What is new in the revision and affects the setup

Part 2. What the SDKs give, and what they do not

Protected resource metadata

Audience validation

The authentication challenge

Scopes, which are not per tool

Which revision each one announces

Part 3. Building the protected resource

Part 4. The hop from the gateway to the MCP server

What the gateway does today

Part 5. The enterprise extension, which changes the picture

And here comes the problem

What is already in the gateway, undocumented

Part 6. Per-tool authorisation

Part 7. What still has no standard

Part 8. Who covers what

Reference configuration

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

TL;DR

You are here: the cross-cutting layer

The analogy: the civil registry and building permits

Part 1. What Keycloak is today

Project status

The internal architecture in five pieces

The minimum conceptual model

Part 2. Who can authenticate against it

Part 3. Humans and machines

People

Workloads with no person behind them

Keycloak against SPIFFE

Part 4. MCP, and the two missing standards

What the specification requires

What Keycloak supports

Part 5. The confused deputy

Part 6. Two hours of task against five minutes of token

Part 7. Sizing and operating

On Kubernetes

The official sizing figures

The benchmark

Security

Part 8. The mapping to the ENS

Part 9. The configuration to leave written down

Part 10. What Keycloak does not solve

Antipatterns

Startup checklist

Closing

See also

Sources

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

TL;DR

You are here: the door nobody chose

The analogy: the switchboard that also hands out keys

The real routes

The native endpoint

The REST routes

Administration and discovery

Configuration and transport

Access control, and the correction to the previous post

allow_all_keys

require_key_mcp_access_defined

The hierarchy

The fail-open

What is done well

The bill that never shows up

How to measure it with what is already there

Tool selection

How many tools a model can take

How it fails

The optimum is not a number

`allow_all_keys`

`require_key_mcp_access_defined`

A hardened `config.yaml`