Mtls on lo0 — Tech Blog

Model chain of trust (4/4): who serves the model and which machine you trust

Sun, 26 Jul 2026 17:00:00 +0200

The three previous articles closed questions about the artefact: what contract the served model exposes, where its bytes come from and why you trust them. What remains is the one almost nobody asks until an auditor asks it: nothing proves that the process currently returning tokens on port 8000 is the inference engine and not something that slipped into the cluster and inherited its token. And a second one, worse: whether the machine it runs on is trustworthy, and against whom.

These are different problems with different projects. Workload identity is solved by SPIFFE/SPIRE, a CNCF graduated project since 2022. Environment isolation is covered by Kata Containers and Confidential Containers, the latter promoted to CNCF incubating on 22 July 2026, four days before this article was published. The first is cheap and almost always worth it. The second is expensive, more than the vendor’s press release suggests, and is only justified against a specific threat model.

TL;DR

User identity ≠ workload identity. OIDC says which person is behind a request; it does not say which process serves it. ServiceAccount tokens, static keys and gateway API keys are bearer credentials: whoever holds them, is them.
SPIFFE defines a URI identifier (spiffe://domain/path) and a verifiable document (SVID, X.509 or JWT) delivered by a Workload API that does not require the workload to hold any prior secret. SPIRE implements it with node and workload attestation; defaults: default_x509_svid_ttl 1 h, default_jwt_svid_ttl 5 min, ca_ttl 24 h.
Identity is only worth something if something enforces it: Istio/Envoy via SDS, Cilium with mutual authentication (with serious reservations) or an ext_authz with OPA in front of the gateway.
Three rungs of isolation: container with a shared kernel → Kata Containers (its own kernel and VM; v3.32.0 from June 2026) → Confidential Containers (hardware TEE, hypervisor outside the trust base, remote attestation with Trustee per RFC 9334).
The overhead of confidential mode runs from ~0 % to ~28 % depending on what you measure: GPU-only in CC with large batches sits around 4-8 %, while the full stack (a CVM with Intel TDX plus an H100 in CC), measured independently in 2026, gives +21.8 % to +27.8 % on TTFT and −17.7 % to −21.1 % on throughput. Honest heuristic: reserve 15-25 % extra capacity.
MIG and vGPU are forbidden in CC mode according to NVIDIA’s reference architecture, and every GPU on a node goes in the same mode.
Closing criterion: the order is contract → digest → signature verified at admission → workload identity → TEE.

The analogy: the biosafety laboratory

A laboratory handling pathogens solves two problems much like ours, separately.

The first is who gets in. The company badge is not enough: at the airlock there is a specific credential, short-lived and automatically renewed (if lost, it expires on its own within an hour), issued only after independently checking that the applicant is where they say they are and is who they say they are. That credential is the SVID; the prior check is attestation.

The second is which room you work in, and here there are containment levels. On the open bench the air is shared and anything aerosolised affects everyone: the normal container, with a kernel shared by all the neighbours. One rung up, the biological safety cabinet, with a physical barrier and its own airflow: Kata Containers. At the very top, the maximum containment laboratory, where maintenance staff change filters without ever seeing the sample: the TEE, where the operator maintains the machine but cannot read its memory. And before opening the airlock somebody verifies that the integrity of the suit and of the room is as expected: remote attestation with conditional secret release.

The uncomfortable lesson: the maximum containment level is a cost decision before it is one more degree of security. Nobody builds a BSL-4 to grow yeast. The useful question is who it isolates from, not how much.

Part A — Workload identity

Six services talking to each other and none of them knows who the other is

In a serious inference cluster, like the seven layers one, the gateway talks to the serving engine, the engine queries the vector database, the trace collector receives spans from everyone and the agents with MCP invoke tools that in turn call the gateway. Many east-west connections and, in most deployments, none of them authenticated.

User identity says which person is behind the request and is resolved with OIDC, as when putting Keycloak in front of MCP. Workload identity says which process issues it: there is no human on the other side, no browser, no consent, and the process is born and dies in seconds. Its three usual substitutes fail for different reasons:

Mechanism	Where it fails
Projected `ServiceAccount` token	SA granularity, not workload granularity: two pods with the same SA are indistinguishable. It does not cross the cluster boundary and does not federate. It is a bearer token on the filesystem
Static key	It does not rotate, it gets shared over Slack, it does not distinguish issuer from bearer; once leaked, it is valid until somebody notices
Gateway API key	It authenticates the client, not the process. LiteLLM’s virtual keys are for quota and budget, not for proving identity

There is no cryptographic link between credential and process: copy the credential, copy the identity. In a multi-tenant cluster like the H100 cluster one, real isolation then depends on network topology and not on identity.

SPIFFE: the standard

SPIFFE and its implementation SPIRE graduated together in the CNCF on 20 September 2022. As of July 2026 they are settled infrastructure, not a bet.

The SPIFFE ID is a URI of the form spiffe://<trust-domain>/<workload-identifier>, for example spiffe://inferencia.example/ns/inferencia/sa/vllm: a name, not a credential. The trust domain is the root of trust (organisation, environment or site), and the guidance recommends separating workloads from different sites or security environments into different domains.

The SVID is the credential that proves that ID. The X509-SVID carries the SPIFFE ID in the URI-type SAN, not in the CN, which the specification discourages as a source of identity, and it is the preferred format. The JWT-SVID exists for when there are L7 proxies terminating TLS in the middle, with the replay risk the documentation warns about:

Certificate:
Issuer: C=ES, O=SPIFFE
Validity
Not Before: Jul 26 08:00:00 2026 GMT
Not After : Jul 26 09:00:00 2026 GMT
Subject: C=ES, O=SPIRE, CN=vllm.inferencia
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
URI:spiffe://inferencia.example/ns/inferencia/sa/vllm

One hour of validity, and what authenticates is the SAN URI: the CN is decorative.

The Workload API is the elegant piece: a local UNIX socket from which the workload obtains its SVID, its private key and the trust bundle, without presenting any secret. The documentation is explicit: “the Workload API does not require that a calling workload have any knowledge of its own identity, or possess any authentication token”. That solves the cold start of all identity cryptography: the secret you would need in order to obtain the first secret.

SPIRE: how that document is issued

SPIRE has a server (the authority that signs) and one agent per node (which exposes the Workload API), and it chains two checks.

Node attestation verifies that the agent runs where it says it does. In Kubernetes the reference attestor is k8s_psat, which validates a projected token with an audience against the API server; on bare metal there are TPM-based attestors. A note of honesty: SPIRE v1.15.1, from 28 May 2026, is a security patch over an incorrect PKCS7 validation in the azure_imds attestor that allowed attested documents to be forged and a virtual machine to be impersonated.

Workload attestation answers the opening question. The k8s attestor receives the PID of the process that opens the socket, infers the pod from its cgroup membership and queries the kubelet for the metadata. From there come the selectors: namespace, service account, pod name and UID, labels, owner, node and, above all, container and image by tag or by digest. From SPIRE v1.15.0 (19 May 2026) Sigstore support stopped being experimental, enabling selectors by signature verification status, certificate subject and issuer, and transparency log. That closes the series’ circle: you can require that the engine’s SVID go only to the container whose image carries a valid cosign signature from the expected issuer, which is what was verified in article 3/4. Provenance goes from being a one-off admission check to a precondition of identity at runtime.

spire-server entry create \
 -parentID spiffe://inferencia.example/spire/agent/k8s_psat/prod/nodo-gpu-a \
 -spiffeID spiffe://inferencia.example/ns/inferencia/sa/vllm \
 -selector k8s:ns:inferencia \
 -selector k8s:sa:vllm \
 -selector k8s:container-name:vllm \
 -selector k8s:container-image:vllm/vllm-openai@sha256:aa11bb22cc33 \
 -x509SVIDTTL 3600 \
 -federatesWith spiffe://sede-b.example

The selectors are conjunctive: a pod in another namespace, with another SA or with another image does not obtain that SVID even if it shares the node.

The default TTLs have three consequences. Rotation is continuous, not a one-off event: code that loads the certificate once at startup will break exactly one hour later. Server downtime has a clock: twenty minutes are invisible, two hours bring down the cluster’s authenticated communications, and the scaling guide admits that “a single SPIRE Server instance represents a single point of failure”, with the datastore as the bottleneck and indicative figures running from two replicas of 1 CPU and 1 GB for 10 agents to eight of 16 CPU and 16 GB for 5,000 agents and 10,000 workloads. And tag-based selectors are fragile: the runtime may report one tag or another depending on the node and the moment, so the correct selector is the digest, as in article 2/4.

Federation between trust domains

Two sites with independent clusters, or an external partner exposing a reranker, should not share an authority. SPIFFE federation makes each domain publish a bundle endpoint with its public trust material and the other poll it periodically: the https_web profile authenticates it with web PKI and https_spiffe with an X509-SVID from the domain itself, enabling automatic rotation and revocation of the root. The relationship is one-directional, bundles from different domains must never be merged and refresh is by polling with a default suggestion of five minutes. In practice, site A’s gateway accepts requests from site B’s engine without sharing a CA or an identity database, and cutting the relationship means deleting a bundle, not revoking certificates.

From identity to effective authorisation

An SVID blocks nothing by itself; somebody has to compare the presented ID against a policy.

Istio + SPIRE is the best-worn integration: Istio detects a UNIX socket implementing Envoy’s SDS API and the proxy obtains its identities from there instead of from istiod, mounted with the SPIFFE CSI Driver (recommended over hostMount). Two conditions break deployments: SPIRE’s trust domain and Istio’s must match exactly, and SPIRE only issues to previously registered workloads, including Istio’s own components. With that, policy is written in terms of identity and not IP:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: vllm-solo-desde-gateway
 namespace: inferencia
spec:
 selector:
 matchLabels:
 app: vllm
 action: ALLOW
 rules:
 - from:
 - source:
 principals: ["inferencia.example/ns/inferencia/sa/gateway"]
 to:
 - operation:
 methods: ["POST"]
 paths: ["/v1/chat/completions", "/v1/completions"]

It is a layer on top of the NetworkPolicy from layered hardening: there we closed communication pairs by topology; here we also require who calls and on what path.

Cilium offers mutual authentication backed by SPIRE, tempting if you already use eBPF for the data network. It deserves criticism: it is still marked beta in the stable documentation, and The New Stack published a reasoned critique arguing that the design does not preserve mTLS properties over the life of the connection (it uses the handshake only to authenticate and discards the session keys) and that the identity model rests on eventually consistent per-node IP-identity caches, which can allow traffic that policy should deny.

The inference gateway and LiteLLM do not speak SPIFFE natively as of this article, and saying so is better than suggesting an integration that does not exist. The pattern that does work is the one the project itself documents: an Envoy in front that terminates mTLS with an X509-SVID or validates a JWT-SVID and delegates to OPA via ext_authz. The gateway carries on with its own job (quota, budget, routing), as we saw when choosing an OSS gateway and in the L7 router; workload identity is resolved one hop earlier.

AI agents and MCP: the missing piece

An agent invoking tools over MCP is a shared secret with legs: it is handed a static credential, the MCP server cannot know whether the caller is the legitimate agent or any process holding the same string, and the blast radius is the entire set of exposed tools. The threat model for agent isolation warned that a granted permission is a usable permission; with bearer credentials it is also transferable.

SPIFFE provides what is missing: the agent presents a short-lived SVID obtained through attestation and the MCP server authorises against the SPIFFE ID. In 2026 this stopped being theory: there is work at the IETF on dynamic OAuth client registration based on trusted SPIFFE issuers, and in June 2026 Google published a SPIFFE-based Agent Identity in its IAM. None of it is fully standardised.

Two honest caveats. SPIFFE answers “who”, not “why”: to know whether the action is the one the user asked for you need traceability (MCP with OTel) and runtime detection with Tetragon. And both identities must be composed: the agent proves its own with an SVID and propagates the delegating user’s in the token, authorising on the pair. Collapsing them into one is how you end up with agents able to do, in the system’s name, things no user could.

Part B — Isolation of the execution environment

### Kata Containers: the virtual machine sandbox

Kata replaces runc with a runtime that boots a lightweight virtual machine per pod, with its own kernel and a kata-agent inside; for Kubernetes it is transparent via RuntimeClass. As of July 2026 the stable branch is 3.32.x (3.32.0 is from June 2026, with Rust 1.94, Go 1.25.11, QEMU 11.0.1, guest kernel 6.18.35 and containerd 2.3), and in April 2026 the 4.0.0 preview was published, making the Rust runtime (runtime-rs) the default and leaving the Go one deprecated until 5.0.0, because of memory footprint.

For GPUs the route is VFIO passthrough. The NVIDIA GPU Operator automates it with sandboxWorkloads.enabled=true and sandboxWorkloads.mode=kata, installing the VFIO Manager, the Sandbox Device Plugin, the Confidential Computing Manager and the Kata Manager. It requires virtualisation and ACS in the BIOS, IOMMU, kata-deploy 3.29.0 or later, containerd (not CRI-O), the KubeletPodResourcesGet feature gate (on by default from Kubernetes 1.34) and removing the NVIDIA drivers from the host: the GPU is handed over whole to the VM and the guest manages it.

apiVersion: v1
kind: Pod
metadata:
 name: vllm-kata
 namespace: inferencia
spec:
 runtimeClassName: kata-qemu-nvidia-gpu
 containers:
 - name: vllm
 image: vllm/vllm-openai@sha256:aa11bb22cc33
 resources:
 limits:
 nvidia.com/pgpu: "1"

Two measurable costs. Memory: the VM reserves its own and you have to declare it as overhead in the RuntimeClass so the scheduler does not overcommit the node. And startup: according to the SESAME'24 study on serverless confidential containers, cold start goes from around 6 s with runc to around 7 s with Kata, and warm start from 1 s to 2 s. Acceptable for an engine that takes minutes to load weights; unacceptable for ephemeral functions.

Confidential Containers: the TEE

Confidential Containers (CoCo) takes the hypervisor and the platform operator out of the trust base. The CNCF promoted it to incubating on 22 July 2026, with more than 150 active contributors, 26 repositories and over 1,200 merged pull requests, and with Microsoft Azure, Intel, AMD, IBM, NVIDIA, Alibaba and Red Hat behind it. A serious project, but incubating means it is not graduated and that its integration surface changes between versions.

The components are the CoCo pods (unmodified containers run in a TEE via Kata) and Trustee: KBS (Key Broker Service, which releases secrets conditionally), AS (Attestation Service, which validates the hardware evidence) and RVPS (Reference Value Provider Service, which holds the reference values). Inside the guest, the Attestation Agent collects the evidence and the Confidential Data Hub consumes the secrets.

The flow is a clean instance of the RATS architecture from RFC 9334: the agent inside the TEE is the attester, the Attestation Service the verifier and the Confidential Data Hub the relying party. The evidence is a hardware-signed report with the boot measurements; the AS validates it against the RVPS values; and only if it matches does the KBS release the key that decrypts the image layers or the weights. Without valid attestation there is no key, and without a key there is no model: that is what stops the operator extracting it.

A mapping warning: the confidential-containers/operator repository was archived in February 2026 and deployment moved to the Helm charts and the trustee-operator. As of July 2026 the latest chart release is v0.21.0, aligned with kata-deploy 3.31.0, and Trustee v0.20.0, with TLS 1.3 and post-quantum cryptography, external plugins for the KBS and multi-GPU support via Intel Trust Authority and NVIDIA NVSwitch.

apiVersion: confidentialcontainers.org/v1alpha1
kind: TrusteeConfig
metadata:
 name: trusteeconfig
 namespace: operators
spec:
 profileType: Restrictive
 httpsSpec:
 tlsSecretName: trustee-tls-cert

The Permissive profile exists for development and should not leave it: it is the mode in which attestation does not block.

The hardware: CPU and GPU

On the CPU side the two relevant TEEs are AMD SEV-SNP and Intel TDX; NVIDIA’s reference architecture sets EPYC Milan/Genoa and Xeon Emerald/Granite Rapids as the baseline. Host support is no longer the hard part.

On the GPU side, the technical description of the design of the first confidential GPUs explains what CC mode on an H100 buys you. Memory is split into a Compute Protected Region that hardware firewalls stop both the CPU over PCIe and other GPUs over NVLink from accessing. Everything crossing the CPU-GPU boundary passes through bounce buffers encrypted with AES-GCM-256, and command buffers and CUDA kernels are encrypted and signed before crossing the bus. There is a chain of trust from GPU boot with a signed attestation report, and only NVIDIA-signed firmware runs in CC mode, validated against NRAS or locally in isolated environments. And the performance counters are disabled in hardware as a side-channel mitigation: that is the telemetry much of observability with DCGM depends on.

What it does not protect: the package’s HBM memory is not encrypted (it is considered resistant to interposers, which is a claim about physical difficulty and not a cryptographic guarantee); nothing against denial of service, because the operator who cannot read you can still switch you off; nothing against timing or access-pattern side channels; and nothing against a bug in your own code inside the enclave or prompt injection, which remains the job of guardrails.

On Blackwell, NVIDIA announces (2 July 2026) NVLink encryption that extends confidential computing to up to 8 GPUs, non-existent on Hopper and a condition for serving a large model with tensor parallelism. The reference architecture lists H100, H200, RTX Pro 6000 Blackwell Server Edition and B200 in single-GPU passthrough, and H100/H200 in PPCIe mode and B200 for multi-GPU.

The overhead numbers, with source and with judgement

Here it pays to distrust everybody, papers included. The published spread runs from 0 % to 28 % because the studies do not measure the same thing.

Source (date)	What it measures	Result	Nature
NVIDIA, Blackwell blog (Jul. 2026)	B200 with CC enabled	−1.0 % to −7.5 % throughput; per-token latency under 8 %; “up to 98 % of native performance”	Vendor claim
Corvex, HGX B200 with TDX (2026)	“Verified” deployment, encrypted NVSwitch	“Near-native performance”, with no figures of its own	Commercial claim
arXiv 2409.03992 (2024)	H100 GPU only in CC, Llama-3.1 8B/70B	−0.36 % to 6.85 % throughput; TTFT up to +19 %; overhead → 0 as the model grows	Independent, partial
arXiv 2509.18886 (Sep. 2025)	CPU TEE (TDX/SGX) and GPU TEE (H100), Llama2 7B/13B/70B	CPU TEE: under 10 % throughput and 20 % latency. GPU TEE: 4-8 %	Independent
arXiv 2607.19353 (May 2026)	Full stack: H100 in CC inside a CVM with TDX, Mistral-7B and Qwen3-30B-A3B under load	TTFT +21.8 % and +27.8 %; throughput −17.7 % and −21.1 %; closed-loop, 11.5-20.2 %	Independent, complete

The reconciliation: the penalty is in the data path, not in the GPU’s compute. NVIDIA itself quantifies the bottleneck: the effective bandwidth of the CPU-GPU interconnect in CC mode is limited by the CPU’s encryption throughput, “around 4 GB/s”. From there, three rules. The higher the compute/I-O ratio, the lower the overhead: a 70B with large batches and long sequences amortises the toll almost entirely, and a 7B with small batches and short prompts pays it in full. Adding the CPU TEE brings its own cost, and that is where most of the difference between 4-8 % and 20 % lies. And model loading and cold start are the worst cases, with tens of gigabytes crossing an encrypted bus at 4 GB/s, which makes everything discussed in from disk to HBM and in speeding up cold start more important.

The recommendation from the authors of the 2026 study, reserving between 15 % and 25 % extra capacity, is the figure I would take to a capacity planning exercise, and not the vendor’s “98 % of native performance”. Both can be true at once; only one is prudent for sizing.

When each rung is worth it

Scenario	Kata	CoCo (TEE)	Reason
Tenants that do not trust each other	Yes	Depends	Kata removes the shared kernel, a real escape vector
Unaudited third-party code or models	Yes	Optional	A shared kernel is too much surface for someone else’s code
Model intellectual property against the infrastructure operator	Not enough	Yes	The canonical case: third-party hosting, cloud, a partner providing the iron
Regulated third-party data on infrastructure you do not control	Not enough	Yes	See defence and healthcare
Sovereign on-premise, one tenant, trusted operator	Useful	Expensive over-engineering	It protects you from an adversary you do not have
Low latency with small batches	Yes	Bad idea	The worst point on the bus-encryption cost/benefit curve

The nuance that saves the most money: the two axes are independent. Kata without a TEE is cheap and delivers most of the isolation between tenants; CoCo only adds something if your threat model includes the operator. If you build a sovereign factory with your own iron and your own staff and you answer “no” to that question, the TEE is a cost with no return.

Operational traps

MIG and vGPU against confidential mode. NVIDIA’s GA 1.0.0 reference architecture is blunt: MIG and vGPU are forbidden in CC mode, and mixed-mode nodes are not supported; every GPU on a host goes in CC or none does. The forums show the typical contradiction of a moving area: in February 2026 a moderator states that MIG+CC is not supported and in April a user cites the MIG page suggesting that Hopper and Blackwell already allow it. As of this article the reference architecture overrides the product page: if your multi-tenancy rested on partitioning the GPU with MIG, enabling CC sends you back to “one whole GPU per tenant”.

Startup grows non-linearly. SESAME'24 breaks down where: SEV memory provisioning adds around one second per 2 GB of guest (SEV pins every page in advance, and an inference VM allocates a lot), the OVMF firmware about three seconds and layer decryption another three to five. Scaling from 0 to 16 instances goes from 16 s with runc to 190 s: irrelevant for a stable Deployment, a design change for aggressive autoscaling with KEDA.

Version coupling is brutal. Kata, kata-deploy, GPU Operator, containerd, QEMU with its own patches, guest kernel, NVIDIA driver and Trustee are a matrix that has to be treated as a unit: the reference architecture fixes Kubernetes 1.32 or later, Kata 3.29, GPU Operator 26.3.1 or later, containerd 2.2.2 or later and QEMU 10.1. Add it to the weight the stack’s operators already carry.

Attestation fails after a firmware update, and that is normal. The RVPS reference values describe a specific boot state. When you patch the SEV-SNP firmware, the microcode or the GPU firmware, the TCB version changes, the evidence stops matching and the KBS does not release keys, so the pods do not start. Since these updates usually respond to a security bulletin, the typical scenario is “we applied the critical patch on a Friday and on Saturday inference would not start”. The correct procedure reverses the order: update the reference values first with an overlap window, and patch the iron afterwards.

The attestation service is a single point of failure by design. If the KBS does not answer, no new confidential pod gets its key; those already started survive, but any rescheduling or scaling does not. The same dependency as the SPIRE server and the same treatment: real high availability, its own alert and rehearsed degradation.

Closing the series: the complete chain

Article 1/4 set the contract: what the endpoint exposes and what control plane governs it reproducibly. In 2/4 the subject was the provenance of the bytes: which registry they come from, by immutable digest and not by a moving tag. 3/4 added the cryptographic proof of what they are, with signatures, build attestations and an AIBOM verified at admission. This 4/4 closes with who runs them and where: an SVID that proves the serving process’s identity and an isolation rung chosen for the adversary you actually have. If one of the four fails, the other three are worth less than they look: an impeccably signed model served by an unidentified process on a machine a third party can dump is still a problem.

With a limited budget, the order follows cost per unit of risk removed. First, digests everywhere: almost free, and it removes a whole family of substitution attacks. Second, signature verified at admission, low cost and the first thing an auditor asks to see. Third, a declarative contract and control plane, which make everything else auditable. Fourth, workload identity with SPIFFE/SPIRE: expensive to operate, but it is the only thing that turns east-west traffic into something authorisable, and it becomes mandatory as soon as agents with MCP arrive. And last, the TEE, only if the infrastructure operator is in your threat model; Kata without a TEE can jump ahead to fourth place if there are tenants that do not trust each other.

Mapping the series to ENS, ISO/IEC 42001 and the EU AI Act

The detail is in technical controls ENS × ISO 42001 × EU AI Act; this table is the summary by link.

Link	ENS measure (RD 311/2022)	ISO/IEC 42001 (Annex A)	EU AI Act
1/4 Contract and control plane	`op.exp.2`; `op.mon.1`	A.6 life cycle	Art. 12 (record-keeping); Art. 13 (transparency)
2/4 Registry and distribution by digest	`op.ext.3`	A.10 third parties	Art. 11 and Annex IV (technical documentation)
3/4 Signature, attestations and AIBOM	`op.exp.6`; `op.ext.3`	A.6.2 and A.7 data and traceability	Art. 15(5), data and model poisoning
4/4-A Workload identity	`op.acc.1/2/5`; `op.exp.11`; `mp.com.2-3`	A.9 use of the AI system	Art. 15(5), unauthorised third parties
4/4-B Isolation and TEE	`mp.info.3`; `op.exp.2`; `mp.com.4`	A.6 life cycle controls	Art. 15(4) robustness; Art. 15(5)

Art. 15(5) carries the most weight: it requires high-risk systems to be “resilient against attempts by unauthorised third parties to alter their use, outputs or performance”, and it enumerates data and model poisoning and confidentiality attacks. Links 3 and 4 are its technical implementation. For the management framework, see ISO/IEC 42001 as an AIMS and the AI Act mapping.

For an inference factory

Start workload identity with the gateway↔engine pair and with the agents, not with the whole cluster. Registering all forty workloads on day one is the most reliable way to abandon the project: register two entries, set up the Istio policy that requires the gateway’s principal and live with it for a couple of weeks to learn what breaks when the certificate rotates. Then the agents with MCP, where static credentials are the entire threat model.

Separate the Kata decision from the TEE decision, and take them in that order. Kata is a RuntimeClass change with a second of startup and some memory. CoCo is a redesign: you lose MIG and the GPU’s performance counters, you gain ten seconds or more of startup, you pay 15-25 % of capacity and you add two dependencies whose failure prevents workloads from starting. Evaluate the second with a written threat model, not with a slide.

If you go to CoCo, rehearse patch day before you need it. The failure that will take your inference down will not be an attack, it will be a firmware update that misaligns the RVPS reference values. Write the runbook (update the RVPS with overlap, patch, retire the old value), rehearse it on one node and monitor the attestation success rate as a first-class metric, just as for SPIRE: alert on attestation failures before they turn into 401s at the gateway.

That closes the series: from the contract to the artefact, from the artefact to its proof, and from the proof to the process and the machine. What lies below is no longer a chain of trust but daily operation: seeing what each process does with Tetragon and measuring whether what is served is still good with evals. Trust is established once; vigilance is continuous.

Sources

CNCF, SPIFFE and SPIRE Projects Graduate from CNCF Incubator (20 sep. 2022) — https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/
SPIFFE, SPIFFE Concepts (SPIFFE ID, trust domain, SVID, Workload API) — https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/
SPIFFE, SPIFFE Federation specification — https://spiffe.io/docs/latest/spiffe-specs/spiffe_federation/
SPIFFE, Scaling SPIRE (dimensionamiento y punto único de fallo) — https://spiffe.io/docs/latest/planning/scaling_spire/
spiffe/spire, Releases (v1.15.0 de 19 may. 2026; v1.15.1 de 28 may. 2026, parche de azure_imds) — https://github.com/spiffe/spire/releases
spiffe/spire, Kubernetes Workload Attestor plugin (selectores, kubelet, cgroups, Sigstore) — https://github.com/spiffe/spire/blob/main/doc/plugin_agent_workloadattestor_k8s.md
spiffe/spire, SPIRE Server configuration reference (default_x509_svid_ttl, default_jwt_svid_ttl, ca_ttl) — https://github.com/spiffe/spire/blob/main/doc/spire_server.md
Istio, SPIRE integration (Envoy SDS, SPIFFE CSI Driver, trust domain) — https://istio.io/latest/docs/ops/integrations/spire/
SPIFFE, OPA Authorization with Envoy and JWT-SVIDs — https://spiffe.io/docs/latest/microservices/envoy-jwt-opa/readme/
The New Stack, How Cilium’s Mutual Authentication Can Compromise Security — https://thenewstack.io/how-ciliums-mutual-authentication-can-compromise-security/
Riptides, Bringing SPIFFE to OAuth for MCP — https://riptides.io/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads/
CNCF, Confidential Containers becomes a CNCF incubating project (22 jul. 2026) — https://www.cncf.io/blog/2026/07/22/confidential-containers-becomes-a-cncf-incubating-project/
Confidential Containers, Attestation with Trustee (KBS, AS, RVPS, CDH) — https://confidentialcontainers.org/docs/attestation/
Confidential Containers, Deploy Trustee in Kubernetes (11 feb. 2026) — https://confidentialcontainers.org/blog/2026/02/11/deploy-trustee-in-kubernetes/
confidential-containers/trustee, Releases (v0.20.0: TLS 1.3, PQC, multi-GPU ITA, NVSwitch) — https://github.com/confidential-containers/trustee/releases
IETF, RFC 9334 — Remote ATtestation procedureS (RATS) Architecture — https://www.rfc-editor.org/info/rfc9334/
Kata Containers, Kata Containers 4.0.0 Preview (28 abr. 2026, runtime-rs por defecto) — https://katacontainers.io/blog/release-4-0-0-preview/
kata-containers, Release 3.32.0 — https://github.com/kata-containers/kata-containers/releases/tag/3.32.0
NVIDIA, Deploy with Kata Containers — GPU Operator — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html
NVIDIA, Deploying Proprietary Models Securely with Confidential Computing on Self-Hosted Kubernetes (GA 1.0.0; MIG y vGPU prohibidos en CC) — https://docs.nvidia.com/enterprise-reference-architectures/deploying-proprietary-models-confidential-compute-self-hosted-kubernetes/latest/reference-implementations.html
CACM, Creating the First Confidential GPUs (CPR, bounce buffers, contadores deshabilitados, HBM no cifrada) — https://cacm.acm.org/practice/creating-the-first-confidential-gpus/
NVIDIA, Confidential Computing on H100 GPUs for Secure and Trustworthy AI (límite de ~4 GB/s, NRAS) — https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/
NVIDIA, Hardware-Rooted AI Security That Won’t Slow You Down (2 jul. 2026; Blackwell, NVLink cifrado hasta 8 GPU) — https://developer.nvidia.com/blog/hardware-rooted-ai-security-that-wont-slow-you-down
Corvex, Confidential Computing Meets NVIDIA HGX B200 (claim comercial) — https://www.corvex.ai/blog/confidential-computing-meets-nvidia-hgxtm-b200-secure-ai-without-the-performance-trade-off
arXiv 2409.03992, Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study — https://arxiv.org/html/2409.03992v1
arXiv 2509.18886, Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs (23 sep. 2025) — https://arxiv.org/abs/2509.18886
arXiv 2607.19353, Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX (may. 2026; heurística del 15-25 %) — https://arxiv.org/html/2607.19353
Segarra et al., Serverless Confidential Containers: Challenges and Opportunities (SESAME'24) — https://carlossegarra.com/assets/papers/sesame24-serverlesscoco.pdf
EU Artificial Intelligence Act, Article 15: Accuracy, Robustness and Cybersecurity — https://artificialintelligenceact.eu/article/15/
EU Artificial Intelligence Act, Article 12: Record-Keeping — https://artificialintelligenceact.eu/article/12/

Hardening and secrets in the sovereign LLM stack: defence in depth

Thu, 11 Jun 2026 11:00:00 +0000

Part of the operational series on squeezing a generic on-premise 4×H100 SXM 80GB LLM cluster. The sibling pieces: the document ingestion from PDF to indexed chunk that fills the vector database, the embeddings and rerankers service with TEI in production that feeds it, and, the most directly coupled to this post, the GitOps of the inference stack with Flux, because GitOps and secrets share a chicken-and-egg problem we solve here. Assembling all of this into a conversational assistant (LibreChat + LiteLLM + RAG) is covered by another post in the series, still in draft.

TL;DR

A sovereign LLM inference stack is, at minimum, six services: a gateway (LiteLLM or equivalent), an inference engine (vLLM), an embeddings/rerankers service (TEI), a vector database, a state database (conversations, users) and a front end (LibreChat). Six services are six attack surfaces, and the assistant “works” as soon as the gateway returns tokens, long before it is hardened. This post walks through defence in depth layer by layer:

Secrets: never in the clear in git. sealed-secrets (asymmetric encryption, controller in the cluster) vs SOPS/age (file encryption, simple, pure GitOps) vs External Secrets Operator + Vault (dynamic secrets, native rotation). GitOps’s chicken-and-egg problem.
Network: default-deny NetworkPolicy + explicit allow; L3/L4 with the standard NetworkPolicy, L7 and DNS-based egress with Cilium (eBPF); egress control so the data does not leave the perimeter; internal mTLS.
Pod security: runAsNonRoot, readOnlyRootFilesystem, dropped capabilities, seccomp RuntimeDefault, restricted Pod Security Standards, nothing privileged.
Supply chain: pin by digest, scan with Trivy, sign with cosign, admission control that rejects anything unsigned.
AuthN/Z: LiteLLM virtual keys, OIDC/LDAP in the front end, Kubernetes RBAC with least privilege.
Runtime: detection and, optionally, enforcement with Tetragon.
Data at rest: storage encryption, credentials for the vector database and the state database.

The thesis: hardening reduces the blast radius, it does not eliminate it. That is why you prioritise by impact, and the first job on a 4×H100 is the gateway↔vector store pair and egress-deny.

The analogy: the sovereign office

Picture an office holding sensitive documentation: the corpus that feeds the RAG, the users’ conversations, the engine’s credentials. It is not enough for the street door to lock. Real security is defence in depth: several layers, each assuming the previous one can fail.

The mapping is direct:

The building’s perimeter is egress control: making sure a leaked secret or a compromised process cannot exfiltrate the corpus to an external server. It is the most underrated layer and the first one I put in place in a sovereign deployment.
Access control at every interior door is the default-deny NetworkPolicy: the front end does not talk to the vector database directly because it has no reason to; only the strictly necessary pairs are open.
The fire partitions are pod isolation (restricted) and namespaces: if one service catches fire, the fire does not jump to the one next door.
The doorman who checks credentials is the admission control that verifies each image’s cosign signature before letting it into the cluster.
The cameras are Tetragon/Falco: they record what each process did and detect (or kill) anything anomalous.
The safe with a rotated key is the secrets: encrypted at rest, out of git, and rotated so a stolen key expires.

No layer is sufficient on its own. The building is secure because crossing them all is expensive.

Layer 1 — Secrets: GitOps’s chicken-and-egg problem

The stack is deployed by GitOps: the sibling piece on Flux reconciles the state declared in git against the cluster. That is excellent for manifests, but there is a foundational problem: the inference engine needs the Hugging Face token to download the model, the state database needs its password, the gateway needs its master key. Those secrets cannot go into git in the clear. Anyone with read access to the repo, and in an organisation with generous read that is a lot of people, plus every repo backup, plus every fork, would read them.

The chicken and the egg: GitOps wants all state to be in git, but secrets cannot be in git in the clear. The solution comes in three families:

Family A — sealed-secrets (asymmetric encryption, controller in the cluster)

Bitnami Sealed Secrets is a controller running in the cluster plus a client CLI, kubeseal. It uses asymmetric cryptography: there is a key pair. The public one is held by developers and is used to encrypt; the private one lives only in the cluster’s controller and is used to decrypt. The flow:

The developer takes a normal Secret and encrypts it with kubeseal, which obtains the controller’s public key. Result: a SealedSecret resource.
That SealedSecret is committed to git in the clear — it is encrypted, it is not readable.
The controller in the cluster detects it, decrypts it with its private key and creates the real Kubernetes Secret in the target namespace.

The key property: because the private key never leaves the cluster, neither the developer nor anyone with access to git can decrypt. And the encryption includes the namespace name: a SealedSecret sealed for inferencia cannot be moved to front and decrypted there — it behaves as if each namespace had its own key. The controller also manages rotation of the sealing keys, labelling them as active or compromised.

Advantage: it fits pure GitOps perfectly, the encrypted secret lives with the rest of the state. Limitation: the secret, once decrypted, ends up as a normal Kubernetes Secret — it sits in etcd, and etcd has to be encrypted separately (we will see that in layer 7).

Family B — SOPS + age (file encryption, simple)

SOPS encrypts the whole YAML/JSON file (or only its values) and leaves it in git encrypted, decrypting it only at deployment time. It supports cloud KMS (AWS/GCP/Azure), PGP and, the relevant one for on-premise sovereignty, age, a simple serverless offline encryption scheme. The GitOps operator (Flux brings native SOPS integration) decrypts on reconciliation using the age key stored in the cluster.

Advantage: simple, scriptable, with no server to maintain; the operational weight falls on safeguarding the age key. It is the usual recommendation for small teams or for getting started. Limitation: rotation is manual (re-encrypt everything with the new key) and there are no dynamic secrets.

Family C — External Secrets Operator + Vault (dynamic secrets, native rotation)

The External Secrets Operator (ESO) keeps no secrets in git at all. All that goes into git is an ExternalSecret: a reference saying “the password field comes from path secret/data/vectordb in such-and-such a SecretStore”. The operator reads from an external store, typically HashiCorp Vault, and synchronises the Kubernetes Secret. Vault keeps the secrets in its own encrypted backend and serves them over an authenticated API, so that they never reside in git and, with its dynamic engines, it can issue short-lived database credentials that expire on their own.

Advantage: native rotation, dynamic secrets, centralised auditing, a single point of governance. Limitation: you have to operate Vault (sealing/unsealing, high availability, access policy), which is real work.

How to choose, and rotation

Criterion	sealed-secrets	SOPS + age	ESO + Vault
Secret in git	encrypted	encrypted	reference only
Extra server	lightweight controller	none	Vault (heavy)
Rotation	automatic sealing keys	manual (re-encrypt)	native / dynamic
Dynamic (short-lived) secrets	no	no	yes
Fit with pure GitOps	excellent	excellent	good (references)
Operational curve	low	very low	high

For a sovereign 4×H100 getting started, SOPS/age or sealed-secrets cover 90% at minimal cost. When the system grows and requirements for frequent rotation and auditing appear, typical in ENS High Category scenarios, you migrate to ESO + Vault. The rotation rule: a secret that never rotates is a secret that, once leaked, is leaked forever. Rotation does not prevent the leak; it bounds its window of validity. We will come back to that when we talk about blast radius.

Layer 2 — Network: default-deny, egress and the arithmetic of the surface

The arithmetic of communication pairs

Without a network policy, in Kubernetes every pod can talk to every pod. That is the default posture, and it is the worst one for a sovereign system. With $N$ services, the number of possible ordered communication pairs (who-can-call-whom) is:

$$P_{\text{abierto}} = N \cdot (N-1) \approx N^2$$

With our $N = 8$ components (gateway, vLLM, TEI, vector database, state database, front end, plus the secrets controller and the observability one), that is $8 \cdot 7 = 56$ possible directed pairs. Fifty-six paths along which a compromised service could pivot laterally.

Now we apply a default-deny NetworkPolicy: nothing talks to anything except what is explicitly allowed. The real whitelist of an LLM stack is small. The strictly necessary pairs:

front end → gateway
gateway → vLLM
gateway → TEI
gateway → vector database
gateway → state database
front end → state database (sessions/users)

That is $E = 6$ edges. The communication surface falls from $56$ to $6$:

$$\frac{E}{P_{\text{abierto}}} = \frac{6}{56} \approx 0.107$$

Almost 89% of the possible paths are closed. The front end can no longer touch vLLM or TEI or the vector database directly; if somebody compromises the front end, they have no network route to the corpus. This is the difference between $\sim N^2$ and a whitelist $E \ll N^2$.

Standard L3/L4, L7 and egress with Cilium

Kubernetes’ native NetworkPolicy operates at L3/L4: it selects pods by label and allows/denies by port and protocol. That covers the bulk of the whitelist. But it has limits: it does not understand DNS or HTTP. This is where Cilium comes in, applying policies in the kernel via eBPF and extending them with CiliumNetworkPolicy:

Egress by FQDN/DNS: instead of pinning IPs (which change), allow egress only to huggingface.co so vLLM can download the model and block the rest. Critical for egress control.
L7: allow only certain HTTP methods/paths between the gateway and an internal service.
Enforcement modes: in the default mode, an endpoint with no policy selecting it has everything open; in always mode, everything is denied until a policy explicitly opens it. For a sovereign system, always + default-deny is the target posture.

Egress control is the sovereign piece par excellence. An internal service being unable to open connections to the internet except to a handful of explicit destinations means that, even if vLLM or the gateway is compromised, the corpus cannot leave the perimeter. The RAG is the asset of value; egress-deny is what stops it leaking out.

Internal mTLS

The NetworkPolicy says who can talk to whom, but it neither encrypts nor authenticates east-west traffic. For that, mTLS (mutual TLS: client and server authenticate each other). Cilium offers native mutual authentication; alternatives such as Linkerd or Istio give you a full service mesh. For a six-service stack, Istio’s full mesh is usually over-engineering; Cilium’s mutual authentication or Linkerd (lighter) is proportionate. The effect: an attacker on the cluster network cannot impersonate the gateway or eavesdrop on internal traffic in the clear.

Layer 3 — Pod security: restricted, and nothing privileged

The Pod Security Standards define three profiles: privileged (no restrictions), baseline (the reasonable minimum) and restricted (hardened). The target for the whole LLM stack is restricted, applied via Pod Security Admission with a namespace label in enforce mode. The restricted profile requires:

runAsNonRoot: true and a runAsUser other than 0. No container runs as root.
readOnlyRootFilesystem: true. The root filesystem is read-only; anything that needs to write goes to an emptyDir or an explicit volume. An attacker cannot leave persistent binaries in the container.
drop: ["ALL"] of Linux capabilities. No capabilities that are not needed.
seccomp RuntimeDefault, which applies the runtime’s profile and blocks dangerous syscalls.
allowPrivilegeEscalation: false, nothing privileged, no host namespaces.

The nuance with GPUs: pods using an H100 load NVIDIA’s device plugin, which historically tempted people to relax the securityContext. You do not need to run the inference container as privileged in order to use the GPU; device access is managed by the plugin, and the vLLM pod can and should run restricted. This is a fire partition: if vLLM is compromised, the attacker has a non-root process, with no capabilities, a read-only FS and trimmed syscalls — a terrible starting point for escalation.

Layer 4 — Image supply chain

The container image is third-party code running with access to your data. Three composed controls:

Pin by digest, not by tag. vllm/vllm-openai:latest is a moving target; @sha256:... is immutable. Pinning the digest guarantees that what was deployed is exactly what was audited.
Scanning with Trivy. Before promoting an image, Trivy enumerates its CVEs and composes the SBOM (component inventory). The pipeline fails if there are unmitigated critical vulnerabilities.
Signing with cosign and verification at admission. The Sigstore project (cosign for signing, Fulcio as a CA for ephemeral certificates via OIDC, Rekor as an immutable transparency log) allows the image to be signed. And Sigstore’s policy-controller, or Kyverno, is an admission controller that verifies the signature before admitting the pod: an image without a valid signature does not enter the cluster.

This is the doorman from the analogy: he checks each image’s credential (signature) at the door. A poisoned image uploaded to a registry, or a hijacked tag, stays outside because it carries no signature from the trusted issuer.

Layer 5 — AuthN/Z: virtual keys, OIDC and minimal RBAC

Three identity control points, from the outside in:

Front end (OIDC/LDAP). The human user authenticates against the corporate identity provider. The front end does not invent its own user system; it delegates to OIDC. This gives SSO, MFA and centralised revocation.
Gateway (LiteLLM virtual keys). The gateway issues virtual keys: each team, application or user has its own key with a budget, a rate limit and permitted models. The gateway’s master key, and the real API keys towards the engine, are never seen by the client; clients only handle their virtual key, revocable individually. If a virtual key leaks, that one and only that one is revoked.
Kubernetes RBAC with least privilege. Each service’s ServiceAccount has exactly the permissions it needs. The vLLM pod does not need to list Secret objects in other namespaces or create pods. Restrictive RBAC means a stolen service account token opens very little.

Layer 6 — Runtime: the cameras with Tetragon

The previous layers are preventive. What is missing is detection: what if something, despite everything, runs where it should not? Tetragon is observable security and runtime enforcement over eBPF, Kubernetes-aware. It hooks kernel events — process_exec, tcp_connect, security_file_open — with typical overhead below 1%, and it can move from observing (recording the event) to enforcement (killing the process or cutting the connection in the kernel, Sigkill).

The operational rule, which I set out in detail in the bubblewrap + Tetragon runbook: adopt first, block later. First you deploy in observation mode to build a baseline of the stack’s normal behaviour — which binaries vLLM executes, what the gateway connects to — without false positives. Only afterwards do you promote the clear rules to enforcement: kill any process that tries to read secret paths, or cut every tcp_connect to destinations outside the egress whitelist. Falco is the detection-only alternative over eBPF; Tetragon adds enforcement. These cameras also produce the audit evidence: what each service ran and which attempt was blocked.

Layer 7 — Data at rest

The last thing we protect is data standing still:

Storage encryption. The persistent volumes, where the vector database and the state database live, on top of disk encryption (LUKS/dm-crypt) or Ceph-level encryption. A disk stolen from the datacenter does not reveal the corpus.
etcd encryption. Remember that Kubernetes Secret objects, once decrypted by sealed-secrets or ESO, are normal objects in etcd. You have to enable etcd encryption at rest, or the secret sits in the clear in the control plane.
Database credentials. The vector database’s password and the state database’s are first-class secrets (layer 1), never embedded in the manifest or in environment variables in the clear in git.

Exposure table: service × port × external egress?

This table is the input for writing the NetworkPolicy resources. The egress column is the one that decides what leaves the perimeter.

Service	Internal port	Permitted callers	External egress?
front end	3080	(human ingress via OIDC)	No
gateway (LiteLLM)	4000	front end	No
vLLM	8000	gateway	Only `huggingface.co` for the initial download; zero in operation
TEI (embeddings/reranker)	8080	gateway	Only model download; zero in operation
vector database	6333	gateway	No
state database	5432	gateway, front end	No
secrets controller	—	(control plane)	No
observability	9090	internal scraping	No

The reading: no service needs external egress in normal operation. vLLM and TEI need it once to pull the model, and that permission can be temporary or restricted by FQDN to huggingface.co. Everything else is total egress-deny. If a service starts attempting outbound connections this table does not contemplate, Tetragon records it and, in enforcement, cuts it.

Blast radius: a leaked secret, with and without defence

The blast radius measures how much damage a compromise does. Let us model it for the realistic worst case: the state database’s credential leaks.

Without hardening (flat network, unrotated secret, no egress-deny, no detection):

The secret gives access to the state database from any pod (flat network → 56 open pairs).
The secret does not rotate → it is valid indefinitely; the exploitation window is $\infty$ until somebody notices.
Without egress-deny, the attacker dumps the whole database to an external server.
Without Tetragon, nobody finds out until the public incident.
Blast radius: the entire state database, exfiltrated, undetected, for an indefinite time.

With hardening (default-deny, ESO + Vault with dynamic credentials, egress-deny, Tetragon in enforcement):

The secret is only usable from the pod that has a network route to the state database (1-2 pairs, not 56).
With Vault’s dynamic credentials, the secret expires — say within a window $T$ of hours; past $T$, it is worthless.
Egress-deny prevents the dump to the internet: the attacker can read, but cannot take anything out.
Tetragon records the anomalous access and, in enforcement, kills the process attempting the outbound connection.

The qualitative reduction is enormous, but let us be quantitative about the window. If a static secret is valid forever and a rotated one with period $T$ is valid for at most $T$, and compromises arrive at rate $\lambda$, the expected number of live and exploitable secrets at any given instant goes from growing without bound to being capped at $\lambda \cdot T$. With rotation every 24 h ($T = 1$ day) against “never”, a given secret’s window falls from months to a day: a reduction of one to two orders of magnitude in temporal exposure. Combined with the reduction in network pairs ($56 \to \sim 2$, a factor of $\sim 28\times$) and egress-deny (from exfiltration possible to impossible by the direct route), the blast radius is drastically reduced.

But let us be honest: it does not reach zero. The attacker with the credential can read the state database during the window $T$ from the compromised pod. Hardening turned “indefinite, silent catastrophe” into “a bounded, detected incident with no exfiltration by the direct route”. That is exactly what defence in depth promises: not invulnerability, but that the cost of a full compromise is high and its radius small.

The ENS / NIS2 angle

These layers are not voluntary hygiene: they implement specific controls. The mapping is developed in detail in the technical controls ENS × ISO 42001 × EU AI Act post; here, the actionable summary:

Hardening layer	ENS measure (RD 311/2022)	NIS2 / framework link
Encrypted secrets + rotation	`op.exp.11` (cryptographic keys), `mp.info.3` (encryption)	Credential management; in High Cat., HSM
Default-deny NetworkPolicy + segmentation	`mp.com.1` (perimeter), `mp.com.4` (flow separation)	NIS2 art. 21: network security measures
Egress control	`mp.com.1` + `op.mon.1`	Exfiltration prevention
Internal mTLS	`mp.com.2-3` (confidentiality/integrity in transit)	Mandatory TLS
Restricted Pod Security	`op.exp.2` (hardened configuration)	Configuration hardening (CIS)
Trivy + cosign + admission	`op.exp.6` (malicious code), `op.ext.3` (supply chain)	NIS2 supply chain
OIDC + virtual keys + RBAC	`op.acc.1-2-5` (identification, access, authentication)	MFA in High Cat.
Tetragon runtime	`op.mon.1` (intrusion detection)	Monitoring and response
Encryption at rest + etcd	`mp.info.3` (encryption), `mp.si` (media)	Data at rest

The honest note for auditing: hardening reduces risk, it does not eliminate it, and maturity is demonstrated by prioritising by impact. A competent auditor does not want to see nine half-finished layers; they want to see that egress-deny on the critical asset (the corpus) and secrets management are solid before mTLS is perfect between low-sensitivity services. For the management and governance context, see also ISO/IEC 42001 as an AIMS and the EU AI Act mapping onto the architecture.

Applied to the generic 4×H100 cluster

In a real deployment on 4×H100 SXM 80GB, you do not harden the nine layers at once. The order, prioritised by impact:

Egress-deny on vLLM, TEI and the vector database, first. It is the barrier that stops the corpus leaving the perimeter, and it goes in as soon as the services start. Allow egress by FQDN to huggingface.co only during the model’s initial download; afterwards, zero. It is the layer with the highest return per hour invested.
Secrets for the inference engine and the databases, out of git. The Hugging Face token vLLM uses to download the model, the vector database’s and the state database’s passwords, the gateway’s master key: into sealed-secrets or, if there is already a rotation requirement, into ESO + Vault. Never in values.yaml in the clear.
Default-deny NetworkPolicy + the 6-edge whitelist. The gateway and the vector database are the most exposed services — the gateway because it receives all the traffic, the vector database because it holds the embedded RAG. Closing everything that is not on the whitelist cuts lateral pivoting from $\sim N^2$ to the 6 real edges.
restricted Pod Security in the inference namespace, including the vLLM pod with a GPU (it needs no privileges to use the H100).
cosign + admission so only signed images get in; Trivy in the GitOps pipeline of Flux.
Tetragon in observation, baseline, and then enforcement over the egress of the sensitive pods.
Encryption at rest for the databases’ volumes and for etcd.

The gateway and the vector database are the two I harden first inside the whitelist: the gateway for being the exposed face, the vector database for holding the asset that egress-deny protects. The rest is built on top, layer by layer, always assuming the previous one can fail.

What hardening does NOT solve

To close honestly, what these layers do not cover and what needs other pieces:

Prompt-level attacks (jailbreak, indirect injection via the RAG corpus): that is the job of guardrails and LLM Guard, not of NetworkPolicy.
Agents with legitimate permissions doing something harmful: the runtime isolation from the agent isolation post bounds what an agent can touch, but a granted permission is a usable permission.
The human factor: a well-managed secret shared over Slack is still leaked. Rotation bounds the window, it does not eliminate the mistake.
Zero-day vulnerabilities in the components themselves: Trivy detects what is known; the unknown gets through until the CVE is published.

Hardening is a multiplier on the cost of attacking, not an impassable wall. Its value lies in making an individual compromise bounded, detected and with no way out — and that, for a sovereign system holding sensitive data, is exactly the goal.

Mtls on lo0 — Tech Blog

Model chain of trust (4/4): who serves the model and which machine you trust

TL;DR

The analogy: the biosafety laboratory

Part A — Workload identity

Six services talking to each other and none of them knows who the other is

SPIFFE: the standard

SPIRE: how that document is issued

Federation between trust domains

From identity to effective authorisation

AI agents and MCP: the missing piece

Part B — Isolation of the execution environment

Confidential Containers: the TEE

The hardware: CPU and GPU

The overhead numbers, with source and with judgement

When each rung is worth it

Operational traps

Closing the series: the complete chain

Mapping the series to ENS, ISO/IEC 42001 and the EU AI Act

For an inference factory

See also

Sources

Hardening and secrets in the sovereign LLM stack: defence in depth

TL;DR

The analogy: the sovereign office

Layer 1 — Secrets: GitOps’s chicken-and-egg problem

Family A — sealed-secrets (asymmetric encryption, controller in the cluster)

Family B — SOPS + age (file encryption, simple)

Family C — External Secrets Operator + Vault (dynamic secrets, native rotation)

How to choose, and rotation

Layer 2 — Network: default-deny, egress and the arithmetic of the surface

The arithmetic of communication pairs

Standard L3/L4, L7 and egress with Cilium

Internal mTLS

Layer 3 — Pod security: restricted, and nothing privileged

Layer 4 — Image supply chain

Layer 5 — AuthN/Z: virtual keys, OIDC and minimal RBAC

Layer 6 — Runtime: the cameras with Tetragon

Layer 7 — Data at rest

Exposure table: service × port × external egress?

Blast radius: a leaked secret, with and without defence

The ENS / NIS2 angle

Applied to the generic 4×H100 cluster

What hardening does NOT solve

See also