Ebpf on lo0 — Tech Blog

How we use our infrastructure to meet defence standards with AI

Thu, 18 Jun 2026 11:00:00 +0200

In defence, the challenge of putting AI into operation is not the model: it is security and compliance. The rules that apply, the allied interoperability standards, the data-centric security rules, the accreditation frameworks and the principles of responsible use, impose concrete requirements on access control, classification, confidentiality, traceability, supply chain and sovereignty. A demo on a laptop connected to the internet ignores all of that; an operable platform has to satisfy it, and has to be able to prove it to an accreditation body.

This article is not a tour of technologies. It is the opposite: we take the requirements imposed by the standards of the sector and, for each one, we show how our infrastructure is used to meet it. The infrastructure is the same real platform we use in other sectors, two RKE2 clusters (Kubernetes v1.35) with Cilium as the network, GitOps with Flux, Keycloak as the identity server, inference on our own GPUs and, in pre-production, scheduling with Volcano, adapted to an environment with stricter security requirements. Every component is here because it solves a regulatory requirement, not because it looks good on a diagram. And, like all infrastructure, it is measured by efficiency and cost, which is what makes compliance sustainable.

It is worth starting with an idea many people are unaware of: standards for this already exist in the sector, and nothing has to be invented. There are allied interoperability standards, there is a defence-specific DevSecOps reference model on Kubernetes, there is a security framework (Zero Trust) and a supplier accreditation scheme, and there is a body of principles for the responsible use of military AI. Let us first lay out that map, and then walk the requirements one by one.

The map of standards in the sector

Four families of standards intersect in a defence AI platform, and understanding what each one covers avoids reinventing wheels.

Allied interoperability. In a coalition, systems have to understand each other. Federated Mission Networking (FMN) defines how mission networks are federated across nations through successive “spirals”; the MIP (Multilateral Interoperability Programme) and its JC3IEDM data model standardise command and control; Link-16 and formats such as NVG (NATO Vector Graphics) cover tactical and situational exchange. They are, in defence, the equivalent of what FHIR and DICOM are in healthcare: the common language without which AI is an isolated system that cannot be integrated into the operation.

Data-centric security: STANAG 4774 and 4778. This is the most distinctive piece of the sector. STANAG 4774 defines the syntax of a confidentiality label (policy, classification, security categories), and STANAG 4778 defines the binding mechanism that ties that label to the data throughout its whole life cycle and between the parties that share it. Instead of protecting only the perimeter or the system, the information object itself is protected: the classification travels with the data and the access decision is taken against the label. Both STANAGs are in the process of ratification by the nations and are moving from recommended practice to a requirement for mission-critical software.

Defence DevSecOps on Kubernetes. To build and deploy defence software there is a very concrete reference model: the DoD Enterprise DevSecOps Reference Design on Kubernetes, embodied in platforms such as Platform One and its Big Bang product line (declarative continuous delivery of hardened packages on a Kubernetes cluster), supported by Iron Bank, the repository of hardened container images compliant with the DISA STIGs, continuously scanned. The associated philosophy is cATO (Continuous Authority To Operate): accreditation stops being a one-off milestone and becomes a state maintained through continuous monitoring. For defence, it is the equivalent of the Kubernetes-native standard that MONAI represents in medical imaging: an agreed way to package and operate software with guarantees about the substrate we already use.

Security and accreditation frameworks. The DoD Zero Trust Reference Architecture (v2.0) sets the guiding principle, do not trust by default, always verify, with adoption targets towards 2027. Supplier accreditation is governed by CMMC 2.0 (whose Phase 2, with mandatory third-party certification for level 2, arrives in November 2026) over the 110 controls of NIST SP 800-171. In Spain and the EU, the Esquema Nacional de Seguridad (the Spanish National Security Framework) at high category and the CCN-STIC guides of the Centro Criptológico Nacional (the Spanish National Cryptologic Centre) play an equivalent role for systems handling sensitive or classified information.

Responsible AI in defence. NATO adopted six Principles of Responsible Use (PRU) for AI in defence: lawfulness, responsibility and accountability, explainability and traceability, reliability, governability and bias mitigation, reinforced by its AI Strategy revised in 2024 and by the DARB (Data and AI Review Board). An important legal nuance: the EU AI Act expressly excludes (art. 2.3) AI systems intended exclusively for military, defence or national security purposes. That is, in defence the responsible AI framework is not the AI Act, but NATO’s PRU, national rules and, as a management system, ISO/IEC 42001.

With this map on the table, let us walk the requirements.

Requirement: coalition interoperability (FMN, MIP/JC3IEDM)

In an allied operation, a system that does not integrate is useless however good it is. The requirement, derived from FMN, from the command and control data model (MIP/JC3IEDM) and from the tactical formats (Link-16, NVG), is that AI must consume and produce information in the common formats of the coalition, not in a dialect of its own.

How we meet it: just as in other sectors we put an interoperability layer at the boundary, here the platform integrates through the agreed gateways and formats, so that AI never talks directly to the mission systems but consumes and delivers information already normalised to the coalition standard. This keeps AI as a component that couples into the federated architecture, respecting its profiles and its data governance, instead of a silo that forces bespoke integrations at every deployment. Interoperability is also a precondition for traceability and labelling: only if the data enters and leaves in a known format can its confidentiality label be attached and preserved.

Requirement: do not trust by default, Zero Trust and verified access (DoD ZT RA, NIST 800-171)

The guiding principle of modern defence security is Zero Trust: no entity, user, service or node, is trusted because of its position in the network; every access is authenticated, authorised and continuously verified. NIST 800-171 details the access and identification controls that underpin it.

How we meet it: a single identity server, Keycloak, governs authorisation with OAuth 2.1 and OpenID Connect, issuing short-lived tokens and binding them to their recipient with audience validation (aud), so that a credential issued for one service is useless against another. Between components handling sensitive information we require mutual TLS (mTLS): it is not only the user or the service that is authenticated, but the workload itself taking part in the exchange. And the network operates in default-deny (next section), so that connectivity is not a right conferred by being inside the cluster, but an explicit permission. Workloads use service accounts with no permissions by default, and administrator access to the Kubernetes API is federated against Keycloak over OIDC with scoped roles: technical access too is authenticated, limited and audited. Zero Trust is not a product you buy; it is a property that emerges from strong identity, least privilege and continuous verification, and that is how the platform is built. Verification, moreover, does not stop at sign-in: tokens are short-lived and are renewed, policies are re-evaluated on every request, and the posture of each workload, signed image, applied security policy, determines whether it can run and communicate. A component that stops complying with its policy stops being able to operate; it is granted no grace period for having been inside.

Requirement: data-centric security and labelling (STANAG 4774/4778)

Unlike other sectors, in defence the classification travels with the data. The requirement, derived from STANAG 4774/4778, is that every information object carries its confidentiality label and that access decisions are taken against that label, not against the system hosting it.

How we meet it: authorisation decisions are not limited to roles, but incorporate attribute-based access control (ABAC) evaluating the classification label of the data against the clearance of the requester. The policy is expressed as code and evaluated at a central decision point (a policy engine of the OPA/Open Policy Agent kind) that the application layer and the gateways consult before serving a resource. The label, in accordance with STANAG 4774, travels with the object, and the binding under STANAG 4778 is preserved across the platform, in the object metadata in storage and in the annotations that travel with the request. For an AI agent, this means that its token gives it access to a classification level, and that data above that level is simply not handed to it, regardless of where it is stored.

This approach embodies the classic no-read-up rule: an entity with a given clearance does not access information above its level, and the decision is taken on the object’s label, not on assumptions made by the system. When a workflow requires moving information to a lower level, a declassification or sanitisation, it is an explicit, audited step, with its own policy, never a side effect of a query. The label is preserved in the object metadata in storage and in the annotations accompanying the request, so that the STANAG 4778 binding is not lost when crossing the components of the platform.

Requirement: segregation by mission and level (multitenancy, NIST 800-171, ENS)

Different missions, different classification levels and different compartments cannot share a common space without controls. The requirement: strong isolation, so that what belongs to one mission or level is not accessible from another.

How we meet it: each mission or compartment lives in its own Kubernetes namespace, with scoped RBAC, ResourceQuota and LimitRange, dedicated storage and databases per operator, not shared behind an application filter, and its own network segmentation. Identity is segregated by realm in Keycloak. The whole definition lives in Git and is reconciled by Flux (GitOps), so that segregation is declarative and auditable through change history. Where the classification level demands it, separation goes beyond the cluster: operation runs on physically distinct clusters or sites, and exchange between classification domains, when necessary, is done through controlled cross-domain solutions, not by opening a hole between namespaces. At the highest levels, each security domain can be an independent cluster with its own chain of trust, its own identities and its own governance, preventing a single control plane from spanning classifications that must never be mixed. The choice between a separate namespace, cluster or site is itself an accreditation decision: it is documented, justified according to the level and compartment, and reflected declaratively in the repository.

Requirement: confidentiality, encryption and exfiltration prevention (NIST 800-171, ENS, Zero Trust)

NIST 800-171 (system and communications protection families) and the ENS require confidentiality, encryption of communications and prevention of information leakage. In defence, exfiltration is not an abstract risk: it is the adversary’s objective.

How we meet it: the network is managed by Cilium (eBPF) with a default-deny model. A pod has no connectivity beyond what a policy grants it explicitly, flow by flow, which turns exfiltration into something blocked in the kernel. A component can only talk to strictly necessary destinations:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
 name: inferencia-egress-minimo
 namespace: mision-alfa
spec:
 endpointSelector:
 matchLabels:
 app: serving-inferencia
 egressDeny:
 - toEntities: [world]  # no egress outside the cluster
 egress:
 - toEndpoints:
 - matchLabels:
 app: gateway-modelos
 - toEndpoints: # internal DNS, without this it does not resolve
 - matchLabels:
 k8s:io.kubernetes.pod.namespace: kube-system
 k8s-app: kube-dns
 toPorts:
 - ports: [{ port: "53", protocol: UDP }]

Two nuances separate a correct policy from a leaky one: you have to explicitly allow DNS and filter by endpoint identity, not by CIDR, because in an identity-based network filtering by IP ranges does not behave the way intuition suggests. On top of this we add flow observability with Hubble and runtime security with Tetragon (eBPF), which watches process execution, file access and connections inside the container, making it possible to detect and block anomalous behaviour; an unexpected process in a sensitive workload is exactly the signal you want to see in real time. Encryption in transit is covered by cert-manager (TLS at the edges) and Cilium’s transparent node-to-node encryption. Added to this are the Pod Security Standards in restricted profile (no privileges, no root) and disciplined management of secrets and keys, which do not live in plain manifests.

Requirement: verifiable software supply chain (Iron Bank/STIG, SBOM, CMMC)

The software supply chain is a first-order attack vector, and the sector treats it as such: images hardened in line with the STIGs, a component inventory (SBOM) and cryptographic verification of provenance. CMMC and NIST 800-171 require it in the supplier flow; Executive Order 14028 popularised the SBOM.

How we meet it: we start from hardened base images (in the spirit of Iron Bank and the DISA STIGs), we generate an SBOM (CycloneDX/SPDX) per artefact and we sign images and SBOMs with Sigstore/Cosign, a graduated CNCF project, with keyless signing and a transparency log. In the cluster, an admission control (Kyverno/policies) rejects any image that is not signed by a trusted identity, so software whose provenance cannot be verified never runs. The rule is expressed as code and verifies the signature of every image before allowing it to run:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
 name: exigir-imagenes-firmadas
spec:
 validationFailureAction: Enforce
 rules:
 - name: verificar-firma-cosign
 match:
 any:
 - resources:
 kinds: [Pod]
 verifyImages:
 - imageReferences:
 - "registro.interno/*"
 attestors:
 - entries:
 - keys:
 publicKeys: |-
 -----BEGIN PUBLIC KEY-----
 ...trusted key...
 -----END PUBLIC KEY-----

An image without a valid signature is rejected at admission, before it runs. Since the whole deployment is GitOps, the chain, from the commit to the signed image to the policy that admits it, is traceable end to end. This satisfies the supply chain control of NIST 800-171 and fits the cATO model: verification is continuous, not a one-off stamp.

Requirement: sovereignty and disconnected or air-gapped operation (classified networks, DDIL)

Defence networks frequently operate isolated from the internet (air-gapped) or under DDIL conditions, degraded, intermittent or low-bandwidth connectivity at the tactical edge. And the data cannot leave to an external API under any circumstances. The requirement: the platform has to work without internet and without taking the data outside.

How we meet it: inference is served locally, on our own GPUs, with a high-performance serving engine (vLLM) and open-weight models that can be hosted entirely inside the perimeter. Everything the platform needs, container images, models, dependencies, is mirrored into internal registries and repositories, so that the GitOps deployment reconciles from internal sources and requires no internet access. A model gateway centralises access and reverses the burden: the default behaviour is local inference, and any egress is an exception that, besides having to be enabled explicitly, would collide with the default-deny network policies. Sovereignty does not rest on a promise, but on several mutually reinforcing controls, gateway, network policy and isolation, so that a configuration slip does not translate into a leak.

One point that cannot be overlooked in defence: no component should “phone home”. Many tools send telemetry, check licences or download updates against external servers by default, and in a classified network that is unacceptable. That is why default-deny is not only a measure against the adversary, but also against the software itself: if an image attempts an undeclared telemetry connection, the policy blocks it and Tetragon logs it. The absence of phone-home is verified, not trusted.

Requirement: resilience and operation at the tactical edge (DDIL)

Much defence operation happens under DDIL conditions, degraded, intermittent or low-bandwidth connectivity, and even in total isolation. The requirement: the platform must remain useful when the link to the rear falls, and reconcile when it returns.

How we meet it: the declarative GitOps model fits disconnected operation naturally. An edge site keeps its own copy of the desired state and reconciles against local registries and repositories, so it operates without depending on a permanent connection; when the link returns, it synchronises. Inference is served locally with models already hosted on the node, with no external calls. Traces and audit events are stored locally and forwarded (store-and-forward) when there is a link, without losing the record. The high-availability control plane and the separation into sites allow controlled degradation instead of collapsing as a block. Resilience is not an add-on: it is a property of the declarative design and of sovereign inference.

Requirement: robustness and protection of the AI system itself (PRU reliability)

NATO’s reliability principle requires AI systems to have explicit and well-tested capabilities, and to be secure against manipulation. This adds a requirement that other sectors treat with less urgency: protecting the model itself, its weights and its behaviour, as a sensitive asset.

How we meet it: model weights are treated as sensitive data, hosted inside the perimeter and with access controlled by the same identity and network policies as everything else. The model gateway is the point where input and output safeguards are applied (validation, limits, filtering) and where a model can be constrained or disconnected. The signed supply chain (SBOM and Cosign) extends to model artefacts, so that only what has verifiable provenance is executed. And traceability feeds continuous evaluation: measuring the model’s behaviour over time is what makes it possible to detect degradation or manipulation before it affects the operation. Robustness is built with the same controls, identity, network, supply chain, traces, now applied to the model as well. Added to this are safeguards against adversarial inputs, prompt injection, manipulated data, at the gateway, which validates and constrains what goes into and out of the model. In defence, assuming the adversary will try to manipulate the AI is not pessimism: it is the design’s starting assumption.

Requirement: reliable execution of mission workloads and efficiency (availability, sustainability)

Serving AI in-house is only sustainable if the GPU, a scarce and expensive resource, is used well, and certain mission workloads must run reliably and to completion. The default Kubernetes scheduler schedules pod by pod, which for distributed workloads produces half-started jobs that hold GPUs without making progress.

How we meet it: GPU allocation is governed by Volcano (a CNCF incubating project), today in pre-production, with gang scheduling (PodGroup with minAvailable: the job starts complete or does not start), per-mission queues with quota and priority, fair-share (DRF) to share out fairly, pre-emption so that an urgent workload can overtake a deferrable one, and topology awareness (NVLink/PCIe) to reduce fragmentation. This raises the effective utilisation of the GPU fleet, which is what makes sovereign inference viable. On the state of the art: Kubernetes 1.34 took DRA (Dynamic Resource Allocation) to general availability, accelerator allocation by attributes instead of by count, and Kueue covers job quota management; both combine with a gang scheduler. It is managed with metrics: effective versus allocated utilisation, fragmentation, throughput per GPU and GPU-hours per mission. At the edge, where the GPU is even scarcer, these same techniques, together with quantised and smaller models, are what make it possible to serve useful inference with the hardware available instead of depending on a datacenter that may not be reachable.

Requirement: explainability, traceability and governability (NATO PRU, NIST AU)

Three of NATO’s six Principles of Responsible Use bear directly on infrastructure: explainability and traceability, governability and responsibility/accountability. Plus the audit (AU) controls of NIST. The requirement: being able to reconstruct every AI decision, keep human control and be accountable.

How we meet it: every inference is recorded as an auditable trace with Langfuse, input, output, model and version, cost, latency and mission, stored in our own components locked behind network policies that prevent it from leaving. That trace provides the traceability required by the PRU and by the NIST AU controls, and is the basis for human oversight, which is exercised at the gateway (review, limits and, crucially, the possibility of intervention and disconnection demanded by the governability principle: an AI system in defence must be able to be deactivated or constrained by the operator). An audit log is only worth anything as evidence if it is intact and ordered in time: that is why events are kept in append-only storage and clocks are kept synchronised by NTP. Instrumentation converges on the OpenTelemetry GenAI semantic conventions.

Accountability also requires being able to attribute a decision: which model, which version, which input and which operator were involved. That is why each trace links the inference to the identity that requested it and to the exact version of the model artefact that answered, so that the chain “who asked what, with which system and with what result” can be reconstructed. Without that attribution, NATO’s responsibility and accountability principle remains a statement of intent.

Requirement: continuous accreditation and reproducible operation (cATO, CMMC, ENS)

The sector has replaced one-off accreditation with cATO: an authorisation that is maintained as long as it is continuously demonstrated that the controls remain in place. The ENS and CMMC likewise require configuration management and traceability of changes.

How we meet it: the entire platform state, deployments, network policies, quotas, RBAC, identity, admission policies, is described in Git and applied by Flux over GitOps. Every change is a revision with an author, a date and a reviewer; every environment is reproducible from its repository. Security policies are code (Kyverno/OPA) evaluated continuously, not a list someone reviews once a year, and compliance monitoring is permanent. This is what cATO looks like in practice: an audit stops being a snapshot and becomes a query against a state that is maintained and verified without interruption, with the ability to reconstruct exactly what was deployed on a given date. A practical benefit is that compliance evidence generates itself: the declarative state in Git, the admission records, the image signatures and the traces are, together, the body of evidence an accreditation needs, produced continuously instead of gathered by hand the week before the inspection.

The piece that ties it all together: ISO/IEC 42001 and NATO’s principles

Everything above consists of technical controls that satisfy concrete requirements. But a management system is needed to order, evaluate and improve them, with defined roles and responsibilities. In defence, that governance has two complementary references: NATO’s Principles of Responsible Use, which set the ethical and operational “what”, and ISO/IEC 42001, the certifiable standard for AI management systems, which supplies the “how” of continuous management. Given that the AI Act excludes defence, this combination, PRU plus ISO 42001, is the governance backbone for responsible, demonstrable military AI. It is, precisely, the content of the AI Management Systems course we will publish shortly.

A view of the sector as of June 2026

It is worth pinning down the snapshot of the moment, because several currents have converged and they condition any platform decision.

An “AI-first” agenda and AI on classified networks. In January 2026, the US Department of Defense set an AI agenda at “war speed”, and through the spring agreements were closed to deploy generative AI models on classified networks. The signal is clear: military AI is moving from pilot to operation, which raises the demands on the infrastructure that supports it.

The substrate is hardened Kubernetes. Platform One, Big Bang and Iron Bank consolidate the pattern: Kubernetes clusters with STIG-compliant images, a signed supply chain and cATO. Defence AI is deployed on that same cloud-native substrate, not on bespoke systems.

Zero Trust with a deadline. Zero Trust adoption targets sit around 2027, which turns strong identity, microsegmentation and continuous verification into requirements with a calendar, not aspirations.

Supplier accreditation is tightening. CMMC Phase 2, in November 2026, ends self-attestation for anyone handling controlled information: third-party certification against the 110 NIST 800-171 controls will be required. The software supply chain, SBOM and Sigstore signing, moves from good practice to a contractual condition.

Data-centric security, operational at last. With STANAG 4774/4778 advancing through ratification, confidentiality labelling bound to the data becomes a real requirement for mission software, and not just a recommendation.

Defence-specific AI governance. NATO’s Principles of Responsible Use and the DARB are maturing as the reference framework, occupying the space the AI Act leaves free by excluding defence. And GPU economics, together with the maturity of open-weight models, pushes compute towards in-house infrastructure, precisely where classified data and disconnected operation demand it.

Europe pushes its own autonomy. In parallel, the EU is strengthening its defence industrial and technological base, through the European Defence Fund and the European Defence Agency, with growing emphasis on technological sovereignty. For a European platform, this reinforces the bet on in-house infrastructure, locally hostable models and minimal dependence on external services: exactly the direction it is built in.

Where we stand in that picture: with the platform on Kubernetes, eBPF as the network and security substrate, a signed supply chain, GPU scheduling maturing in pre-production, sovereign and disconnectable inference by design, traceability in auditable format and the management framework (PRU plus ISO 42001) as the next step. We are not chasing fashion; we are on the line the sector has decided to advance along.

Summary: requirement by requirement

Each rule or standard in the sector imposes a requirement, and each requirement is satisfied by a concrete piece of the infrastructure. Zero Trust and verified access, with Keycloak, mTLS and default-deny. The data-centric security of STANAG 4774/4778, with ABAC over classification labels. Segregation by mission and level, with multitenancy and, where appropriate, physical separation and cross-domain. Confidentiality and leak prevention, with Cilium in default-deny and Tetragon. A verifiable supply chain, with STIG-hardened images, SBOM and Sigstore signing plus admission control. Sovereignty and disconnected operation, with local inference over open models and internal registries. Reliable execution and efficiency, with Volcano. The explainability, traceability and governability of NATO’s PRU, with auditable traces and human control at the gateway. Continuous accreditation (cATO) and reproducible operation, with GitOps and policy as code. And the governance that ties it all together, with NATO’s PRU and ISO 42001.

It is not a PowerPoint: it is infrastructure that operates, that complies and that, being infrastructure, is measured by efficiency and cost. That, the cost control of an AI platform, is what the next article is about.

Sources

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

Sun, 14 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article of the energy track (C3). C1 made the conceptual inventory of the tools; here we get practical: how the measurement stack is deployed on Kubernetes, where each piece takes its figures from, how accurate it is against a power meter, how much overhead it adds, and how energy is attributed per pod and per MIG instance. It is the equivalent of B2/B3 for energy: to stop talking about what each tool measures and start building the bench that measures it. With commands; no recommendations, only the mechanics.

The measurement stack on Kubernetes

To measure energy per workload on a GPU cluster, the practical stack has four pieces, and the good news is that three of them you already have if you do observability:

The new piece compared with a normal observability stack is Kepler, which adds energy attribution per pod. DCGM already gives GPU power; Prometheus stores it; Grafana draws it. It is reused infrastructure, not a new system, in line with GPU observability via DCGM.

DCGM exporter: the base of GPU power

GPU power, the dominant source in LLM inference, comes from NVIDIA’s dcgm-exporter (part of the GPU Operator), which exposes DCGM metrics to Prometheus. The key metric is DCGM_FI_DEV_POWER_USAGE (instantaneous power in W), alongside utilisation (DCGM_FI_DEV_GPU_UTIL) and memory. With that, energy is the integral of power over time across the workload’s window. DCGM gives power at card level; to split it per pod or per MIG instance you need a layer on top, which is Kepler.

Kepler in depth: eBPF and the sources

Kepler (Kubernetes-based Efficient Power Level Exporter) is a Prometheus exporter that measures energy at container, pod and node level, using eBPF to keep overhead low and scientific methods to improve accuracy (Kepler · GitHub). Its strength is that it combines several power sources:

Source	What it covers
RAPL	CPU and DRAM
NVML	NVIDIA GPU power
ACPI	platform power management
Redfish / IPMI	platform power (BMC)
Regression models	estimation when there is no real metric

For the GPU, the dcgmEndpoint parameter points at the dcgm-exporter metrics URL, which enables power attribution per MIG instance (Kepler · configuration). That is: DCGM gives the card’s power, Kepler splits it between the pods (and MIG partitions) that use it, via eBPF and models. The result is per-pod energy metrics in Prometheus, ready for Grafana, usable for scheduling or scaling, consumption reporting and visualisation (Kepler · GitHub).

Deployment: a DaemonSet on top of Prometheus

The deployment is a DaemonSet (one Kepler per node), pointing at the existing Prometheus and at the dcgm-exporter:

# Helm, pointing at the dcgm-exporter for GPU attribution
helm install kepler kepler/kepler \
 --namespace kepler --create-namespace \
 --set dcgmEndpoint="http://dcgm-exporter:9400/metrics"

Prerequisites: a kernel with eBPF, the dcgm-exporter running (from the GPU Operator) and Prometheus scraping. Once deployed, Kepler exposes metrics such as kepler_container_joules_total per pod, queried with PromQL and drawn in Grafana. Energy per workload stops being a manual estimate and becomes a continuous time series, just like cost with OpenCost.

PromQL queries: energy per pod

With Kepler exporting, these are the queries that build the panels. The exact names vary by version; check your /metrics:

# Cumulative energy per pod (joules)
kepler_container_joules_total

# Mean power per namespace over the last hour (W)
sum by (namespace) (rate(kepler_container_joules_total[1h]))

# GPU power per node (from DCGM, W)
sum by (node) (DCGM_FI_DEV_POWER_USAGE)

# Energy per token: power ÷ throughput (with a gateway/engine metric)
sum(rate(kepler_container_joules_total[5m])) / sum(rate(vllm_generation_tokens_total[5m]))

The last one closes the circle with the cost track: it crosses Kepler’s energy with the engine/gateway tokens to give J/token in real time, per workload. With that in Grafana you have energy per pod, per namespace and per token, continuously, the energy equivalent of the OpenCost cost report.

A calibration procedure

Since Kepler estimates, it is worth calibrating once per node type. A simple procedure:

Power meter at the node input (or a PDU/BMC reading), measuring the node’s real power.
Controlled load: a stable sweep at a known power (the engine at 80 % GPU, say).
Compare: the node’s sum of kepler_container_joules_total plus idle against the meter reading over the same window.
Adjust Kepler’s factor or model if the deviation is noticeable, and document the residual error.
Repeat for each node type (calibration is not transferable between different hardware).

With that residual error documented, Kepler’s continuous figure has a known error bar, which is exactly what a CSRD auditor or a technical committee will ask for. Measuring without calibrating gives a number with no credibility; calibrating once turns the continuous estimate into a defensible figure.

Accuracy: estimation vs power meter

The data honesty of the energy track: Kepler estimates (eBPF + models + counters), it does not measure with a power meter. Accuracy depends on the source: with NVML/DCGM and RAPL available, the figure is fairly faithful (it reads hardware counters); when it falls back on regression models (with no real metric), it is an estimate with more error. Good practice: calibrate against a physical power meter at least once per node type, adjust, and use Kepler as a continuous proxy. The meter is the point-in-time ground truth; Kepler, the continuous series calibrated against it. For a CSRD report that an auditor may challenge, that initial calibration is what gives the continuous figure credibility.

Overhead: the cost of measuring

Measuring energy has its own compute cost, worth budgeting for. In a Kubernetes environment, the overhead of the monitoring system is split between the Prometheus server and the targets: the Prometheus server consumes ~150 % of CPU (1.5 cores) and ~15 GB of RAM, while the targets (node exporter and dcgm-exporter) together use ~35 % of CPU and ~300 MB of RAM (arXiv · Container-level Energy Observability). Kepler uses eBPF precisely to keep its own overhead low. The reading: the bulk of the cost of measuring is in Prometheus (which you probably already have for observability), not in the energy exporters. Adding Kepler to an existing observability stack is marginal; standing up Prometheus just for this would be expensive. That is why measuring energy is nearly free if you already observe, and a project if you do not.

Zeus: for the bench and for optimisation, not for continuous production

Zeus (ml.energy) plays a different role: it is not a continuous production exporter, it is the bench tool for measuring and optimising the energy of one specific workload (a benchmark, a training run), with power capping and frequency selection. It fits into the benchmarking flow (alongside GuideLLM/AIPerf): during a sweep, Zeus measures the energy per configuration and lets you test the effect of power capping on performance. The practical division: Kepler/DCGM in production (continuous per-pod attribution), Zeus on the bench (fine measurement + optimisation of one workload), CodeCarbon if you want carbon context integrated into an ML pipeline.

CodeCarbon and Scaphandre in practice

The other two pieces, with their specific role:

CodeCarbon integrates inside the code of an ML pipeline (a decorator or a context manager wrapping the training or the inference), and returns the energy and the CO₂eq of that block, using nvidia-smi for the GPU and estimating CPU/RAM. It fits when you want the carbon figure per experiment or per ML job, integrated into the data flow, not at infrastructure level. It is the closest to the power meter among the Python tools.
Scaphandre is a metrology agent in Rust that reads RAPL and gives process/VM power with raw hardware accuracy. For CPU it is very accurate; for GPU it has to be combined with NVML. It fits when you need the most faithful CPU metrology, or a lightweight agent independent of Kubernetes.

The division with Kepler: Kepler attributes energy per pod in the cluster (the infrastructure view); CodeCarbon gives carbon per ML job (the pipeline view); Scaphandre gives raw CPU metrology. They do not compete; they cover different layers, and a mature platform uses whichever matches the question.

Choosing the stack by objective

Objective	Tool	Why
Continuous per-pod attribution (production)	Kepler + DCGM	eBPF, per pod/MIG, into Prometheus
Fine measurement + optimisation (bench)	Zeus	measures and optimises (power capping)
Carbon in an ML pipeline	CodeCarbon	integrated CO₂eq, close to the power meter
Raw CPU accuracy (RAPL)	Scaphandre	hardware metrology
Calibration / ground truth	power meter	the standard everything is calibrated against

It is not “one tool”, it is a stack by objective: continuous production (Kepler/DCGM), bench (Zeus), reporting (CodeCarbon), calibration (power meter). Mixing them according to the phase is what gives a complete and credible energy measurement.

From figure to action

Measuring is the start; the value is in acting. With energy per pod in Prometheus:

Reporting (CSRD): the carbon footprint per workload, calibrated, ready for regulatory reporting.
Energy/carbon-aware scheduling: move deferrable load to hours or nodes of lower intensity (connects with the hourly carbon of the Spanish grid).
Power capping: with Zeus, trim peak power with little performance loss, lowering the W in the numerator of J/token.
Optimisation: identify pods with poor energy efficiency (a lot of energy per token) and attack them with quantisation or batching.

The measurement stack is not a panel to look at: it is the input to scheduling, optimisation and reporting decisions. Just as OpenCost closes the cost loop, Kepler+DCGM closes the energy one.

From energy to carbon: the final layer

The energy figure (kWh) becomes carbon by multiplying by the grid intensity, and here the practical stack connects with the hourly source. In Spain, the intensity is published by Red Eléctrica (esios) and aggregated by ElectricityMaps, hour by hour. The pattern:

# Carbon per workload: energy (kWh) × grid intensity (gCO2/kWh)
(sum by (namespace) (increase(kepler_container_joules_total[1h])) / 3.6e6)
 * on() group_left() grid_carbon_intensity_gco2_kwh

where grid_carbon_intensity_gco2_kwh is a metric you feed from esios/ElectricityMaps (an exporter or a periodic push). With that, the Grafana panel shows gCO₂ per namespace and per token, hour by hour, not an annual average. And since Spanish intensity varies a lot (from ~80 in solar hours to ~250 on gas-heavy nights), that hourly resolution is what enables carbon-aware scheduling: moving the deferrable work to the clean hours lowers the measured carbon. The measurement stack does not only report carbon; with the hourly figure, it allows it to be reduced.

Energy efficiency KPIs to watch

KPI	What it indicates	Where from
J/token per model	energy efficiency	Kepler ÷ engine tokens
Mean power per pod	consumption per workload	`rate(kepler_container_joules_total)`
gCO₂/token (hourly)	real footprint	energy × grid intensity
Idle energy	waste	power of pods with no traffic
Effective PUE	DC overhead	total energy ÷ compute energy

These five turn the energy panel into a decision tool: which model is more efficient, which pod wastes, when it pays to schedule. Without them the stack is a pretty panel; with them, it is the input to optimisation and reporting.

Example: energy attribution on a multi-tenant cluster

With Kepler splitting per pod, an energy report per team on the example node (8×H100, 5.6 kW of boards, ~7.84 kW with PUE), over one month:

Team	Energy/month (kWh)	gCO₂ (ES grid ~160)	Mean J/token
A · chat-prod	~3,100	~496 kg	~2.2 J/tok
B · batch	~1,400	~224 kg	~2.8 J/tok
C · experimentation	~900	~144 kg	~6.5 J/tok
idle	~250	~40 kg	—

What it reveals: A is efficient (FP8, high utilisation → 2.2 J/token); C is inefficient (6.5 J/token, underused GPU), just as it dragged the idle cost in the FinOps track. The carbon column takes the figure into the per-team CSRD report. And the idle row (~250 kWh, ~40 kg CO₂) is pure wasted energy. Without Kepler, that split does not exist: you would only see the node’s total consumption, with no idea whose each watt is. With it, energy is attributable and actionable, exactly like cost with OpenCost. In fact, the two panels together (OpenCost cost + Kepler energy) give the complete picture of efficiency per team.

Energy and carbon-aware scheduling

The highest-return action the stack enables: scheduling by energy and carbon. With energy per pod and hourly grid intensity in Prometheus, two policies:

By carbon: move the deferrable load (training, ingestion, overnight batch) to the hours of lowest grid intensity. In Spain, the hours of peak solar (~80 gCO₂/kWh) against gas-heavy nights (~250). The same workload emits ~3× less when scheduled at the right hour.
By efficiency: consolidate workloads to eliminate idle (a powered GPU with no work is wasted energy), and direct each workload to the hardware most efficient for it.

Both use the same data (energy per pod + hourly intensity) that the stack already exposes, and they connect with cluster scheduling (Kueue/Volcano). It is the difference between measuring carbon (and reporting it) and reducing it (by scheduling). The C3 stack enables both.

Deployment checklist

Step	Action	Verification
1	dcgm-exporter (GPU Operator)	`DCGM_FI_DEV_POWER_USAGE` in Prometheus
2	Kepler DaemonSet, kernel with eBPF	`kepler_container_joules_total` per pod
3	`dcgmEndpoint` pointing at the exporter	GPU/MIG attribution works
4	Grid intensity metric (esios)	`grid_carbon_intensity` in Prometheus
5	Calibration vs power meter (once per node type)	error bar documented
6	Grafana panels (J/token, carbon, idle)	KPIs visible

Step 5 (calibration) is the one most often skipped and the one that gives the figure credibility; step 3 (dcgmEndpoint) is the one most often forgotten, and without it the GPU is not attributed.

Energy by phase: prefill vs decode

A level of detail the stack allows and that guides optimisation: power is not constant during a request. Prefill (processing the prompt) is compute-bound and draws a lot of power in a burst; decode (generating tokens) is memory-bound and draws a lower peak but for longer. With fine DCGM sampling that bimodal profile is visible, and it has consequences:

A workload with long prompts (a lot of prefill) has a different power profile from one with long generation (a lot of decode), even if they produce the same tokens.
Power capping affects each phase differently: trimming peak power mostly touches the prefill.

For most cases the mean J/token is enough; to optimise energy efficiency thoroughly, separating prefill and decode consumption (aligning the DCGM window with each phase) reveals where the spend is. It is the energy equivalent of separating TTFT and TPOT in benchmarking, the same principle, measuring the two phases separately because they behave differently.

State of the art: the LLM resolution gap

An honest note on the limits of current tooling. The general-purpose tools (Kepler, CodeCarbon, Scaphandre) measure well at machine, process and pod level, but they lack LLM-specific resolution: out of the box they do not distinguish prefill from decode energy, nor energy per token by phase, nor the energy cost of a reasoning block against a normal response. It is an open gap in the field: per-pod attribution exists, but attribution per token and per inference phase has to be built by crossing power (DCGM/Kepler) with engine metrics (vLLM exposes tokens per phase). The 2026 state of the art is a mature stack for infrastructure and an emerging one for LLM-specific resolution, exactly the crossing that the reproducible harness (S4) has to build by hand. Acknowledging it avoids promising a per-token accuracy that the general-purpose tools do not deliver on their own.

Limits and traps (data-driven)

Uncalibrated estimation. Kepler estimates; without calibrating against a power meter at least once, the figure can carry a noticeable error, especially when it falls back on regression models.
Forgetting the dcgmEndpoint. Without pointing Kepler at the dcgm-exporter, GPU (and MIG) attribution does not work; it is the key parameter for GPU.
Accounting for overhead badly. The bulk of the cost of measuring is in Prometheus; if you stand it up just for energy, budget for it (1.5 cores, 15 GB).
Confusing bench and production. Zeus measures one workload finely; Kepler attributes continuously in production. Using one for the other gives the wrong figure.
Misaligned windows. For energy per token, the power window and the token window must coincide; if they do not, the J/token does not correspond.

With the measurement stack in place, the energy track has its instrumentation; the following articles (C6 carbon, C8 TCO and regulation) build on this data. The foundation is this: a continuous per-pod measurement, calibrated, on top of the observability infrastructure you already have.

Closing

Measuring the energy of an AI cluster in production does not require a new system: it requires adding Kepler, per-pod attribution via eBPF, to the DCGM, Prometheus and Grafana stack you already have for observability. That is the central idea of C3: the cost of measuring is almost entirely in the Prometheus you already pay for, and Kepler turns it from “card power” into “energy per pod, per team and per token”. With two cautions that separate a credible figure from a decorative panel: calibrate once against a power meter (to have an error bar that survives an audit) and point the dcgmEndpoint (without which the GPU is not attributed). Done that way, energy stops being the soft axis of sustainability and becomes an actionable time series: the J/token per model that exposes inefficiency, the hourly carbon that enables clean scheduling, the idle that gets recovered, and the CSRD report that holds up. For the sovereign platform, the Kepler+DCGM stack closes the energy loop just as OpenCost closes the cost one, and together, with GuideLLM’s goodput, they give the three faces of the dashboard measured on the same iron, in euros, joules and grams of CO₂.

Sources

Kepler · GitHub (CNCF, eBPF, por pod/nodo) — https://github.com/sustainable-computing-io/kepler
Kepler · configuración (fuentes, dcgmEndpoint para MIG) — https://github.com/sustainable-computing-io/kepler/blob/main/docs/user/configuration.md
CNCF · explorando Kepler (consumo de aplicaciones cloud) — https://www.cncf.io/blog/2023/10/11/exploring-keplers-potentials-unveiling-cloud-application-power-consumption/
Red Hat · Kepler: monitorización eficiente de energía para Kubernetes — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
arXiv · Container-level Energy Observability in Kubernetes (overhead) — https://arxiv.org/pdf/2504.10702
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Zeus Project (ml.energy) — https://ml.energy/zeus/

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

Tue, 09 Jun 2026 17:00:00 +0200

Operational companion to The contractor with the master key. That post explains the why and the where: the threat model, the five isolation families, which domain uses each. This one is the how, with commands. If you have not read it, read it first: here I take for granted what the blast radius is, why bwrap runs without root and what Tetragon watches. The procedure comes in two independent tracks, client and cluster, because, as the sibling post argues, the control extrapolates but the primitive gets rewritten.

TL;DR

Two reproducible procedures. Client (workstation): install ai-jail (it wraps bubblewrap), generate the per-project .ai-jail, audit with --dry-run, fix the allowlists with --bootstrap, use --lockdown for anything you do not trust, and leave the agent without git push permission. Cluster (RKE2 with Cilium + Tetragon): put the pod baseline in place (unprivileged securityContext, seccomp: RuntimeDefault, default-deny NetworkPolicy), put the untrusted agent’s pod in a microVM with runtimeClassName: kata, and deploy the Tetragon TracingPolicy files in two phases: observe with action: Post to raise the baseline, then promote to action: Sigkill on tcp_connect (egress) and security_file_open (secret paths). The golden rule of the Tetragon phase: adopt first, block later; never put a Sigkill into production without having seen the events in observation mode first.

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

ai-jail wraps the sandbox; on Linux it needs bubblewrap separately, on macOS it needs no extra dependency.

# ai-jail (macOS and Linux)
brew tap akitaonrails/tap && brew install ai-jail
# or, with cargo:
cargo install ai-jail
# or, with mise:
mise use -g ubi:akitaonrails/ai-jail

# bubblewrap on Linux (pick your distro)
sudo pacman -S bubblewrap # Arch
sudo apt install bubblewrap # Debian / Ubuntu
sudo dnf install bubblewrap # Fedora

Check the binary is there and that bwrap runs without root (it must not ask for sudo):

ai-jail --version
bwrap --ro-bind / / --unshare-all echo "bwrap ok without root"

If bwrap fails asking for privileges, your kernel has unprivileged user namespaces disabled; enable them (sysctl kernel.unprivileged_userns_clone=1 on older Debian/Ubuntu) before continuing.

A1 — The per-project .ai-jail file

On the first launch inside the project, ai-jail creates a .ai-jail (TOML) that is committable to the repo: any colleague who clones inherits the same policy.

cd ~/Projects/mi-app
ai-jail claude # creates .ai-jail and launches Claude Code inside the sandbox

The generated file:

# .ai-jail — sandbox configuration (commit it to the repo)
command = ["claude"]
rw_maps = [] # extra writable directories
ro_maps = [] # extra read-only directories

Before trusting the sandbox, audit it. --dry-run --verbose prints every mount point, every isolation flag and the full bwrap command, without executing anything:

ai-jail --dry-run --verbose claude

Read the output and confirm three things: that $HOME is mounted as a tmpfs (not the real one), that ~/.ssh, ~/.aws and ~/.gnupg do not appear among the mounts, and that the only writable directory is the project one. If you need an extra directory:

ai-jail --rw-map ~/Projects/shared-lib claude # extra, writable
ai-jail --map /opt/datasets claude # extra, read-only

Other agents, same binary:

ai-jail codex
ai-jail opencode
ai-jail bash # bare shell to debug the sandbox
ai-jail -- python script.py # any command

A2 — Permission allowlists with –bootstrap

--bootstrap generates each agent’s permission configuration, with sensible allow/deny/ask, and takes a backup before overwriting:

ai-jail --bootstrap

What it produces, in summary:

Agent	File	Base policy
Claude Code	`~/.claude/settings.json`	allow: `git status/diff/log`, `ls`, `grep`, `cargo`, `npm`, `python`, `docker compose` · ask: `git push`, `rm`, `docker run` · deny: `rm -rf`, `sudo`, `chmod 777`, `git push --force`
Codex	`~/.codex/config.toml`	`approval_policy = "on-request"`
OpenCode	`~/.config/opencode/opencode.json`	permissions for `bash`, `edit`, `write`

The operational key: git push is under ask, not allow, and git push --force under deny. The agent can commit, branch and rebase locally as much as it wants; none of that touches the remote. (If you use the Claude Code /sandbox, also set "allowUnsandboxedCommands": false to close the dangerouslyDisableSandbox escape hatch, which is opt-out out of the box.)

A3 — Lockdown for anything you do not trust

To audit third-party code or run an agent over a project you do not know, --lockdown goes further: project mounted read-only, GPU/Docker/display disabled, --rw-map/--map ignored, $HOME a pure tmpfs with no host dotfiles, network cut with --unshare-net and the environment wiped with --clearenv.

ai-jail --lockdown bash

It is the most restrictive sandbox possible short of a VM. Use it as your mental default for anything that is not your own code on your own machine.

A4 — The safety net: git with no push

It is not a flag, it is a property of the environment that changes the risk arithmetic. If the project is in git with a remote, and the agent does not have push permission, the worst case, that it corrupts every file in the project, is reverted with:

git checkout . # back to the last commit
# and if it touched .git (unlikely): delete the dir and re-clone

The remote was never touched. Sandbox for the filesystem + git for the code + manual push is already a reasonable level for daily use: ai-jail protects your data and the system, git protects the code, and the decision to publish is still yours.

Track B — Cluster (RKE2 with Cilium + Tetragon)

The untrusted agent, or the inference that executes generated code, runs as a pod. The same principle as on the client, different primitives. We assume a generic RKE2 cluster with Cilium as CNI and Tetragon already deployed (the eBPF agent DaemonSet on every node).

B0 — The pod baseline

Before any eBPF, the standard kit. Unprivileged securityContext, read-only root, default seccomp:

apiVersion: v1
kind: Pod
metadata:
 name: ai-agent
 namespace: agentes
 labels:
 app: ai-agent
spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 10001
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: agent
 image: registry.interno/ai-agent:pinned
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop: ["ALL"]
 volumeMounts:
 - { name: work, mountPath: /work }  # the only writable one
 volumes:
 - name: work
 emptyDir: {}

And the default egress cut, the cluster twin of --unshare-net. A default-deny egress NetworkPolicy in the namespace, opening only DNS and the bare essentials:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-egress
 namespace: agentes
spec:
 podSelector: {}
 policyTypes: ["Egress"]
 egress:
 - to:
 - namespaceSelector:
 matchLabels: { kubernetes.io/metadata.name: kube-system }
 ports:
 - { protocol: UDP, port: 53 }
 - { protocol: TCP, port: 53 }

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

For genuinely untrusted code, take it out of the shared kernel. With Kata deployed there is a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
 name: kata
handler: kata

And the pod asks for it with one line, runtimeClassName: kata, running in its own microVM with a dedicated kernel instead of sharing the node’s:

spec:
 runtimeClassName: kata  # ← the pod runs in a microVM, not on the node's kernel
 # ...the rest the same as B0

It is the cluster twin of isolation by construction: a kernel exploit inside the pod does not reach the node.

B2 — Tetragon, observation phase (Post)

Now the layer that separates a platform with runtime visibility from one without. Observe first, never kill from the outset. A TracingPolicyNamespaced, scoped to the namespace and to the agent’s label, that reports (does not kill) three things: process executions, network connections and opens of sensitive paths. action: Post only emits the event.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-observa
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- outbound connections ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchActions:
 - action: Post
 # --- opens of sensitive files ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets"
 - "/work/.git/config"
 matchActions:
 - action: Post

(Process executions need no kprobe: Tetragon emits process_exec/process_exit natively.) Deploy it and watch the events live from the node’s Tetragon pod:

kubectl apply -f agente-observa.yaml
# readable events, filtered by namespace:
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
 tetra getevents -o compact --namespace agentes

Leave this running for a typical working day of the agent. Note which destinations it really connects to (your internal registry, your HF mirror, your vLLM endpoint) and which paths it opens. That is your baseline: the list of what is legitimate. Without this step, a Sigkill kills good work and generates an availability incident, precisely what ENS asks you to avoid.

B3 — Tetragon, enforcement phase (Sigkill)

With the baseline in hand, promote to blocking. Two rules. The first: kill any connection whose destination is not on the allowlist — NotDAddr inverts the match: it fires for everything that is not those networks. The second: kill any attempt to open a secrets path.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-enforce
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- egress: kill everything that is NOT on the allowlist ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr"
 values:
 - "127.0.0.1"
 - "10.0.0.0/8" # internal cluster network
 - "172.16.10.20" # internal registry (example)
 matchActions:
 - action: Sigkill
 # --- reading secrets: kill the process ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets/kubernetes.io/serviceaccount/token"
 - "/work/.ssh"
 matchActions:
 - action: Sigkill

kubectl apply -f agente-enforce.yaml

Now the agent can do whatever it likes inside the pod, but the instant it tries to connect to a disallowed destination or read the service account token, Tetragon kills it in the kernel, before the packet leaves or the read returns bytes. It is the cluster twin of the curl blocklist and the unmounted ~/.ssh, but applied at runtime and over any binary, not just the ones you know about.

Operational warning. Enforcement with Sigkill requires a recent kernel with eBPF support for the action (5.10+ is safe). Deploy agente-enforce in a test namespace first, and keep agente-observa active in parallel: if the block fires, the Post event tells you exactly what triggered it. Adopt first, block later.

The client ↔ cluster equivalence table

The same vector, the two primitives. This is “extrapolating the technology” made explicit:

Threat vector	Client (workstation)	Cluster (RKE2)
`$HOME` / writable root	`$HOME` as ephemeral tmpfs (`bwrap`)	`readOnlyRootFilesystem: true` + `emptyDir`
Arbitrary egress	`curl`/`wget` blocklist · `--unshare-net`	default-deny NetworkPolicy + Tetragon `NotDAddr`→`Sigkill`
Reading secrets	`~/.ssh`/`~/.aws`/`~/.gnupg` not mounted	secrets outside the pod + Tetragon `security_file_open`→`Sigkill`
Kernel escape	Landlock (2nd VFS barrier)	`runtimeClassName: kata` (microVM, own kernel)
No escape hatch	process inside `bwrap`, no way out	no `privileged`, `drop ALL`, `allowPrivilegeEscalation:false`
Damage to the code	git remote with no `push` → `git checkout .`	GitOps + PR review, the agent does not apply to `main`
Visibility	`--dry-run --verbose` (static, pre-run)	Tetragon `tetra getevents` (dynamic, at runtime)

Checklist of gotchas

Do not put in a Sigkill without passing through Post. The observation baseline is not optional: it is what separates “blocking a C2” from “killing your own fine-tuning job”.
The .ai-jail gets committed; secrets do not. The TOML is policy, not credentials. Check you are not putting paths holding sensitive data into rw_maps.
readOnlyRootFilesystem breaks apps that write to /tmp. Mount an emptyDir at /tmp as well as the work one.
A NetworkPolicy with no DNS rule leaves the pod blind. Open port 53 to kube-system or nothing resolves.
Kata is not free. It adds startup latency and not every workload with special devices (GPU passthrough) fits; reserve it for the untrusted, not for everything.
The Claude Code /sandbox does not cover MCP or hooks unless you enable sandbox-runtime. If your agent uses MCP servers, assume they run with full permissions until you do.
NotDAddr with literal IPs ages badly. Document the allowlist and review it when the registry or the inference endpoint changes; consider stable internal CIDRs instead of loose IPs.

References

ai-jail (Fabio Akita), GPL-3.0: https://github.com/akitaonrails/ai-jail
bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io
Tetragon — TracingPolicy: https://tetragon.io/docs/concepts/tracing-policy/
Tetragon — enforcement (Sigkill/Override): https://tetragon.io/docs/concepts/enforcement/
Kata Containers — Kubernetes RuntimeClass: https://katacontainers.io
Kubernetes — Pod Security & seccomp: https://kubernetes.io/docs/tutorials/security/seccomp/
Kubernetes — Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Cilium: https://cilium.io

The contractor with the master key: isolating AI agents from workstation to cluster

Tue, 09 Jun 2026 16:00:00 +0200

First post of a pair on AI agent isolation. This one fixes the what and the where: the complete map of isolation primitives and which domain each belongs to. The sibling runbook fixes the how, with commands: ai-jail and bubblewrap on the client, Tetragon TracingPolicy and RuntimeClass on the cluster. If you are only going to read one, this one gives you the mental model; the other one gives you the files to copy and paste.

TL;DR

An AI agent that runs code needs access to your filesystem and your tools: compiler, linter, grep, make, cargo, npm. That is the minimum to be useful. The problem is that alongside that access travels the ability to read ~/.aws/credentials, exfiltrate your SSH keys or fire an rm -rf outside the project directory. And no malicious model is needed: a compromised dependency in an npm install is enough, because the well-meaning agent and the poisoned post-install script run with the same permissions. The answer is not to trust the LLM’s good intentions; it is to isolate in order to bound the blast radius. This post walks the five isolation families of 2026, from the process sandbox to the full VM, and sorts them into two columns: what applies on the client (the developer’s workstation: bubblewrap, ai-jail, sandbox-exec, Landlock, the native Claude Code and Codex sandboxes) and what applies on the cluster (where the agent or the inference runs on Kubernetes: namespaces+seccomp, gVisor, Firecracker/Kata microVMs and eBPF/Tetragon as the layer for live observation and enforcement). The thesis: the threat model is the same in both places; the tools are not. Policy extrapolates; the primitive gets rewritten.

The analogy: the contractor with the master key

You hire a builder for a renovation. He is competent and acts in good faith. But two things happen that you do not control. The first: he may misread the instruction and knock down the wrong partition wall. The second, worse: his toolbox may have been tampered with before he came through your door — somebody put something inside — and when he opens it in your living room, that something activates.

Nobody sensible gives him the master key to the whole building. You open the room he works in, you leave him the tools he needs, and you keep the study with the safe and the server room locked. If the renovation goes wrong, by error or by sabotage, the damage stays in that room.

An AI agent is that contractor. The sandbox is the key policy: you give it the project room and the tools, not the master key to the system. And here is the twist that justifies two posts: the builder works in two different buildings. One is your flat, the developer’s workstation, with your credentials, your ~/.ssh, the browser’s password vault. The other is the datacenter, the cluster where inference and autonomous agents serve customers, with data from several tenants at once. The key policy is identical in both: least privilege, bound the radius. But the lock on your flat’s door is not the same as the one at the datacenter. In the flat you fit a cylinder lock (bubblewrap). At the datacenter you put a guard watching every door and a separate wing of the building (Tetragon + microVM). Same principle, different hardware. That is extrapolating the technology, not copying it.

The threat model: what a runaway agent can do

Before choosing a lock it helps to enumerate the burglar. The attack surface of an agent that executes arbitrary bash breaks down into five concrete threats. Not all are defended by the same layer and, an uncomfortable fact that the security documentation itself acknowledges, no single layer covers them all.

Threat	What the agent does	Minimum layer that cuts it
Filesystem out of scope	Reads `.env`, `~/.ssh/id_rsa`, system secrets; modifies sources outside the project	Process sandbox (path allowlist)
Arbitrary network egress	Exfiltrates data, receives instructions from a remote C2, calls unauthorised APIs	Network blocking / NetworkPolicy / microVM
Kernel syscall surface	A kernel exploit from inside the container escalates to the host (shared kernel)	gVisor or microVM (dedicated kernel)
Cross-tenant leakage	One customer’s workload reads another’s data on a multi-tenant platform	microVM (de facto standard)
Secret exfiltration	Extracts tokens and environment variables via `/proc` or the environment	`--clearenv` / tmpfs `$HOME` / secrets outside the pod

There is a sixth threat that no sandbox solves: prompt injection. If an attacker manages to slip instructions into the agent’s context — a poisoned comment in the code, a malicious file the agent reads, an adversarial response from a tool — the agent will execute those instructions with whatever permissions the sandbox grants it. Isolation shrinks the impact radius of a successful injection; it does not prevent the injection. That is why the sandbox is a layer, not the solution: on top go input validation, tool-call allowlists and output auditing. The sentence to internalise: isolation does not make the agent trustworthy; it bounds what an untrustworthy agent can break.

Two domains, one policy

The builder works in two buildings. The distribution of tools reads better in two columns:

The client: isolating the agent on the workstation

Here the agent is a coding assistant — Claude Code, Codex, OpenCode, Cursor — that a developer launches on their machine. One process, one user, and right next to it the juiciest assets that exist: ~/.aws/credentials, ~/.ssh, ~/.gnupg, the browser’s password store. The matching tier is the lightest one: the process sandbox.

bubblewrap (Linux) and sandbox-exec (macOS). bubblewrap (bwrap) is the same sandbox Flatpak uses to isolate each desktop app: a ~50 KB binary, ~4,000 lines of C, maintained by the GNOME team and, the key property, it runs without root via CLONE_NEWUSER, creating namespaces with no elevated privileges. It mounts $HOME as an ephemeral tmpfs and only exposes the project directory, writable; the rest of the system becomes invisible. On macOS the equivalent is sandbox-exec with SBPL profiles: a legacy Apple API, officially deprecated and with no public replacement, but working today. Parity between the two is not exact — on macOS the GPU (Metal) and the display (Cocoa) are system services and sandbox-exec does not restrict them — but both protect what matters: access to the sensitive areas of the filesystem.

Landlock as a second barrier. bubblewrap isolates via namespaces and mounts; Landlock, a Linux Security Module available since kernel 5.13, restricts access at the VFS level, independently of namespaces. It does not replace bwrap: it complements it. It closes vectors that mount-based isolation does not cover on its own (escape routes via /proc, symlink tricks inside allowed mounts) and acts as a safety net should the namespace machinery have a bug. It is defence in depth inside the client itself, and it degrades cleanly to a no-op on kernels that do not support it.

Dev containers, when reproducibility is needed. A dev container (devcontainer.json, what Codespaces and Cursor use) is a Docker container with a configuration layer on top. It gives reasonable filesystem isolation and easy reset (destroy and recreate), but it shares the host kernel, the same limitation as any Docker, and it tends to be long-lived, accumulating state. For an agent running your own team’s code on your own machine it is a good repeatability story; it is not, on its own, the isolation layer for untrusted code.

What wraps all of this. A hand-written bash script works, but it does not scale to a team. The tools that package it:

ai-jail (Rust, GPL-3.0): wraps bwrap/sandbox-exec with per-project config in a .ai-jail file (TOML, committable to the repo, so the whole team inherits the same policy), auto-detection of GPU/Docker/display, a --lockdown mode (project read-only, network cut with --unshare-net, --clearenv), --dry-run for auditing, and --bootstrap to generate each agent’s permission allowlists. It is tool-agnostic: the same binary serves Claude, Codex, OpenCode or Crush. It also applies Landlock automatically on 5.13+ kernels as defence in depth.
The Claude Code /sandbox: since October 2025 Claude Code ships its own sandbox that uses, exactly, bubblewrap on Linux and sandbox-exec on macOS. Its Sandboxed Bash isolates shell commands, but not the file tools, the MCP servers or the hooks, which run with the process’s full permissions unless you enable the beta sandbox-runtime package, which wraps the whole process. There is one nuance worth knowing: if a command fails because of a restriction, the agent can retry with dangerouslyDisableSandbox — it is opt-out, not opt-in.
Codex CLI: three modes via --sandbox (read-only, workspace-write, danger-full-access); the recommended default is workspace-write. The philosophy is deliberate: Codex does not provide the isolation, it delegates it to the environment that wraps it. danger-full-access only makes sense inside a microVM.
Cursor: its cloud agents run in isolated VMs; /worktree creates a single-use isolated worktree per task, and /best-of-n launches several attempts in parallel in separate worktrees.

The cluster: isolating the agent in production

The second building is the datacenter. Here the “agent” can be an autonomous agent running with no human in front of it, or the inference service itself executing generated code, or a multi-tenant workload where one customer’s pod must not touch another’s data. The process is no longer one: they are pods in a Kubernetes cluster (RKE2/RKE3 on a typical sovereign platform). The primitives change in nature.

The pod baseline. First of all, the standard kit: Linux namespaces, seccomp (RuntimeDefault) to trim the syscall surface, cgroups for resource limits, an unprivileged securityContext (runAsNonRoot, readOnlyRootFilesystem, dropping all capabilities) and NetworkPolicy to cut egress. It is the cluster equivalent of the process sandbox allowlist. Necessary, but it shares the kernel with the host: insufficient for genuinely untrusted code.

gVisor (runsc). Google’s user-space kernel: it intercepts the workload’s syscalls before they reach the host kernel and serves them inside a Linux kernel reimplemented in Go (the Sentry). The surface exposed to host kernel vulnerabilities drops sharply, while keeping fast startup and a low footprint. It is the middle ground when the risk of a kernel escape is real but the overhead of a microVM is not acceptable.

Firecracker / Kata microVMs. The de facto standard for untrusted code in 2026. Firecracker (AWS’s VMM written in Rust, on top of KVM) gives each sandbox a dedicated Linux kernel: a kernel exploit inside the microVM does not reach the host by construction. It is what sits underneath Vercel Sandbox (GA January 2026) and E2B. On Kubernetes, Kata Containers brings that model to a RuntimeClass: you mark the untrusted agent’s pod with runtimeClassName: kata and it runs in its own microVM instead of sharing the node’s kernel. For multi-tenant with generated code, this is the baseline, not the luxury.

eBPF / Tetragon: the layer we already have. Here is the piece that separates a platform with runtime observability from one that only trusts its configuration. The previous layers are static: they define what the pod can do before it starts. Tetragon, Cilium’s runtime security component, based on eBPF, is dynamic: it observes, in the kernel and at minimal cost, every process execution, every network connection and every file open in every pod, and it can act inline. It does not replace the sandbox; it watches it from inside the kernel. Where bubblewrap on the client blocks curl with a command blocklist, Tetragon on the cluster hooks tcp_connect in the kernel and, if the destination is not allowed, kills the process with Sigkill before the packet leaves. Where the client hides ~/.ssh behind a tmpfs, Tetragon hooks security_file_open and reports, or kills, any attempt to read a sensitive mounted path. It is the guard patrolling the corridors while the microVMs are the walls. And it is exactly the kind of control that materialises the ENS monitoring and traceability measures (op.mon, op.exp) without instrumenting the application: visibility lives in the kernel, not in the agent’s code.

The landscape table

The five families, their relative isolation strength, their startup cost and the domain where they live:

Tier	Primitive	Isolation	Startup	Natural domain
Process sandbox	Seatbelt · bubblewrap	Baseline	~0 ms	Client (Claude Code default)
Dev container	Docker + seccomp	Moderate	seconds	Client / cluster (repeatability)
User-space kernel	gVisor (`runsc`)	Strong	ms	Cluster (medium multi-tenant)
microVM	Firecracker · Kata	Strongest (practical)	<1 s	Cluster (untrusted code)
Full VM	KVM · EC2	Maximum	30 s+	Cluster (external boundary, compliance)
Runtime enforcement	eBPF · Tetragon	Cross-cutting	always on	Cluster (watches+kills on top of any tier)

Tetragon gets its own row on purpose: it is not a rung on the ladder, it is a cross-cutting layer that operates on top of any of the others. It stacks with all of them.

A numerical note on why the “startup” column decides as much as the “isolation” column. A full VM wins on raw isolation but takes tens of seconds to provision; for an agent that needs a fresh environment per request or per session, that cost is prohibitive. A Firecracker microVM starts in under 1 second and a process sandbox in ~0 ms. That is why the dominant pattern in 2026 is not “the most isolated VM”, but a full VM as the external boundary plus a microVM as the per-request execution unit inside, the architecture of Vercel, AWS Lambda and E2B. On the client the arithmetic is the opposite: the developer launches the agent dozens of times a day interactively, and a startup measured in seconds would break the flow; hence the process sandbox, with its microsecond overhead, is the correct default.

Extrapolate, do not copy

The thesis of this pair of posts fits in one sentence: the threat model is invariant across domains; the primitive that implements it is not. The client and the cluster defend exactly the same five vectors — filesystem, network, kernel, multi-tenant, secrets — but with toolboxes that do not overlap. Every control has its twin on the other side:

$HOME as an ephemeral tmpfs (client) ↔ readOnlyRootFilesystem + emptyDir (cluster).
A curl/wget blocklist in bwrap (client) ↔ a TracingPolicy on tcp_connect in Tetragon plus NetworkPolicy (cluster).
--unshare-net in lockdown (client) ↔ default-deny NetworkPolicy (cluster).
No escape hatch, the process lives inside bwrap (client) ↔ no privileged, no hostPath, RuntimeClass kata (cluster).
~/.ssh and ~/.aws never mounted (client) ↔ secrets outside the pod plus Tetragon watching security_file_open (cluster).

The sibling runbook turns each of these equivalences into concrete files. What matters here is the method: when someone shows you an agent sandbox, be it the Claude Code /sandbox on a laptop or a microVM on a PaaS, the useful question is not “which tool does it use?”, but “which of the five vectors does it close, and which does it leave open?”. The tool gets replaced; the threat map stays.

What no sandbox solves

Three limits that Anthropic’s own documentation states, and which are worth keeping in view so as not to sell smoke:

Egress remains a risk in any sandbox that allows outbound connections. If the agent can open a connection, it can exfiltrate. That is why the client lockdown cuts the network and the cluster uses default-deny NetworkPolicy plus Tetragon: you do not trust “filtering well”, you trust “not letting anything out”.
Code modification remains possible in any sandbox with the project directory mounted writable. The remedy is not a sandbox-technical one, it is git: with the remote untouched and no push permission, the worst case is corrupting the local working copy — git checkout . and start again. The damage never reaches the remote.
No sandbox prevents a compromised prompt from reaching the API. Isolation bounds the impact of an injection; it does not prevent it. The complementary defences — input validation, tool-call allowlists, output auditing — are mandatory alongside isolation, not instead of it.

The operational conclusion: isolation shrinks the blast radius; defence in depth is what closes the circle. A process sandbox for trusted code on a known machine is appropriate and practically free. For an agent acting on user prompts, executing generated code or running multi-tenant, the acceptable minimum in 2026 is a microVM, with Tetragon watching from above. Pick the tier that matches your real threat, check which vector it leaves open, and stack complementary controls on top.

References

bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io · https://docs.rs/landlock
ai-jail (Fabio Akita): https://github.com/akitaonrails/ai-jail
gVisor: https://gvisor.dev
Firecracker: https://firecracker-microvm.github.io
Kata Containers: https://katacontainers.io
Tetragon (Cilium): https://tetragon.io
Vercel Sandbox — concepts: https://vercel.com/docs/vercel-sandbox/concepts
E2B: https://github.com/e2b-dev/E2B
Claude Code sandboxing: https://docs.claude.com/en/docs/claude-code
Codex CLI: https://github.com/openai/codex

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

Sat, 06 Jun 2026 12:00:00 +0200

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

Disaggregated serving. The KV-cache that travels between the prefill pool and the decode pool moves over RDMA. These are large transfers, sensitive to latency and bandwidth, which in multi-node setups go out through the NIC.
Multi-node NCCL collectives. When tensor/pipeline parallelism crosses the node boundary, the all-reduce/all-gather operations no longer go over NVLink but over GPUDirect RDMA on the NIC.

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Three kernel facts the analogy compresses:

The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 piece	What it does	What Cilium eBPF puts in
`kube-proxy` (iptables/IPVS)	Service balancing	LB in eBPF; with `kubeProxyReplacement=true`, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)	VXLAN overlay + NetworkPolicy	native datapath (`routingMode=native`), L3/L4 and L7 NetworkPolicy in eBPF
veth per pod	namespace interface pair	netkit (kernel ≥6.8): namespace overhead ~0
host iptables traversal	netfilter hooks	eBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus = 2-31,34-63 # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33 # kubelet housekeeping (post 3)
IRQ affinity = 0-1,32-33 # NIC IRQs → housekeeping ONLY (this post)
 # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
 --namespace kube-system \
 --set routingMode=native \
 --set bpf.datapathMode=netkit \  # namespace overhead ~0 (kernel >=6.8)
 --set bpf.masquerade=true \
 --set kubeProxyReplacement=true \  # replaces RKE2's kube-proxy
 --set enableIPv4BIGTCP=true \  # 192k super-packets (mlx5/ice NIC)
 --set enableIPv6BIGTCP=true \
 --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
 --set bpf.mapDynamicSizeRatio=0.08 \
 --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
 name: gpu-nic-numa-aligned
spec:
 spec:
 devices:
 requests:
 - name: rdma-nic
 deviceClassName: dra.net  # NICs published by DRANET
 constraints:
 - requests: ["rdma-nic"]
 matchAttribute: "dra.net/pcieRoot" # same PCIe root as the GPU
 # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

References

Cilium, Tuning Guide (netkit, host-routing, BIG TCP, XDP, pin the IRQs and kill irqbalance): https://docs.cilium.io/en/stable/operations/performance/tuning/.
Cilium 1.19 (February 2026), Cilium at Ten Years — hardening of encryption, policies and observability: https://www.infoq.com/news/2026/02/cilium-119/.
Isovalent, Cilium 1.18 (IPv6, encrypted overlay, ingress bandwidth, policy perf): https://isovalent.com/blog/post/cilium-1-18/.
RKE2, Network Options (Canal by default; Cilium with kube-proxy replacement): https://docs.rke2.io/networking/basic_network_options.
Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/.
Kubernetes blog, v1.36: More Drivers, New Features, and the Next Era of DRA (May 2026): https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/.
DRANET (kubernetes-sigs), the network DRA driver and the paper The Kubernetes Network Driver Model (+59.6% all_gather / +58.1% all_reduce): https://github.com/kubernetes-sigs/dranet.
AKS Engineering, Optimizing RDMA performance for AI workloads on AKS with DRANET (April 2026): https://blog.aks.azure.com/2026/04/01/dranet-rdma-optimization-for-ai-on-aks.
Linux network tuning — IRQ affinity, RSS/RPS/RFS and NUMA softirq: https://andreaskaris.github.io/blog/networking/rss-irq-affinity-and-rps/.

eBPF in local inference and statistical drift detection: closing the LLM observability loop in 2026

Wed, 20 May 2026 09:00:00 +0200

TL;DR

Tracing, evals, guardrails, MCP observability: the layers we have already covered see what is happening right now. What they do not see is what changes silently: the agent that answered well last week and this week, without anyone touching anything, answers slightly worse. What they also do not see is the fine mechanics of local inference: why a llama.cpp on an edge device takes 200 ms when it should take 100, which specific runtime function is the bottleneck. This post closes the two series of the week with the two layers that were missing: eBPF applied to local inference (uprobes on llama.cpp, vLLM, libcudart.so, integrated hardware perf counters, with <4% overhead, formalised in the ProfInfer 2026 paper, which is to local inference what Hubble is to the network) and statistical analysis of agent flows to detect drift before your user notices it (KS, PSI, MMD, embedding-space clustering, with Evidently AI, NannyML and WhyLabs as the dominant tools). The three types of LLM drift in 2026, prompt drift, model drift and eval-score drift, demand different tests. The complete stack (tracing, evals, guardrails, MCP observability, eBPF observability, drift detection) forms the loop that any serious agentic system needs in order to operate with a real SLA rather than with hope.

This post closes two series: the post-tracing series (Evals, Guardrails, MCP observability) and the eBPF series (eBPF from zero to Cilium, Tetragon, Hubble, AgentSight). It brings the two threads together: eBPF applied to the local inference engine plus statistical analysis of the flows that every layer produces.

The analogy: the agent’s cardiogram

A doctor who only looks at acute symptoms, the patient arrives with a high fever and you have to act, is practising reactive medicine. To practise preventive medicine, they need time series: blood pressure every year, cholesterol every six months, an ECG when there is a suspicion. It is not “right now” information, it is information about how something that should be stable is evolving. When a time series deviates from its baseline, you investigate before it turns into a high fever.

The LLM observability layers we have seen so far are reactive medicine: tracing tells you what happened in a specific conversation; evals tell you whether that conversation was good; guardrails tell you whether there was a specific threat; MCP observability tells you which tools were invoked and how. They all look at events, not trends.

Drift detection is preventive medicine. It looks at time series, of prompt embeddings, of evaluation scores, of distributions of generated tokens, and fires alerts when something drifts away from its normality. It does not tell you “this answer is bad”; it tells you “the distribution of prompts over the last 6 hours does not look like the distribution of the last month”. There you decide whether to investigate.

And the other half of the post, eBPF in local inference, is the equivalent of the MRI scanner: once you know there is a problem, it lets you see the inside of the model at a resolution no external wrapper gives you. To see which specific runtime function is slow, which CUDA kernel is the bottleneck, how tokens move through the internal buffers before going out to the client.

The two together close the loop: the time series detect that something is wrong, the scanner locates where.

Part 1 — eBPF applied to local inference

Why local inference changes the game

When the LLM runs locally, vLLM on a Kubernetes node, llama.cpp on an edge device, Ollama on a workstation, MLX on macOS, and not behind an external API, observability changes shape:

You control the binary: you can attach hooks that would otherwise be impossible.
The internal buffers exist in accessible RAM: the output-token stream, the logits, the KV caches, the scheduler structures are right there, at addresses a uprobe can read.
There is no cable to sniff: AgentSight’s analogy with SSL hooks does not apply because there is no TLS; the model answers you with an in-process function return, not with an HTTPS response.
The distance between kernel and model is minimal: the CUDA kernels running attention are one syscall deep; eBPF can observe both sides of that boundary with the same tracer.

This opens up a class of observability that is structurally impossible with LLM-as-a-service (the Anthropic, OpenAI or Vertex APIs). For apps serving inference on-premise or on-edge, an inference cluster, a mobile device, an RTX 4090 server in the rack, it is a new layer.

ProfInfer: the paper that formalises the pattern

ProfInfer (arxiv 2601.20755, 2026) is the reference academic piece that systematises what the ecosystem had been doing ad-hoc. The paper’s subtitle says it all: An eBPF-based Fine-Grained LLM Inference Profiler.

What it proposes:

Attaching uprobes dynamically to runtime functions of engines such as llama.cpp (and by extension vLLM, Ollama). No recompiling, no modifying the source code. It is like bpftrace for LLM inference.
Combining runtime events with hardware performance counters. A uprobe tells you when llama_decode runs; a hardware counter tells you how many floating-point instructions executed while it was inside. The correlation between the two is what gives the fine resolution.
<4% measured overhead under real workloads. That is a production-grade cost.
Visualisations in three views: operators (which tensor operations ran), graphs (how they relate) and timelines (when).

The paper focuses particularly on models on mobile platforms (Llama served on a Pixel or an iPhone), where classic observability with Prometheus and exported metrics barely exists. But the pattern applies to any local inference.

Where to hook: the map by engine

Now to the detail of the hooks. The target functions vary by engine:

llama.cpp

llama.cpp is pure C++, with symbols visible in the binary. The typical hooks:

llama_decode: the function that runs one inference pass (processes the current batch). Spans for latency per iteration, tokens processed.
llama_token_to_piece: converts a token ID to text. A hook here captures the stream of generated tokens before returning to the caller. It is the local equivalent of the SSL uprobes: you see the model’s output before it even reaches the consumer.
llama_get_logits: reads the logits of the last decode. If you want to record the model’s probabilities (not just the chosen token), here.
ggml_compute_forward_* (several functions): the operation kernels (matmul, attention, layernorm). Hooks for per-operation profiling.
ggml_backend_*: the backend functions (CPU, Metal, CUDA, ROCm). Hooks here break down the cost per device.

An example with bpftrace:

# Latency and count of llama_decode
bpftrace -e '
uprobe:/path/to/llama-server:llama_decode {
 @start[tid] = nsecs;
}
uretprobe:/path/to/llama-server:llama_decode /@start[tid]/ {
 @decode_lat = hist((nsecs - @start[tid]) / 1000);
 delete(@start[tid]);
}
'

Output: a histogram of decode latencies in microseconds. Zero modification to the binary.

vLLM

vLLM is mostly Python. The C/CUDA symbols live in its native extensions (vllm._C, vllm._moe_C). The typical hooks:

uprobes on vllm._C.* for custom operators (the paged attention kernel, the sampling kernel).
uprobes on libcudart.so and libcuda.so to capture cudaMalloc, cudaLaunchKernel, cudaMemcpy. This maps the cost of host↔device transfers and kernel launches.
Python tracepoints with bpftrace over usdt points: vLLM does not expose native static tracepoints, but they can be placed with USDT (dtrace style) at strategic points in the scheduler.

vLLM also exposes native Prometheus metrics (vllm:num_requests_running, vllm:gpu_cache_usage_perc, and so on). The added value of the eBPF approach is going down from the scheduler’s metrics to the individual functions: when a request is slow, seeing whether it was prefill, decode, scheduler overhead, transfer or synchronisation.

CUDA in general

Independently of the engine, uprobes on libcudart.so capture all the CUDA activity of the process:

cudaMalloc(size) → tracking of device memory allocations.
cudaLaunchKernel(func, ...) → spans for each kernel launch.
cudaMemcpyAsync(dst, src, size, kind) → host↔device transfers.
cudaStreamSynchronize(stream) → synchronisation points (where the host waits for the device).

This gives you a complete timeline of CUDA activity without needing NVIDIA Nsight Systems (which is excellent but heavy and oriented towards development, not continuous production).

Hardware counters: the other half

eBPF can read performance counters from the CPU/GPU’s PMU (Performance Monitoring Unit). That includes instructions executed, cache misses, branch mispredictions and, on GPUs with support, FLOPS, SM occupancy, HBM bandwidth.

Combining:

uprobe: “llama_decode ran from T1 to T2 with tokens=4”.
perf counter: “during that window, L2 cache misses = 15,000, instructions = 2.3 million”.

This lets you answer: why is it slow? Is it memory-bound (many cache misses), compute-bound (all the instructions in the FPU), bandwidth-bound (a lot of data movement)? State of the art for professional profiling.

Comparison with AgentSight

There are two eBPF products for LLMs today with different focuses:

AgentSight (covered in the eBPF series): observes agents that call external APIs. It hooks SSL to see the plaintext of HTTPS calls to the remote LLM, plus stdio for local MCP servers. The client view.
ProfInfer / the eBPF-on-local-inference pattern: observes the engine that runs the model locally. It hooks the engine’s internal functions (llama.cpp, vLLM) and the CUDA layer. The server (internal) view.

They are complementary. If your agent uses the Claude API plus your own local vLLM with Llama 3 for specific tasks, AgentSight sees the first, eBPF/ProfInfer sees the second. If everything is local, clearly the domain of the second. If everything is an external API, of the first.

Use cases for eBPF in local inference

Three cases where it is the right tool:

Fine-grained profiling for optimisation: your vLLM takes 50ms more per token than expected. With eBPF plus hardware counters you pin down which specific kernel it is. Previously this required Nsight Systems in a development session; now it is continuous in production.

Token-level observability without modifying the engine: capturing the stream of generated tokens before returning them to the client. Useful for auditing, for drift detection over the outputs, for local tracing without going through wrapper instrumentation.

Detecting specific degradation: a new version of vLLM introduces a subtle regression in paged attention. With perf counter baselines, you detect the change even if the external metric (tokens/sec) looks the same.

Part 2 — Statistical analysis of flows: detecting drift

Now to the other side of the problem: the time series.

Why tracing, evals and guardrails do not detect drift

The layers we have already seen operate on individual events:

Tracing: a trace of one conversation.
Evals: a score for one answer.
Guardrails: a verdict on one prompt or answer.
MCP observability: spans of one tool invocation.

Each one answers a point-in-time question (“is this all right?”). None of them answers the question of evolution (“is something changing over time?”).

The operational problem: drift is invisible in individual events. If the mean eval score drops from 0.92 to 0.85 over three weeks, no individual evaluation will raise an alarm, since they all continue to be “reasonable”. What changes is the distribution. And that only shows up when you look at many evaluations aggregated over time.

The three types of LLM drift in 2026

FutureAGI consolidates them like this, and the industry is converging on this vocabulary:

1. Prompt drift: someone updates the system prompt and the side effects break cases that used to work. Almost always intentional, but with unanticipated consequences. Detection: comparing response distributions before and after the change, monitoring eval scores per prompt version (linked in Langfuse, see the AgentSight post where we covered prompt management).

2. Model drift: the provider (OpenAI, Anthropic) updates the model without warning. The same prompt produces answers with a slightly different tone, similar but different quality, or degradation on some subset. Detection: comparing today’s response embeddings with the baseline; monitoring rubric scores; alerting if intra-model variance grows.

3. Eval-score drift: the rolling mean of your eval metrics (faithfulness, answer relevancy, custom rubrics) trends downwards. The root cause can be any of the above or a change in the user mix. Detection: alerts on the trends of the eval series.

To these three you can add a fourth, subtler one:

4. Persona drift / user mix shift: the population of users using the system changes. It is not that the model or the prompt got worse; it is that the new users ask different questions and the system, while still just as good at what it was good at, fails on the new material. Detection: embedding clustering of prompts, monitoring the appearance of new clusters or the growth of a minority one.

The key technical concept: embedding-space shift

Stack Pulsar puts it plainly: in LLMs, drift is best measured in embedding space. Classic distances in token space do not capture fine semantics; in embedding space they do.

The canonical pipeline:

Establish a baseline: during a stable period (say the first two weeks after a release), capture a large sample of prompt and response embeddings.
Continuous monitoring: every hour or every day, capture a new sample of production traffic.
Compare distributions: apply a statistical test that compares the current distribution with the baseline in embedding space.
Alert: if the divergence exceeds a threshold, fire an alert and an investigation workflow.

As a bonus, monitor clusters: if your baseline has 5 clusters of prompts (technical questions, general support, sales, and so on) and suddenly a sixth cluster appears that was not there, the most likely explanation is that a new user segment has arrived.

Statistical tests: KS, PSI, MMD

Three tests that any drift system uses, each with its place:

Kolmogorov-Smirnov (KS): non-parametric. It computes the maximum distance between two empirical CDFs. It returns a statistic and a p-value. Advantage: very sensitive to subtle changes, especially in the tails. Disadvantage: with large datasets it is “too sensitive”, firing alarms for changes that are real but clinically irrelevant.

Population Stability Index (PSI): you bin the reference distribution and the current one, then sum (p_actual - p_ref) × log(p_actual / p_ref) over the bins. The canonical interpretation: PSI < 0.1 stable, 0.1-0.25 mild drift, > 0.25 significant drift. Advantage: interpretable, threshold-based, with a tradition of use in credit scoring (Capital One, Goldman Sachs). Disadvantage: less sensitive than KS, so it misses drift in the tails.

Maximum Mean Discrepancy (MMD): measures the divergence between two distributions by embedding each one in a Hilbert space via a kernel. It works for complex multivariate distributions (high-dimensional embeddings). Advantage: the only one that scales reasonably to embeddings of 768/1024/4096 dimensions. Disadvantage: harder to interpret.

The recommended practice in 2026:

PSI for simple features (prompt length, tokens, number of tools invoked).
KS for continuous features where you want high sensitivity.
MMD for embeddings (high-dimensional spaces).

Evidently’s analysis on real datasets showed that KS detects drift 6+ hours earlier than PSI in some incidents. The operational consequence: use KS for early warning, PSI for confirmation with an interpretable threshold.

Tools in 2026

Three products dominate the field:

Evidently AI

Evidently is open-source (Apache 2.0), Python-first. Its value:

HTML drift reports: you generate a report comparing two datasets (reference vs current) and you get an HTML file with all the statistical tests, visualisations and conclusions. No server, no infrastructure; one shareable file.
Native LLM support: on top of tabular, it supports text. It computes embeddings and applies the appropriate tests.
100+ metrics in the suite. It covers everything from a single framework.
Integration with MLflow and kube: CI workflows with reports on every release.

from evidently import Report
from evidently.metrics import DataDriftPreset

ref = load_baseline_dataset() # prompts from last week
cur = load_current_dataset() # prompts from the last hour

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=ref, current_data=cur)
report.save_html("drift_report.html")

When this functionality detects drift, it also tells you which column and which test fired.

NannyML

NannyML has a different focus: estimating model performance when you have no ground truth. The techniques:

CBPE (Confidence-Based Performance Estimation): estimates accuracy using the model’s confidence in its predictions.
DLE (Direct Loss Estimation): estimates the loss directly.

Useful when your LLM app has no immediate human feedback but you want to know whether its quality has dropped. Apache 2.0, Python.

WhyLabs

WhyLabs is commercial (with whylogs as the underlying OSS library), aimed at enterprise production:

Managed SaaS with SOC 2 Type 2 and HIPAA compliance.
Real-time monitoring via continuous log ingestion.
Embedding tracking: native support for embedding distributions, not just tabular features.
Token probability shifts: it monitors the probability distribution of generated tokens, not just metadata.

For regulated companies that do not want to operate their own drift detection platform, it is the lowest-friction option.

Other mentions

Arize Phoenix (seen in the Evals post) includes drift detection as a module. Galileo has commercial products specialised in LLM monitoring. Fiddler AI and Alibi Detect (Seldon) are more generalist alternatives that also cover LLMs.

Tool	Licence	Focus	Typical stack
Evidently AI	Apache 2.0	Drift reports + LLM	OSS Python, HTML reports
NannyML	Apache 2.0	Performance without GT	OSS Python, batch
WhyLabs	Commercial (whylogs OSS)	Enterprise SaaS, embeddings	Continuous logs, compliance
Arize Phoenix	ELv2	Tracing + drift unified	OSS, OTel-native
Galileo	Commercial	Premium LLM monitoring	SaaS, ML expert team
Alibi Detect	Apache 2.0	General drift detection	OSS Python, Seldon ecosystem
Fiddler AI	Commercial	Explainability + monitoring	Enterprise SaaS

Part 3 — The complete stack: how everything fits

Let us recap the layers the two series have covered, ordered from closest to the individual request to closest to the aggregate trend:

Individual EVENTS Aggregated TRENDS
│ │
Tracing ──→ Evals ──→ Guardrails ──→ MCP obs ──→ Drift detection
│ │
AgentSight ──→ Tetragon ──→ Hubble ──→ eBPF on-device
│ │
(what happens) (what changes)

Each layer answers a different question:

Layer	Question it answers	Granularity
Tracing (Langfuse, AgentSight)	What exactly did the agent do?	One session
Evals	Was the answer good?	One answer
Guardrails	Is this prompt/answer safe?	One message
MCP observability	Which tools did it invoke, at what cost?	One tool call
eBPF on agent/network (AgentSight, Hubble)	How did the system behave?	Per process/connection
eBPF on the local engine (ProfInfer-like)	How did the model execute?	Per runtime function
Drift detection	Is something changing silently?	Distribution

None of them replaces the others. Full coverage requires all seven. The practical operation:

Layers 1-3 (tracing, evals, guardrails) are mandatory from day one. Any LLM app in production that does not have them is flying blind.
Layer 4 (MCP) becomes mandatory when there are agents with tools, which is the majority in 2026.
Layers 5-6 (eBPF) become valuable when the scale justifies the operating cost (>10 services, >100 inference pods).
Layer 7 (drift) is the one that is most neglected and most expensive to ignore: it takes a day’s work to get the basic pipeline and it saves weeks of future incidents.

The operational drift pattern in 2026

The minimum recipe any serious LLM app should have:

Step 1 — Establish a baseline

During a stable post-release period (2 weeks minimum), store:

Embeddings of every prompt (vector + metadata: timestamp, user_segment, tenant).
Embeddings of the responses.
Automated eval scores over a sample (e.g. 5-10% of traffic with G-Eval).
The distribution of tools invoked (which tools, with which typical arguments, at what frequency).

Storage: any vector store plus a relational one. Reasonable cardinality at whatever scale you have.

Step 2 — Continuous comparison pipeline

Every hour (or every day depending on scale):

Take the sample for the current period (the last hour).
Apply the statistical tests against the baseline:
- PSI over simple features (prompt length, tokens, num tools).
- KS over continuous features (latency, score).
- MMD over embeddings.
Generate a drift report (Evidently does it in one line of Python).

Step 3 — Alerts and investigation workflow

Configure thresholds and routes:

PSI > 0.25 over tokens consumed: moderate alert (it may be legitimate, investigate segments).
Significant MMD over prompt embeddings: high alert (a change in user mix or a coordinated attack).
Eval rubric score down >5% over a rolling 7d: critical alert.
A new cluster in embedding space accounting for 10%+ of traffic: a review workflow (it may be a legitimate new segment or an anomaly).

Every alert must lead to a drill-down dashboard with the affected segments, not to an empty Slack message. The operational rule: if someone cannot investigate the alert in <5 minutes, it will not get investigated.

Step 4 — Baseline refresh

The baseline is not static. Every N weeks, refresh the baseline by incorporating the “new stable”. If over 3 months the usage pattern has legitimately changed (more international users, new languages), the baseline must reflect it. The typical cadence: quarterly.

Operational traps

A contaminated baseline

You take the baseline from a period that already contained the problem in embryo. Result: the baseline includes the bad behaviour and the tests never fire. Solution: verify the baseline against a second independent sample (for example, the first week vs the second) before blessing it.

A threshold that is too low

PSI > 0.05 fires constantly. Your team learns to ignore the alerts. Calibrate thresholds against the natural noise of your system: run the system with the baseline plus successive weekly samples and measure the PSI distribution; set the threshold a couple of deviations above normal.

Embeddings that are not representative

You use OpenAI’s text-embedding-3-small embedding model to detect drift in a system serving technical questions in Spanish about Cisco networks. Result: the embedding model does not capture the fine semantics of the domain. Solution: use embeddings finetuned for your domain, or one that is strong in multilingual and technical material.

Storage overload

Storing an embedding of every prompt in production at scale (millions of prompts/day) fills disks and increases cost. Stratified sampling: keep 5-10% of traffic, but make sure minority segments are over-represented so you do not lose them.

Confusing drift with “the system works”

Sometimes drift is good drift: new users discover the agent knows how to do X, and suddenly 30% of traffic is for X. The distribution changed because the product found a new use. Before pulling the alarm, check whether the change is desirable.

Privacy in embedding storage

Embeddings can be partially inverted back to their original text with embedding inversion techniques. If the prompts contain PII, storing embeddings for months for drift detection is a leak vector. Encrypt at rest and rotate regularly, or work with aggregated/averaged embeddings.

eBPF in production without profile guardrails

Attaching uprobes to hot-path functions such as llama_decode can hit throughput if it is not done carefully. Always test in staging and monitor the overhead. ProfInfer reports <4%; what you measure may vary with your binary and kernel.

Closing the two series

This week we have written 12 articles that walk the modern LLM inference stack in production from top to bottom:

LLM inference series (4 articles):

KV cache: the working memory that holds up LLM inference — fundamentals.
vLLM on Kubernetes — the engine.
PagedAttention deep dive — how it works inside.
LLM Operators on Kubernetes — orchestration.

eBPF series (4 articles):

eBPF from zero to Cilium — the substrate.
Tetragon — runtime security.
Hubble — network observability.
AgentSight — agent observability.

Post-tracing series (4 articles):

Evals — reactive quality.
Guardrails — preventive safety.
MCP observability — the tool protocol.
This one — drift detection and eBPF in local inference.

If you read the twelve in order you have a reasonably complete map of what it takes to operate AI agents in serious production in 2026, with enough detail not to crash into the usual problems in the first month. And, above all, with the mindset that LLM observability is a stack, not a product: each layer solves a problem, none of them solves all of them, and the combination is what separates an operable system from one that holds up until the first incident.

What is left for future series

MLOps specific to LLMs: continuous fine-tuning, RAG over data lakes, agent training.
Constitutional AI and alignment at runtime: how the model self-regulates with internal guardrails.
GPU networking: InfiniBand, NCCL, GPUDirect, the angle we left untouched.
Edge inference: llama.cpp on phones, MLX on macOS, Snapdragon NPU.
Theoretical inference scheduling: CFS-like algorithms applied to multi-tenant LLM serving.

We will cover them in time. That is it for now, and thanks for reading these twelve posts. If they gave you something, share them with a colleague.

References

eBPF in local inference:

ProfInfer: An eBPF-based Fine-Grained LLM Inference Profiler (arxiv 2601.20755) — the 2026 reference paper.
Monitor LLM Inference in Production 2026 (Glukhov) — Prometheus + Grafana for vLLM/TGI/llama.cpp.
AI Inference Server Observability in Kubernetes (ARMO) — the four signals MLOps tools do not capture.
vLLM vs llama.cpp: Choosing the right engine (Red Hat).

Drift detection concepts:

Tools:

Evidently AI (GitHub) — open-source.
Evidently — official site.
NannyML — performance without ground truth.
WhyLabs — managed observability.
Alibi Detect (Seldon) — general drift detection.
Arize Phoenix — drift integrated with tracing.

Statistical tests:

Data drift detection: PSI vs Kolmogorov–Smirnov (MLPipeline) — a practical comparison.
Population Stability Index for Model Drift Detection.
Which test is the best? 5 methods to detect data drift (Evidently) — the 6 hours of advantage for KS.

Cross-references (the three complete series):

LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.
Post-tracing series: Evals, Guardrails, MCP observability.

AgentSight and the new LLM tracing: zero-instrumentation with eBPF against Langfuse, LangSmith, Phoenix and company

Tue, 19 May 2026 18:00:00 +0200

TL;DR

Observing an LLM agent in production in 2026 splits into two approaches with opposite philosophies. The instrumented one, dominant until 2025, lives in tools such as Langfuse, LangSmith, Arize Phoenix, Helicone, OpenLLMetry/Traceloop or Pydantic Logfire: you install an SDK, decorate your calls, emit spans with the OpenTelemetry GenAI convention (gen_ai.request.model, gen_ai.usage.input_tokens, and so on) and export them to a backend. Enormous depth when you control the code; zero visibility when the agent is an opaque binary you run without instrumenting. The zero-instrumentation one, which AgentSight popularised in the second half of 2025, turns the perspective 180º: it puts eBPF hooks on the uprobes of the SSL/TLS libraries and captures the plaintext of every HTTPS request before encryption, without touching the app’s code, with less than 3% overhead and the guarantee of being tamper-proof (the agent cannot falsify what is seen in the kernel). Combined with BPF capture of stdio for local MCP servers, AgentSight gives you complete observability of any agent, including closed binaries such as Claude Code, Gemini CLI or Cursor, on a Kubernetes cluster. The two families are not enemies: the 2026 reference stack combines both (instrumented for your own apps with LangChain, eBPF for opaque binaries and tamper-proof compliance) over OpenTelemetry GenAI semantic conventions as the common vocabulary the ecosystem is stabilising this year.

This is the fourth and last post of the eBPF series. Part 1: eBPF from zero to Cilium. Part 2: Tetragon: runtime security. Part 3: Hubble: network observability. Here we close the circle with the semantic dimension: what an AI agent does, not only what network it opens or what syscalls it emits.

The analogy: traditional APM vs network sniffer

Anyone who has operated enterprise applications knows the two tribes of monitoring. The APM tribe (New Relic, AppDynamics, Datadog APM): you install an agent or an SDK in each application, mark spans, collect traces with enormous depth inside each process, down to lines of code, SQL queries, Java methods. The wire-level tribe (network sniffers, SolarWinds NPM-style tools, NetFlow): it does not touch the application; it observes the cable, sees protocols, latencies, retransmissions, identifies problems the app does not know it has.

Each one sees different things and both are useful. Anyone who has lived through a serious incident where APM said “all green” while users suffered knows that wire-level would have caught the problem (a saturated middlebox, a badly configured MTU, a TCP timeout). Anyone who has tried to debug a memory leak with sniffers knows it was impossible without APM.

LLM agent observability in 2026 is exactly at this point. The APM-style has been in place for a couple of years: Langfuse, LangSmith, Phoenix, OpenLLMetry. Enormous depth, requires instrumenting the app. The wire-level with eBPF has just arrived: AgentSight is the first project to take it to production. Less depth inside the agent, but it sees any agent without touching anything and it is tamper-proof. Both are useful. The industry is in full coexistence.

Why observing LLM agents is different

Before getting into tools, it is worth pausing on what makes LLM agents specific as subjects of observability:

Non-determinism. The same input can produce different outputs. Reproducing an incident requires capturing exactly the conversation, the model, the parameters and, ideally, the seed. An aggregate “p95 latency” metric falls short; what you need is replay of the individual trace.

A chain of external invocations. A typical agent calls LLM → tools (tool calling) → MCP servers → other APIs → back to the LLM. A chat session can generate dozens of chained calls that have to be correlated by trace_id to understand the decision.

Cost linear in tokens. Every call is paid for in tokens. Without tracing input/output tokens per request, you cannot assign cost to a tenant or a team, nor detect loops that eat your budget in an hour.

Semantic risk. Prompt injection (a user input containing instructions to manipulate the model), jailbreaks, secret leakage through tool calls. It is a class of problem that does not appear in traditional applications, and observability must see it.

Opaque binaries. In 2026, many teams deploy third-party agents such as Claude Code, the Cursor agent, Aider, Gemini CLI or Codex CLI as internal tools. They are not your own applications; they are closed binaries that call the vendor’s API. Instrumenting them is impossible. Observing them requires something else.

Multi-agent and orchestration. More and more architectures have agents that invoke other agents (planner → executor → critic). Observability must understand the topology, not just the individual span.

With these five points in mind, the tools we are about to see differ mainly in which parts of the problem they cover well and which parts they leave blind.

The instrumented approach: how it works

The model is direct and familiar:

Your code calls the LLM or the tools using an official library: openai, anthropic, langchain, llama_index, dspy.
You install a tracer SDK (Langfuse, LangSmith, OpenLLMetry, Logfire) that wraps or monkey-patches those libraries.
Each call emits an OpenTelemetry span with standardised attributes: model used, input/output tokens, latency, parameters, messages, tool invoked, result.
The spans are exported over OTLP to a backend that shows them as a tree of traces.

# Typical example with OpenLLMetry + any SDK
from traceloop.sdk import Traceloop
from openai import OpenAI

Traceloop.init(app_name="my-agent", api_endpoint="https://otel-collector:4318")

client = OpenAI()
# this call automatically emits a span with
# gen_ai.request.model, gen_ai.usage.input_tokens, etc.
resp = client.chat.completions.create(
 model="gpt-4.1",
 messages=[{"role": "user", "content": "..."}]
)

What you see afterwards: a dashboard with each conversation as a trace, each call as a span, the complete prompts and completions (if you opt in), the computed cost, latencies per span, errors flagged.

OpenTelemetry GenAI semantic conventions: the common vocabulary

The fragmentation of the field is being mitigated with OpenTelemetry GenAI Semantic Conventions. It is the CNCF’s effort to get all tools to emit spans with the same attribute names:

gen_ai.system — the provider (openai, anthropic, vertex_ai, and so on).
gen_ai.request.model — the model requested (gpt-4.1, claude-3-5-sonnet).
gen_ai.response.model — the model actually used (sometimes it differs, e.g. fallbacks).
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — counters.
gen_ai.request.temperature, gen_ai.request.top_p, and so on — parameters.
gen_ai.response.finish_reasons — why it ended (stop, length, content_filter).
gen_ai.operation.name — the type of operation (chat, embedding, completion).

In early 2026, the client spans moved from experimental to stable. The rest (server spans, multi-agent events) is still in development. The operational meaning: if your SDK emits these attributes, any backend that understands OTel GenAI can consume them. Moving from Langfuse to Phoenix to Helicone does not imply re-instrumenting, only changing the exporter.

The SIG is actively developing conventions for multi-agent systems: agent teams, tasks, actions, memory, artifact tracking. This is what is missing for complex agent architectures to have a common vocabulary. In 2026 it is experimental; stabilisation is expected at the end of the year or the start of 2027.

Instrumented tools: the 2026 landscape

Tool	Licence	Self-host	Focus	Where it shines
Langfuse	MIT	Yes	LLM observability + evals + prompt mgmt	Best OSS balance, complete suite
LangSmith	Commercial	No	Native LangChain/LangGraph	If you use LangChain, zero-config integration
Arize Phoenix	ELv2 (OSS)	Yes	OTel-native, strong RAG	Vector DBs, retrieval, embeddings
Helicone	Commercial + OSS lite	Yes (lite)	Simple proxy	Setup in minutes, OpenAI-only
OpenLLMetry / Traceloop	Apache 2.0	Yes	OTel SDK for LLMs	Vendor-neutral, exports to any OTel backend
Pydantic Logfire	Commercial	No	App + LLM unified	If you use Pydantic AI, native integration
Weights & Biases Weave	Commercial	Limited	Experimentation + production	If you already use W&B for training
Laminar / Braintrust	Commercial	No / Yes	Evals + tracing	More recent, focused on evaluation

Deep dive: Langfuse

It is worth pausing on Langfuse because in 2026 it is the default choice among the open-source options and the one most teams have adopted this year. It is a YC W23 project, MIT licence, and keeps a sustained release cadence with serious architectural changes between versions.

Four declared pillars: observability (tracing), evaluations, prompt management, playground/datasets. Each one separately has complete commercial products behind it; Langfuse integrates them into a single platform with a single backend.

SDK v4: OTEL-native, not a replacement

The big recent operational change is the SDK v4, a thin layer over the official OpenTelemetry client. The choice is deliberate: instead of maintaining a client of its own that would fall behind the OTel primitives, Langfuse uses the standard SDK and enriches the spans with LLM-specific attributes and helpers. The consequence: any code already instrumented with vanilla OpenTelemetry (@opentelemetry/sdk-node, opentelemetry-sdk in Python) can export to Langfuse without major changes, and the other way round, if tomorrow you want to migrate from Langfuse to another OTel backend, the spans are portable.

In Python the idiomatic decorator is @observe:

from langfuse import observe, get_client

langfuse = get_client()

@observe()
def buscar_documentos(query: str):
 # any internal call is traced as well
 return vector_store.similarity_search(query)

@observe(as_type="generation")
def llamar_llm(prompt: str):
 # marked as "generation" so it appears with LLM metadata
 return openai_client.chat.completions.create(...)

@observe()
def pipeline_rag(pregunta: str):
 docs = buscar_documentos(pregunta)
 return llamar_llm(build_prompt(pregunta, docs))

The call tree is captured automatically: the trace shows pipeline_rag as the root span, with buscar_documentos and llamar_llm as children, nested. Without writing a single with tracer.start_as_current_span(...) by hand.

In TypeScript the equivalent is modular: you install @langfuse/tracing, @langfuse/otel and @opentelemetry/sdk-node, and you can use TS decorators, context managers or manual spans, the three models interoperate. The consequence: third-party libraries that emit OTel spans (openai, @anthropic-ai/sdk, Vercel AI SDK instrumentations) show up in Langfuse with no extra work.

Self-host architecture: designed for serious production

The Langfuse backend architecture has two explicit decisions that set its self-host deployment apart:

Persistence in S3/Blob Storage first. When a tracing event arrives, it is persisted in object storage before touching the database. Only when the later processing confirms OK is it inserted into Postgres/Clickhouse. If the DB goes down temporarily, the events are not lost; they stay in S3 waiting to be reprocessed. For production where losing the traces of an incident is equivalent to losing evidence, this is load-bearing.
Long migrations as background jobs. Schema upgrades that on other platforms mean a downtime window run in the background on Langfuse while the application keeps serving. Upgrade downtime drops drastically.

The officially supported deployment modes:

Docker Compose: for development and POCs. One command, everything up.
VM: a single node, containers, no orchestration. For small environments.
Kubernetes with Helm: the recommended mode for production. Official chart maintained. Supports external Postgres, external Clickhouse, external S3, HPA.

The external dependencies in a typical production setup: Postgres (metadata, prompts, configuration), Clickhouse (tracing events, high-cardinality queries), S3 or a compatible blob store (pending events), Redis (the queue between components). Yes, that is several pieces; it is what holds up the durability and the scale.

Prompt management as a first-class citizen

What sets Langfuse apart from platforms focused only on tracing is that the prompts live in Langfuse, not in the application repo or in spreadsheets. Each prompt has:

Name and version (v1, v2, v3…). Changing the prompt does not require redeploying the app: the app asks the SDK for the prompt, and the SDK caches it and refreshes when there is a new version.
Typed variables: {{user_input}}, {{context}}. Rendering with validation.
Tags and labels: by environment (production, staging), by team, by experiment.
Client and server cache: the SDK caches locally with a configurable TTL, avoiding a roundtrip to Langfuse on every call.
Linkage with traces: each trace records which exact version of which prompt was used. Investigating “this answer came out wrong” leads to prompt version Y, not to “some version of the prompt at some point”.

from langfuse import get_client

langfuse = get_client()

prompt = langfuse.get_prompt("rag-system-prompt", version=3)
# or by label: langfuse.get_prompt("rag-system-prompt", label="production")

compiled = prompt.compile(context=docs_text, user_input=question)
# 'compiled' is the final string, ready to send to the LLM

For teams iterating on prompts daily, this is what avoids the chaos of “which version of the prompt is actually running in production right now”.

Evaluations: four combinable evaluation models

Langfuse covers the four patterns of answer evaluation:

LLM-as-a-judge: you configure a model (typically GPT-4 or Claude) with a rubric and it evaluates each answer. Result: a numeric score (0-1) and a justification. Applicable to automatic tracing (all answers) or batch (a dataset selection).
User feedback: the app lets the user mark an answer as good/bad. The feedback is associated with the trace and the prompt version, which lets you see which versions have a worse rate.
Manual labeling: a UI where human labelers score answers. Useful for golden datasets and for evaluating the judge.
Custom evaluators through the API/SDK: your own evals (a unit test, a business metric) report a score through the API. They integrate with CI.

Combined, they give regression testing of the prompt: you change from v3 to v4, evaluate the golden dataset with LLM-as-judge, compare; if v4 gets worse in any of the segments, the merge fails.

Integrations

Langfuse does not compete with OpenLLMetry, LangChain or LiteLLM: it integrates them. The ones that are tested and documented:

OpenTelemetry: any OTel instrumentation emits to Langfuse over OTLP.
LangChain and LangGraph: a native callback that captures the whole chain.
LlamaIndex: native callback.
OpenAI SDK (Python and TS): a wrapper that adds tracing automatically.
LiteLLM: integration as a callback, which covers 100+ providers through LiteLLM.
OpenLLMetry / Traceloop: they emit to Langfuse like any OTel backend.
MLflow: through an OTel exporter from MLflow to Langfuse.
Vercel AI SDK: native instrumentation.

The strategy is clear: Langfuse is a backend, not an SDK. Your team chooses how it instruments; Langfuse accepts any path. The operational consequence: moving from Langfuse to another OTel backend tomorrow is viable.

When Langfuse is not the answer

So as not to present it as a silver bullet:

If you only use LangChain and have no resources to self-host: LangSmith will give you a smoother integration (it is the same team).
If your only need is a proxy with cost tracking and no evals: Helicone is simpler.
If you want an integrated commercial vendor solution: Datadog LLM Observability, New Relic AI Monitoring or Dynatrace AI are Enterprise alternatives with 24/7 support.
If your load is pure batch massive inference with no agents: you probably do not need semantic tracing; Prometheus + Grafana with OTel metrics is enough.

For everything else, your own apps with serious tracing, multi-tenant with quotas, teams iterating on prompts daily, RAG with continuous evaluation, Langfuse is the safe bet.

Quick choice summary:

LangChain → LangSmith (zero effort, automatic instrumentation).
Your own multi-framework applications with OSS → Langfuse (MIT, self-host, complete).
RAG with vector stores → Arize Phoenix (better retrieval visibility).
Simple proxy, low budget → Helicone.
Strict vendor neutrality → OpenLLMetry/Traceloop.
Pydantic AI → Logfire (same team).

Strengths and weaknesses of the instrumented model

Strengths:

Enormous depth: nested spans with all the context (chain steps, retrieval, embeddings, tool calls).
Semantic vocabulary: the SDK knows the domain (LLM, vector store, agent).
Maturity: three years of evolution, a rich ecosystem, ready-made dashboards.
Built-in evals: the top platforms combine tracing with evaluation (judge LLM, datasets, regression).

Weaknesses:

Requires control of the code: if you cannot instrument, it does not work.
Trust in the app: if the app reports badly or has a bug, so does the trace. It is not tamper-proof.
Coupling to the SDK: version changes in a library can break the instrumentation.
Uneven coverage: Python SDKs are mature; Go, Rust and JS are younger.

The zero-instrumentation approach: AgentSight

AgentSight is the project from the eunomia-bpf group that champions the opposite approach. Its paper on arxiv (2508.02736), presented at the Workshop on Practical Adoption Challenges of ML for Systems, formalises the proposal. The premise is direct:

Instead of instrumenting the agent, observe it at the system boundary.

And “system boundary” means the kernel boundary: the last point before a piece of data leaves the process towards the network or the filesystem. There, with eBPF, things are seen as they are, without the application being able to cooperate in hiding them.

Architecture: three planes

AgentSight assembles three layers:

Plane 1 — SSL/TLS uprobes. eBPF can attach programs to functions in userspace libraries (uprobes). The target functions are the encryption ones: SSL_write, SSL_read from OpenSSL/BoringSSL, and their equivalents in Rustls. AgentSight puts hooks on them that capture the arguments: the plaintext buffer the app passes to be encrypted, just before TLS processes it. On reception it does the symmetric thing: a hook after SSL_read with the freshly decrypted plaintext. Result: AgentSight sees the complete content of any HTTPS request the app makes without needing a man-in-the-middle, certificates or traffic decryption. The payload is plaintext because it was captured before being encrypted.

This works because uprobes are cheap (around 100 ns per invocation) and because apps use common TLS libraries. The few apps that implement their own TLS (rare in production) escape this hook; for those you need a different kprobe or manual instrumentation.

Plane 2 — Kernel events. In parallel, AgentSight observes relevant syscalls through tracepoints: execve (which processes start), connect/accept (network), read/write with file descriptors (filesystem and stdio), unlink, clone. Any action by the agent that has an effect outside the process passes through here. This covers, among other things, shell commands executed by the agent: if a Claude Code agent decides to run rm -rf to “clean up the project”, the execve is seen even if the LLM API does not report it.

Plane 3 — Correlation engine. The two previous planes produce streams of asynchronous events. AgentSight has a userspace component that correlates them causally cross-process: an outgoing HTTP request with bash -c rm -rf can be correlated with the previous LLM response that suggested it, through PIDs, timings and heuristics. The paper mentions the optional use of a secondary LLM (Anthropic Claude, for instance) that analyses the sequence of events and produces semantic alerts: “the agent responded with a tool call that was not in the whitelist”, “the reasoning chain has gone 47 iterations without converging”.

`stdiocap`: capturing the stdio of local MCP servers

One specific piece that deserves its own mention is stdiocap, a separate BPF tool included in the repo. The Model Context Protocol (MCP), popularised by Anthropic in 2024 and mainstream in 2025-2026, has two transport modes: HTTP/SSE (network) and stdio (between the client and the server it starts as a subprocess). Local MCP servers, the ones that run on the same machine and are started by the client as children over pipes, communicate through stdin/stdout/stderr with JSON-RPC.

stdiocap hooks read/write/dup on the stdin/stdout/stderr file descriptors of a target process and records all the JSON-RPC traffic between the MCP client and server. It is the same idea as the SSL capture but for stdio: you observe the conversation without either the client or the server knowing. Typical use case: seeing which tools of the filesystem-mcp MCP server a Claude Code agent has invoked in the last hour, what arguments it passed, what errors it received. Impossible with classic instrumentation (MCP servers are usually third-party binaries).

Guarantees: tamper-proof, kernel safety, <3% overhead

Three properties make AgentSight interesting for production:

Tamper-proof: the observation happens in the kernel (uprobes, syscalls). A malicious or compromised application cannot falsify what is seen. Compare with instrumentation: if the agent decides not to emit the span for its action, it does not appear in Langfuse. Here it has no choice.
Kernel safety: eBPF formally verifies that programs terminate and respect bounds checks. It cannot crash the kernel. The same as in the rest of the eBPF series.
<3% CPU overhead measured on real agent workloads (per the paper). The number compares favourably with SDK instrumentation, which typically adds 5-10% in intensive applications.

What it detects out of the box

The paper and the documentation highlight three classes of detection:

Real-time prompt injection: the correlation engine can apply rules or a detection model over the plaintext captured by the SSL uprobes. If the prompt contains suspicious patterns, “ignore all previous instructions”, a system prompt embedded in a user input, instructions to exfiltrate data, it raises an alert.
Reasoning loops that burn resources: agents that enter infinite loops calling tools without progressing. Detectable because the causal chain does not converge to a “final answer” and the tokens pile up. The correlation engine flags them.
Bottlenecks in multi-agent setups: when several agents coordinate, AgentSight sees the communication matrix between all of them and can detect agents that block while waiting, deadlocks, excessive fan-out.

The clash and the coexistence

The two families look like competitors, but in reality they see different things and complement each other in production.

What only the instrumented one sees

Internal agent variables that never reach the wire: the intermediate state of a LangChain chain, the values before passing them to a tool, how a prompt is built from a template with internal vars.
Deep semantic spans: retrieval > embed > vector_search > rerank > format_context > prompt_template > llm. AgentSight sees only the final call to the LLM; the path taken to build it is invisible.
Evaluations: scoring of answers, judge LLMs, quality regression. This lives only in instrumented platforms.

What only eBPF sees

Opaque binaries: Claude Code, Cursor, Gemini CLI, third-party agents. You do not have the code; you cannot instrument them. Only eBPF sees them.
System-level actions: the agent decides to run git push --force or kubectl delete. The action is seen in the execve. The agent’s instrumentation may not report it (especially if it was a command the agent generated as output without going through an explicit “tool”).
Tamper-proof audit: for regulatory compliance (HIPAA, SOC2, NIS2), having observation the app cannot dodge has formal value. eBPF provides it.
Local MCP servers over stdio: invisible to classic instrumentation unless each server emits its own spans (rare).

What both see, complementarily

Prompts and completions: the instrumented approach emits them with rich metadata; eBPF captures them off the wire. A perfect cross-check for spotting discrepancies.
Calls to external APIs: APM flags it; eBPF confirms it at kernel level.
Latency: APM per span; eBPF measures RTT at TCP level and network connectivity.

Decision matrix

Case	Instrumented	eBPF (AgentSight)
Your own app with LangChain	Yes, first	Optional
Your own multi-framework app	Yes	Optional
Third-party binary (Claude Code, Cursor)	Does not work	Yes, the only path
Tamper-proof regulatory compliance	Insufficient	Yes, required
Zero-trust multi-tenant	Insufficient	Yes, required
Local MCP servers (stdio)	Difficult	Yes, with stdiocap
Answer quality evaluation	Yes, required	No (out of scope)
Depth of the internal chain	Yes, required	No (a black box for AgentSight)
Reasoning loop detection	Possible with plumbing	Yes, built in
Real-time prompt injection	Possible (post-processed)	Yes, in stream

The natural conclusion: for your own apps, instrumented; for opaque binaries or compliance, eBPF; for everything that matters, both.

2026 reference architecture

Four recipes that cover the bulk of real cases:

Setup A — Your own application with LangChain or similar

Needs: depth, evals, a team comfortable with SDKs.

Langfuse self-host or LangSmith cloud as the backend.
OpenLLMetry SDK or LangSmith SDK instrumenting the code.
OpenTelemetry Collector between the app and the backend for routing flexibility (to Langfuse + Tempo + Loki, for instance).
Hubble for the network layer in the cluster (inter-pod latency, drop attribution).

Setup B — Productionising an opaque binary (Claude Code, Gemini CLI)

Needs: observe without touching, audit, control cost.

AgentSight deployed as a DaemonSet over the cluster (or standalone on the node).
Grafana with dashboards fed by AgentSight’s metrics.
An OTLP exporter from AgentSight to an OTel backend (Tempo, Jaeger). The spans will use the GenAI semantic conventions once they are fully standardised.
Tetragon optionally for policy over what the agent is allowed to run (Sigkill if it tries rm -rf or similar).

Setup C — Zero-trust multi-tenant platform

Needs: agents from different clients running on the same cluster, mandatory audit, nobody trusts anybody.

AgentSight as the tamper-proof audit layer. Compliance requires it.
Langfuse multi-tenant for the clients that do instrument.
Tetragon with TracingPolicyNamespaced per tenant (different policies per namespace).
Hubble with persistent flow logs for forensics.
Cilium NetworkPolicy to isolate tenants from each other on the network.

Setup D — A local MCP server on a workstation

Needs: seeing what an agent does with a stdio MCP server.

AgentSight stdiocap pointed at the PID of the client or the server.
Complete JSON-RPC capture to a file or to an OTLP endpoint.
Visualisation: Grafana, or simply jq over the log.

A real use case: if you are integrating an MCP server of your own and want to see what tool calls a Claude Code or Cursor agent makes to your server, stdiocap is the cleanest way. You need to modify neither client nor server.

Operational traps

Sensitive data in prompts (instrumented)

By default, Langfuse, LangSmith and similar tools capture the complete content of prompts and completions. If your app processes PII, secrets or medical data, that goes to your observability backend. Configuring redaction or content opt-out before going to production is mandatory. OTel GenAI has specific flags (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false) to avoid it.

Sensitive data in prompts (AgentSight)

The same problem, worse: AgentSight captures literally what goes to the wire, in plaintext. If the agent talked to api.openai.com with a prompt containing sensitive data, AgentSight has that plaintext. You have to encrypt or redact before storing.

Pinned certificates or non-standard TLS

Some high-security apps do certificate pinning or use unconventional TLS implementations (Go’s crypto/tls, custom BoringSSL). In those cases, the uprobes on libssl do not cover them. AgentSight detects when it cannot observe and reports the gap; you still have to add specific hooks for the alternative SDK.

Token volume and storage

An application with medium traffic can generate millions of tokens a day. If you store them all in Langfuse or Phoenix with long retention, the database grows fast. Strategies: aggressive sampling, short retention for normal sessions and long retention only for errors/anomalies, content redaction and keeping only metadata.

Tracing with sampling and consistency

To cut cost, many installations sample: only 1 in every N traces is persisted. Be careful with inconsistent sampling: a trace can carry several spans across multiple services, and if the sampling decision is taken per span, you end up with incomplete traces. OTel has head sampling (in the SDK, at the start), which is consistent, and tail sampling (in the collector, at the end), which allows finer rules. For LLM, tail sampling is ideal: sample everything, discard only the “normal” traces and keep the ones with errors, high latency or high cost.

Multi-agent and trace propagation

When agent A calls agent B, you have to propagate the trace context (W3C Trace Context headers) so it shows up as a single tree. If you do not, you see two disconnected traces. Modern platforms do it automatically with inject/extract, but if your transport between agents is custom (over Redis pub/sub, over a DB), you have to propagate by hand.

The cost of uprobes on critical libraries

Hooking libssl adds around 100 ns per invocation. On extreme TLS traffic loads (tens of thousands of connections/s per core), that adds up. AgentSight keeps it under 3% on typical agent workloads (which are chatty but not networking-intensive). If your use were sniffing all the HTTPS on the node, it could hurt more.

What we have not covered (next series)

Evals: the next layer after tracing. Phoenix, Langfuse, LangSmith and company offer answer evaluation (judge LLM, datasets, regression). It is a world of its own.
Guardrails and safety: NeMo Guardrails, Llama Guard, Llama Prompt Guard, evaluators specific to prompt injection and jailbreaks.
Deep MCP server observability: how the OpenTelemetry GenAI conventions are being extended to MCP servers for trace-aware tools.
eBPF + on-device inference: when the LLM runs locally through vLLM or llama.cpp, the uprobes can see the output token queue BEFORE it goes to the client. New territory.
Statistical analysis of agent flows: detecting drift, outliers, patterns that indicate degradation.

Closing the eBPF series

This series of four articles has travelled eBPF from first principles to the 2026 frontier:

eBPF from zero to Cilium — what eBPF is, networking hooks, how Cilium skips the TCP/IP stack, BGP Control Plane v2.
Tetragon: runtime security — observability and enforcement of processes in the kernel.
Hubble: network observability — L3-L7 flow logs and the frontier with AI agents.
This one — AgentSight, LLM tracing, instrumented vs zero-instrumentation.

If you have got this far you have the map to sit down with a platform, security or AI team in 2026 and recognise what each piece does, what problem it solves and where to start. That whole stack, Cilium for CNI and BGP, Tetragon for runtime security, Hubble for network observability, AgentSight for AI agents, sharing eBPF as a common substrate, Cloud Native governance and OpenTelemetry vocabulary. It is the clean architecture the industry asked for a decade ago and that finally exists.

References

AgentSight:

AgentSight GitHub (eunomia-bpf) — the project.
AgentSight: System-Level Observability for AI Agents Using eBPF (arxiv 2508.02736) — the formal paper.
AgentSight ACM workshop publication.
AgentSight blog post (eunomia.dev) — accessible description.

OpenTelemetry GenAI semantic conventions:

Instrumented platforms:

Langfuse — MIT, self-host + cloud.
LangSmith — LangChain team.
Arize Phoenix — OSS, OTel-native.
Helicone — simple proxy.
OpenLLMetry (Traceloop) — Apache 2.0, OTel SDK.
Pydantic Logfire — AI observability.

2026 comparisons:

Cross-references from the series:

eBPF from zero to Cilium.
Tetragon: runtime security.
Hubble: network observability.
LLM inference series: KV cache, vLLM on Kubernetes, PagedAttention, LLM Operators on K8s.

Hubble: eBPF network observability, state of the art 2026 and the new frontier with AI agents

Tue, 19 May 2026 06:00:00 +0200

TL;DR

Hubble is Cilium’s native network observability, built on the same eBPF programs Cilium uses for enforcement. It does not duplicate the datapath or instrument the kernel its own way: it listens to the hooks Cilium already has and produces structured flow logs with Kubernetes context included, covering pod, namespace, labels, service, policy verdict and L7 payload where applicable. It is what happens when someone decides that tcpdump with grep does not scale to 10,000 pods and builds a distributed system of their own (a Hubble server per node, plus Hubble Relay as the aggregator, plus a CLI and a UI) with practically zero overhead, because the capture was already happening. In 2026 it is at version 1.19.3 (April 2026), with Cilium 1.19 marking the project’s tenth anniversary; IP options tracing has arrived, along with filtering by encryption status, the drop event tagged with the exact NetworkPolicy that caused it (direct attribution), a stabilised field mask API, and the first wave of ML anomaly detection applied to flows for predictive security in IoT/5G clusters. And, most interesting of all for 2026: a new frontier appears where the same eBPF observes AI agents such as Claude Code, Gemini CLI and MCP agents, intercepting SSL/TLS and stdio without instrumenting the code, which turns the Cilium + Hubble + Tetragon + AgentSight stack into a complete set of tools for understanding what an agentic system does inside a cluster.

This article is part 3 of the eBPF series. Part 1: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack. Part 2: Tetragon: Cilium’s security cousin that sees every syscall in the kernel. Here we complete the observability quadrant: network with Hubble, process with Tetragon, AI agent with AgentSight.

The analogy: tcpdump that speaks Kubernetes

If you have administered networks over the last twenty years, tcpdump and Wireshark have been your daily bread. They capture packets on an interface, parse them, and let you filter with tcp.port == 443 and host 10.0.0.5. They work, they have been working since the 90s, and they are the first thing you open when something smells wrong.

Now stick tcpdump on a Kubernetes cluster with 10,000 pods. The problems show up in order:

One tcpdump session per node. You wanted to “see the traffic between the frontend and the API”; you need SSH to every node, tcpdump on every NIC, synchronised timestamps, manual aggregation.
There is no K8s context. You see a packet from 10.244.5.7 to 10.244.8.42. Which pod was that? Which namespace? Which label? You have to correlate with kubectl get pod -A -o wide every single time.
No L7 understanding. You see a POST to HTTPS, and you cannot know the method and path because it is encrypted on the wire. If there is mTLS between pods, worse.
High cost: full packet capture with a copy to userspace slows down the datapath. Under dense traffic, you notice it.

Hubble is tcpdump redesigned for all of that. It reuses the eBPF programs that are already processing every packet (Cilium puts them there for enforcement) and, while they take their allow/deny decision, they emit a flow event with all the context: source and destination pod identity, namespace, labels, protocol, verdict, and, if Cilium has done L7 parsing via Envoy, the HTTP method, path, status code, DNS query, Kafka topic. That event travels through a ringbuffer to userspace, it is received by the Hubble server that lives inside the node’s Cilium agent, and it is exposed over gRPC. A separate service, Hubble Relay, aggregates the streams from all nodes and gives you a single cluster-wide API. On top of that: a CLI (hubble) and a web UI with a real-time service graph.

Zero additional copies. Zero duplicated parsing. And the result is understandable by anyone who knows what a Pod is.

Architecture: four pieces visible from outside

Hubble is made up of four logical components, all optional depending on what you want to do:

1. Hubble Server (embedded in every Cilium agent)

It lives inside the Cilium agent process (it is not a separate binary). Each node exposes a gRPC endpoint locally on the Unix socket /var/run/cilium/hubble.sock. The server listens to the events the eBPF programs emit to the ringbuffer, enriches them with Kubernetes metadata (which the agent already has in memory), and makes them available to consumers.

Enabling it: --set hubble.enabled=true in Cilium’s Helm chart. By default, the server is only accessible locally; if you want to consume it from another node, it has to be exposed (which is what Hubble Relay does).

2. Hubble Relay (aggregator)

It is a separate Deployment (typically 1 replica, scalable) that connects to every Hubble server in the cluster and aggregates their streams into a single API. When your CLI or UI asks for “the last 1000 flows in the cluster”, the Relay collects them in parallel from all nodes and returns the union.

Enabling it: --set hubble.relay.enabled=true. Without the Relay you only see the traffic of the node you are connected to, which is useful for local debugging but not for a cluster-wide view.

3. Hubble CLI (`hubble`)

A Go binary that speaks gRPC with the Relay (or with a local Hubble server). It supports two main modes:

hubble observe: a real-time stream of flows, with very expressive filters (by namespace, pod, port, verdict, protocol, label).
hubble status: the state of the Hubble cluster (how many nodes connected, lag, flow rate).

And the equivalent of tcpdump’s pcap dump: hubble observe --output jsonpb > flows.json to process afterwards with jq or other tools.

4. Hubble UI

A web frontend that connects to Hubble Relay and shows:

A service graph in real time (which Pod talks to which Service, which protocols it uses, which verdict).
A filterable list of flows.
L7 details where there are any (HTTP method/path/status, DNS query/response).

Enabling it: --set hubble.ui.enabled=true. Useful for presentations to non-CLI teams; it does not replace the CLI for serious debugging.

What you see: the Hubble flow log from the inside

A Hubble flow in JSON format looks roughly like this (simplified):

{
 "time": "2026-05-19T03:12:45.182Z",
 "verdict": "FORWARDED",
 "source": {
 "ID": 5482,
 "identity": 24871,
 "namespace": "prod-api",
 "labels": ["app=checkout", "team=payments"],
 "pod_name": "checkout-7c9f-x8j2",
 "workloads": [{"name": "checkout", "kind": "Deployment"}]
 },
 "destination": {
 "ID": 12041,
 "identity": 18356,
 "namespace": "prod-db",
 "labels": ["app=postgres", "tier=primary"],
 "pod_name": "postgres-0"
 },
 "Type": "L3_L4",
 "l4": {
 "TCP": {
 "source_port": 41982,
 "destination_port": 5432,
 "flags": {"SYN": true}
 }
 },
 "node_name": "rke2-worker-03",
 "Summary": "TCP Flags: SYN"
}

When L7 parsing is active (via embedded Envoy or Hubble’s lightweight parser), the same flow adds:

"l7": {
 "type": "REQUEST",
 "http": {
 "code": 200,
 "method": "GET",
 "url": "/api/v1/cart/items",
 "protocol": "HTTP/1.1",
 "headers": [{"key": "user-agent", "value": "checkout/1.4.2"}]
 }
}

The protocols natively supported for L7 parsing:

HTTP/1.1 and HTTP/2 (including gRPC over HTTP/2).
DNS (queries and responses, with domains, types, response codes).
Kafka (topics, API keys).
TLS handshake (SNI, not the encrypted payload by default).
MySQL, Cassandra (with optional modules).

For HTTP and gRPC, Cilium can enable the embedded Envoy proxy for the flows you want to inspect (not all of them; it is selective via CiliumNetworkPolicy with L7 rules). Without Envoy there is lightweight parsing, but less detailed.

Verdict and drop attribution

Every flow has a verdict: FORWARDED, DROPPED, ERROR, AUDIT, REDIRECTED, TRACED, TRANSLATED. For the DROPPED case, Hubble includes a structured reason (drop_reason) and, since Cilium 1.19, the exact NetworkPolicy that caused it.

That last point changes day-to-day operations. Before, when a pod could not talk to another one, the debug flow was:

See the dropped flow in Hubble.
Look at every CiliumNetworkPolicy in the namespace.
Reason manually about which one, with which labels, is blocking it.

With Cilium 1.19’s attribution, the policy_match_info field tells you directly “it was dropped by the frontend-egress policy, rule 3”. You go from “Sherlock Holmes for 20 minutes” to “kubectl get -o yaml of that specific policy”.

Prometheus metrics and Grafana dashboards

Hubble also exposes aggregated metrics in Prometheus format, separate from the gRPC flow stream. Enabling it: --set hubble.metrics.enabled=true (Helm) plus a list of the set you want to export.

The usual metric groups:

flow: total flows by verdict, source/dest, protocol.
http: requests by method, response code, latency (histogram).
dns: queries, response codes, top-N domains.
tcp: handshakes, retransmissions, congestion window.
drop: drops by reason, with NetworkPolicy attribution.
port-distribution: histogram of active ports.
policy: hits by policy and verdict.

These metrics have rich K8s labels (source_workload, destination_workload, namespace, and so on) that make them pivotable in Grafana. There are prebuilt dashboards on Grafana Labs covering the common cases; importing one and having immediate visibility costs five minutes.

Cost: metrics with many K8s labels can blow up cardinality in Prometheus. For large clusters (>1,000 pods), it is worth reviewing which set you export and using drop rules in Prometheus to keep a lid on it.

Deployment: Helm on one screen

The canonical Cilium install with full Hubble:

# values.yaml
hubble:
 enabled: true
 metrics:
 enabled:
 - dns:query;ignoreAAAA
 - drop
 - tcp
 - flow
 - port-distribution
 - icmp
 - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
 serviceMonitor:
 enabled: true # auto-discovered by kube-prometheus-stack
 relay:
 enabled: true
 rollOutPods: true
 ui:
 enabled: true
 rollOutPods: true
 ingress:
 enabled: true
 className: cilium
 hosts:
 - hubble.example.local

And the install itself:

helm upgrade --install cilium cilium/cilium -n kube-system -f values.yaml

After installing, validate with cilium status (the Cilium CLI) that the Hubble section shows OK, and try your first flow with:

cilium hubble observe --namespace prod-api --pod checkout-7c9f-x8j2

State of the art in 2026

Cilium 1.19 was released in February 2026, marking the project’s tenth anniversary. Hubble reached version 1.19.3 on 22 April 2026. The relevant news:

Direct attribution of drops to a NetworkPolicy

Already covered above; it is probably the most valuable operational change in the release. Any dropped flow carries the name, namespace and specific rule of the policy responsible. It is also available via Prometheus metrics, which allows alerts of the form “policy X is dropping >N requests/second”.

Tracing with IP options

Hubble can now trace individual packets with IP options enabled. It is a mechanism similar to traceroute, but at L3: you put a mark on the packet and Cilium reports it every time the packet crosses a node or an eBPF decision. Useful for debugging multi-cluster paths, fabric mesh, or NetworkPolicies applied at different layers.

Filtering by encryption status

A new CLI flag: hubble observe --encryption-status=encrypted (or unencrypted). Useful for validating deployments with WireGuard or IPsec enabled pod-to-pod: you confirm that the traffic that should be encrypted is, and you spot regressions quickly.

Hubble field mask API stabilised

The field_mask lets you ask for only the parts of the flow you care about, hugely reducing bandwidth and processing when all you need is, say, source/dest and verdict. It used to be experimental; now it is stable and it is default-on in the CLI.

AI-driven anomaly detection (predictive security)

This is the most talked-about addition of 2026. Cilium 1.19 adds hooks so that an external consumer, typically an ML system, can process flows in streaming and detect statistical anomalies: pods that suddenly talk to new destinations, latency spikes on an API, odd DNS sequences. The detection part happens outside the Cilium agent (nobody wants heavy ML in the datapath), but Cilium exposes the flows with the pre-computed features the model needs. The published use cases focus on IoT and 5G, where traffic is high in volume and low in variety, ideal conditions for anomaly detection.

Scaling to 10,000+ pods

Cilium 1.19 has done serious work on scalability: Hubble Relay can now aggregate streams from hundreds of nodes without saturating; the default field_mask reduces inter-node bandwidth; and flows can be sampled under high load if your use is statistical analysis rather than forensic debugging.

Cilium 1.20 in development

Cilium 1.20 is in the development branch. The most relevant bits for Hubble:

Unification of preferIpv6: the hubble.preferIpv6 flag was deprecated in favour of the global preferIpv6 that applies to every Cilium component.
tetragon-python SDK: although it belongs to Tetragon, not Hubble, it sets a trend, namely eBPF policies written in Python instead of YAML. Hubble will probably follow a similar path.

The new frontier: eBPF and AI agents

So far, the classic Hubble content. But there is a 2026 twist worth covering, because it closes the loop with the other series on this blog.

When a Kubernetes cluster starts running AI agents, such as Claude Code, Gemini CLI, or LangGraph-based agents calling APIs and MCP servers, the observability problem changes shape. It is no longer enough to know “which pod talked to which pod” (that is Hubble) or “which process ran what” (that is Tetragon). You need to know:

Which external APIs the agent is calling and with which prompts.
Which MCP tools it is invoking, with which arguments.
How many tokens it consumes, which model it picks, how much it costs.
Whether the agent drifts from expected behaviour (out-of-policy queries, jailbreak attempts, secret leakage).

The traditional solutions, instrumenting the agent’s code with OpenTelemetry or parsing structured logs, do not work well when the agent is a third-party binary (Anthropic’s Claude Code, Google’s Gemini CLI) or when the MCP servers live in other languages with stdio as their transport.

AgentSight: zero-instrumentation for LLM agents

AgentSight (a project from the eunomia-bpf group, the same ecosystem behind several high-profile eBPF runtimes) attacks this problem with the same philosophy as Hubble: do not instrument; listen. It puts eBPF hooks at two critical points:

uprobes on SSL/TLS libraries (libssl, boringssl, rustls). It captures the plaintext before encryption on send and after decryption on recv. For an HTTP call to https://api.anthropic.com/v1/messages, AgentSight sees the full JSON of the prompt and the response without decrypting anything in transit, simply because it has reached the syscall level before the TLS layer does its work.
stdiocap BPF: it captures read, write and dup on a process’s stdin/stdout/stderr file descriptors. This is what makes it possible to observe MCP servers that speak stdio with their client, the usual pattern for local MCP servers. You capture the JSON-RPC going back and forth without either the client or the server knowing.

Reported overhead: <3% CPU, comparable to Hubble in its own regime.

How it fits with Hubble and Tetragon

The three complement each other cleanly:

Hubble tells you: “the agent’s pod opened a TCP connection to api.anthropic.com:443 with verdict ALLOW”.
Tetragon tells you: “the claude-code process with PID 1843 did a connect() to that IP” (plus the binary, the arguments, the pod namespace).
AgentSight tells you: “the HTTPS content of that connection was a prompt messages=[{role:'user', content:'analyze this repo and modify the firewall config'}] and the response included a tool call to read_file with argument /etc/passwd”.

It is the difference between flow, process and semantics. For a security team that wants to watch AI agents in production, all three are necessary. For anyone who wants to understand cost, all three are useful (Hubble for network latency, Tetragon for resource usage, AgentSight for tokens and chosen model).

Emerging use cases

The patterns consolidating in 2026:

Agent audit trail: recording every LLM call and every tool call for compliance, especially in regulated sectors.
Jailbreak and prompt injection detection: applying rules over the prompts captured by AgentSight (similar to Tetragon’s TracingPolicy, but over semantic content).
Cost accountability: seeing which team/agent consumes which tokens, without instrumenting.
Replay and debugging: reproducing an agent’s reasoning in production without asking it to run again (which is non-deterministic).

It is a young field, AgentSight is months old rather than years, but the “eBPF as zero-instrumentation observability” pattern is very clearly spreading beyond network and process. The coming year will see consolidation and, probably, native integration with Hubble.

Common Hubble use cases

Back to Hubble proper, the cases in which any organisation deploys it:

1. NetworkPolicy debugging

The classic use: “this pod cannot reach this Service”. Without Hubble, it meant SSH, tcpdump, comparing rules. With Hubble:

hubble observe --from-pod prod-api/checkout --to-pod prod-db/postgres --verdict DROPPED

If there are drops, you see the policy responsible (Cilium 1.19+). If there are no drops, the problem is not policy: it is DNS, routing or the target service.

2. Auditing inter-namespace communication

For compliance: validating that isolated namespaces are not communicating against what was declared.

hubble observe --from-namespace prod-payments --to-namespace 'NOT prod-db' --output json

3. Exfiltration detection

Outbound traffic to suspicious public destinations. Hubble detects them by IP/SNI, not by payload (which is encrypted):

hubble observe --to-fqdn 'NOT *.example.com' --to-fqdn 'NOT *.internal' --protocol tcp

Combined with Prometheus metrics and Grafana alerts, this gives you an exfiltration radar at zero cost.

4. Real-time service SLOs

The hubble:http:response_time_seconds metrics with source_workload, destination_workload, method and status_code labels allow SLO dashboards without needing to instrument the apps. The SRE sees the p95 latency of checkout → catalog directly.

5. Performance debugging

hubble:tcp:retransmissions_total and hubble:tcp:flags_total{flag="RST"} are early signals of network problems. A rise correlated with a latency regression points you at something in the infrastructure (NIC, switch, MTU) before you go down to investigate the app.

6. Post-incident forensics

Configuring Hubble to export flows to persistent storage (via OTLP to Tempo/Loki, or hubble observe --output jsonpb to S3) gives you forensic capability: if at T+30 days you detect that something was wrong at T, you can reconstruct the traffic.

Hubble and the rest of the observability stack

Hubble does not replace Prometheus, Loki, Tempo or Jaeger; it complements them:

Prometheus: receives Hubble’s aggregated metrics. Hubble exports a native Prometheus endpoint.
Loki: receives the structured flow logs if you export them as logs. Hubble has no native Loki exporter, but a Fluent Bit with an OTLP plugin or a custom one bridges it easily.
Tempo / Jaeger: the Cilium Operator has an OTLP exporter of flows in trace format (each HTTP/gRPC flow is a span). It integrates with Tempo or any other OTLP tracing backend.
Grafana: there are already public Hubble dashboards. Combined with Prometheus, Loki and Tempo, they give you a unified panel: metrics, logs, traces, all correlated by K8s labels.

The full-stack pile seen in production in 2026 (described in Building a Production eBPF Observability & Security Stack for Kubernetes in 2026):

Data: Cilium + Hubble (network), Tetragon (process), AgentSight (AI agent).
Pipeline: OTLP Collector as the single router.
Storage: Prometheus (metrics), Loki (logs), Tempo (traces).
UI: Grafana with domain-specific dashboards.
Alerting: AlertManager with rules over the Hubble + Tetragon metrics.

Comparison with the alternatives

System	Layer	Focus	Model
Hubble	L3-L7 network	K8s cluster with Cilium	eBPF, pull metrics, push flows gRPC
GKE Dataplane v2 obs	L3-L7 network	GKE managed	eBPF (Cilium-based, managed)
Tigera Calico Whisker	L3-L7 network	Cluster with Calico	eBPF + pcap, own UI
Tetragon	Process/syscall	K8s cluster	eBPF, push events gRPC
Falco	Process/syscall	K8s cluster	eBPF in userspace or kernel module
AgentSight	LLM agent	Agentic systems	eBPF (SSL uprobes + stdio)
Beyla (Grafana)	Application	App L7 + tracing	eBPF (uprobes on libs)
Pixie	App + system	Broad cluster visibility	eBPF + PXL script
Parca	CPU/mem profiling	Performance	eBPF profile sampling

If your CNI is Cilium, Hubble is the natural entry point and it does not compete with the rest: it complements them. For Calico clusters, Whisker is the equivalent. For profiling, Parca. For AI agents, AgentSight. The era of “one tool for everything” is passing: the modern stack combines several specialised pieces, all eBPF-based, exposed via OTLP.

Operational traps

Cardinality in Prometheus

Hubble metrics with all the K8s labels can blow up Prometheus. Measure cardinality before exporting everything. The most prolific metrics are flow and httpV2; start with drop and port-distribution and add the rest incrementally.

L7 visibility costs CPU

Enabling L7 parsing via embedded Envoy adds load to the agent (not to the base datapath, but yes to the node’s envoy proxy). For heavy HTTP traffic, measure. For flows where you only need L4, leave Envoy disabled.

Hubble Relay without HA

A single Relay replica is a single point of failure for the CLI and the UI (not for the local agent, which keeps working). For production, deploy with replicas: 2+ and topologySpreadConstraints so both do not go down together.

Encryption status reporting depends on the Cilium config

The new --encryption-status filter only gives real data if Cilium has encryption enabled (WireGuard or IPsec). Without that, everything is unencrypted and the filter contributes nothing.

UI exposed without auth

Hubble UI has no native auth. If you expose it through Ingress, there has to be authentication in front of it: OIDC via oauth2-proxy, mTLS, IP allowlist. It is not optional.

Storage that does not scale

If you keep flows for days for forensics, the volume is serious. For a cluster with 100 active pods, easily 1-10 GB/day of flow logs. Plan the lifecycle (compaction, retention, cold storage) before enabling it.

What we have not covered

Mesh / multi-cluster Hubble: aggregating flows from several Cilium clusters into a single Relay. Use case: a cross-cluster view, debugging a distributed service mesh.
hubble export: local persistence on the agent’s disk for forensics with low retention.
Anomaly detection with your own models: how to connect the gRPC stream to a custom ML consumer.
AgentSight in depth: the project deserves its own article. Next instalment.
eBPF for LLM serving profiling: how to measure vLLM’s TTFT, TPOT and throughput without instrumenting, using uprobes on libcudart.

References

Hubble and Cilium:

Hubble GitHub — main repo.
Hubble — Network Observability (Cilium docs) — official reference.
Cilium 1.19 release notes (InfoQ, feb 2026) — tenth anniversary and 1.19 news.
Cilium releases — all releases.
Hubble L7 HTTP Metrics — Grafana dashboard 19423 — ready to import.
End‑to‑end L7 Visibility with Cilium Hubble (cloud-cod.com, mar 2026).
Cilium Hubble Observability Platform Internal Analysis (Young-ju).
CiliumNetworkPolicy Python Hubble: L7 Visibility 2026 — one of the threads on the Python SDK.

State of the art 2026 and the full stack:

eBPF + AI agents:

AgentSight (GitHub eunomia-bpf) — the referenced project.
Harnessing eBPF for High‑Performance LLM Workloads (Klizo Solutions).

Cross-references:

Part 1: eBPF from zero to Cilium.
Part 2: Tetragon: Cilium’s security cousin.
LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators — where the traffic Hubble observes carries the prompts AgentSight inspects.

Tetragon: Cilium's security cousin that sees every syscall in the kernel

Tue, 19 May 2026 05:00:00 +0200

TL;DR

Tetragon is the runtime security and observability engine that the Cilium project published as a companion to the CNI. Its job is not to route packets, Cilium is already there for that, but to observe what happens inside the node’s processes in real time: which binary runs in each pod, which files it opens, which syscalls it invokes, which capabilities it asks for, which network connections it establishes, which kernel modules get loaded. It does this by loading eBPF programs into the kernel’s hook points (kprobes, tracepoints, uprobes, LSM hooks) and filtering the relevant events inside the kernel itself with a declarative language expressed as a CRD (TracingPolicy and TracingPolicyNamespaced). The result is a stream of events enriched with Kubernetes metadata (pod, namespace, labels) that costs less than 1% of CPU and, the thing that sets Tetragon apart from the competition, can block actions inside the kernel, killing the process with SIGKILL or overwriting a syscall’s return value, before they finish executing, with no race conditions. Against Falco (which parses syscalls in userspace, 5-10% overhead, detection-only), Tetragon is “cheaper and with enforcement”; against the bare kernel, it is “a declarative layer your operations colleague can read”. This article is the extensive introduction you need to take it on seriously: architecture, all the hooks and selectors, the operating modes, a guide to use cases (exec auditing, sensitive file access, container escape, cryptomining, rootkit detection, network observability) and the traps you see in production.

This article is part 2 of the eBPF series. Part 1, eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack, covered basic eBPF, the networking hooks (XDP, TC, sock_ops), how Cilium implements the datapath and the BGP Control Plane v2 CRDs. Here we take those same eBPF hooks and use them for something different: observing and, if needed, stopping what the cluster’s processes do.

The analogy: auditd on steroids in eBPF

Anyone who has spent a few years administering Linux has used auditd. It is the classic kernel subsystem for auditing syscalls: you configure a rule with auditctl (for instance, “monitor any open on /etc/shadow”) and the kernel sends events to a userspace daemon that persists them. It works, but it has two limitations that weigh heavily on modern Kubernetes clusters:

No Kubernetes context. auditd reports processes by PID and UID. Knowing which pod, which namespace, which image, which labels, the information that actually matters when responding to an incident, requires correlating afterwards with data from cri-o or containerd. It is operationally miserable.
No granular enforcement. auditd can generate events, but it cannot take the decision to kill the offending process before the syscall finishes. You leave that to a higher layer that reads the events, processes them and kills the process… if it gets there in time. A race by design.

Tetragon is auditd on steroids: the same conceptual ideas, hooks on syscalls and events to userspace, but implemented with modern eBPF, with filtering inside the kernel so you do not pay the cost of waking the daemon for every irrelevant syscall, with Kubernetes metadata injected by an agent that knows the cluster, and with actions executed inside the kernel itself without waiting for userspace to decide. If the rule says “kill any process that opens /etc/shadow from the prod namespace”, the decision is taken in the kernel kprobe and SIGKILL is delivered before the open completes. There is no race; there is no window between detection and action.

What Tetragon is, architecturally

Tetragon is an agent deployed as a DaemonSet (one pod per node) and a set of CRDs that define the policies to apply. The agent has four responsibilities:

Load eBPF programs into the hook points the active TracingPolicies demand.
Maintain a cache of Kubernetes metadata (pods, namespaces, labels) by reading the API server, so it can enrich every event with the right context.
Collect the events the eBPF programs emit (via ring buffers) and serialise them.
Export the events to configurable destinations: stdout JSON (typical in sidecars or log-collection agents), gRPC streaming (to consume them from Hubble or another consumer), a file, or Fluentd/Loki/SIEM.

The eBPF programs are not written by the user. Tetragon generates the bytecode from the TracingPolicies: it reads the declarative policy, decides which hooks to attack, which arguments to read from the kernel, which filters to apply inline and which actions to execute. The user only writes YAML.

This separation of declarative policy → generated eBPF bytecode is what makes Tetragon usable. Writing eBPF programs by hand is a specialist’s job; writing a TracingPolicy is the job of an SRE with a good example in front of them.

The two CRDs: TracingPolicy and TracingPolicyNamespaced

Tetragon exposes exactly two main CRDs:

TracingPolicy (cluster-scoped, cilium.io/v1alpha1): applies to the whole cluster, every node, every pod. Suitable for platform policies (the whole cluster must be audited the same way): for example, “log every execve in every pod” or “kill any process that tries to load a kernel module”.
TracingPolicyNamespaced (namespaced, same group and version): defined inside a namespace and applied only to the pods of that namespace. Suitable for policies with per-tenant autonomy: for example, “in the prod-payments namespace, kill any outbound connect to an IP outside the corporate range”.

Both CRDs have exactly the same internal structure. The difference is one of scope. The distinction was introduced precisely to allow multi-tenancy: the central security team defines cluster-wide TracingPolicy objects and each tenant can add its own with TracingPolicyNamespaced without needing cluster-admin permissions.

Anatomy of a TracingPolicy

A policy is made up of:

Hook points: which kernel events to observe.
Arguments: which data to read when the hook fires.
Selectors: filters evaluated inside the kernel to discard irrelevant events and, optionally, execute actions when they match.

Supported hook points

The official documentation lists five families of hook points:

kprobes: hook a kernel function. Syscalls are a particular case (when syscall: true) because their ABI differs from that of internal functions. Typical examples: sys_open, sys_openat, sys_connect, tcp_connect, do_mount, commit_creds. It is the most versatile hook and the one used 80% of the time.
tracepoints: hook static tracepoints compiled into the kernel. More stable across kernel versions than kprobes (they do not depend on function names that can change). Examples: syscalls/sys_enter_openat, sched/sched_process_exec.
uprobes: hook functions in userspace libraries or binaries. They serve to observe runtime primitives such as libssl functions, libc, the Go runtime, the JVM.
USDT tracepoints (User Statically Defined Tracepoints): static tracepoints defined in userspace binaries (like those MySQL, PostgreSQL and OpenJDK expose). Useful for application observability.
lsmHooks (LSM, Linux Security Module): hooks of the LSM subsystem, where SELinux/AppArmor plug in. They allow security policies very similar to traditional MAC but programmable with eBPF. Example: file_open, inode_unlink, socket_bind.

Arguments

Every hook can read the arguments of the function it is attached to. The supported types cover the primitives (int, uint64, bool, string, char_buf) and higher abstractions (file, path, sock, linux_binprm, capability, bpf_attr, cred). The high-level types are pointers to kernel structures that Tetragon knows how to parse; instead of having to read an offset, you write type: file and Tetragon gives you the full path of the descriptor’s file.

There is an important capability detail: on kernels ≥ 5.4, Tetragon can read up to 327,360 bytes of an argument if the large buffers flag is enabled. That is the difference between being able to audit execve with all of its long argv complete versus truncating them at 256 bytes and losing context.

Selectors: filtering in the kernel

Selectors are what make Tetragon cheap. Without them, every syscall on the node would fire an event that would travel kernel → ring buffer → agent → processed → filtered → discarded. With selectors, the filtering happens inside the eBPF program itself, in the kernel, and only the events that matter reach userspace.

The available selectors include:

matchArgs: filters by an argument’s value. Operators: Equal, NotEqual, Prefix, Postfix, GreaterThan, LessThan, Mask, SPort (source port), DPort (dest port), Family (AF_INET vs AF_INET6), State (socket state).
matchPIDs: filters by PID; useful for targeted observation.
matchBinaries: filters by the binary executing the syscall (absolute path), with Operator: In, NotIn, Prefix. Essential for avoiding noise from legitimate system processes.
matchNamespaces: filters by Linux namespace (Pid, Mnt, Net, Ipc, Cgroup, User). It allows policies specific to processes in containers versus the host.
matchCapabilities: filters by the process’s effective capabilities. Block actions requiring CAP_SYS_ADMIN that run in pods that should not have them.
matchNamespaceChanges: detects namespace changes (typical of container escape).
matchCapabilityChanges: detects capability changes (privilege escalation).
matchActions: the actions executed when all the preceding matchers hit.

Actions: from a simple Post to Sigkill

When a selector matches, an action is executed. Tetragon defines several:

Post: emits an event to userspace (the observability case). It supports rateLimit to avoid flooding the agent if the condition fires a thousand times per second. The syntax accepts 5 for 5 seconds, 5m for 5 minutes, 1h for 1 hour.
Sigkill: sends SIGKILL to the offending process from inside the kernel, before the syscall completes. This is the only thing that guarantees enforcement without a race.
Override: overwrites the syscall’s return value. Useful for making the process believe the syscall failed (Override -EPERM) without killing it. A better experience for apps that can handle errors; worse for apps that assume success.
Signal: sends any arbitrary signal (not just SIGKILL).
NoPost: does not emit an event, useful when combined with another selector that does emit and you only want the action without duplicated telemetry.
FollowFD and UnfollowFD: mark a file descriptor to follow its lifecycle and enrich subsequent events with the original path. Useful for auditing “which process read this file after opening it”.
TrackSock and UntrackSock: the same for sockets.
GetUrl and DnsLookup: make HTTP requests or DNS resolutions from the kernel. Designed for integrations with external systems (security webhooks, IP reputation lookups).
NotifyEnforcer and CleanupEnforcerNotification: communication with Tetragon’s enforcement subsystem for complex actions.

Modes: detection vs enforcement

A policy can be declared in one of two explicit modes:

enforce: enforcement actions (Sigkill, Override, Signal) are active. This is production.
monitoring: enforcement actions are ignored; only Post events are emitted. This is the “let us see what would happen if this were switched on” mode, critical for testing policies without breaking applications.

Control is done with the spec.options[].name: policy-mode field and value: monitoring or enforce. It is the best practice: start in monitoring, collect events for days, tune the selectors until no false positives come out, and only then switch to enforce.

Full example: blocking writes to `/etc/passwd` in the prod namespace

A realistic policy, commented line by line:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-passwd-write
 namespace: prod
spec:
 kprobes:
 - call: "fd_install"
 syscall: false # kernel function, not a syscall
 args:
 - index: 0
 type: int  # file descriptor
 - index: 1
 type: "file" # struct file*
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values:
 - "/etc/passwd"
 matchActions:
 - action: Sigkill  # kills the process
 rateLimit: "1m" # max once per minute
 options:
 - name: policy-mode
 value: enforce  # enforcement mode active

fd_install runs every time a process obtains a new file descriptor; the second argument is the file’s file struct. Tetragon knows how to resolve it to its absolute path. The matchArgs compares that path with /etc/passwd. If it matches, Sigkill kills the process before the descriptor even becomes usable. rateLimit: 1m stops the agent from saturating if a malicious application tries it in a loop.

Common use cases

Now to real use. These are the six cases that show up in any serious Tetragon deployment in 2026.

1. Execution auditing (`execve`)

The most basic use case and, even so, the most valuable. Which binaries are running in each pod? In a container that is supposed to run only nginx, suddenly seeing an sh or a wget is almost always a red flag.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-execve
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm  # struct linux_binprm*
 selectors:
 - matchActions:
 - action: Post  # events only, no enforcement

With no filters: every execve in the cluster generates an event. With K8s metadata, the event includes pod, namespace, container, image, labels. You turn it into a stream of events towards your SIEM and set up rules: “alert if I see sh, bash, nc, curl, wget or python running in any pod of the prod-api namespace”.

An enforcement variant: instead of Post, use matchBinaries with Operator: NotIn and a whitelist, plus Sigkill if the binary is not on the list. A very rigid box, but effective in pods that are “single-binary” (like a Go microservice).

2. Sensitive file access

Detecting (or blocking) reads and writes on critical files: /etc/shadow, /etc/kubernetes/, Secret mounts, /var/run/docker.sock, /proc/*/cmdline.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: sensitive-file-access
spec:
 kprobes:
 - call: "security_file_open" # LSM-ish via kprobe
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/etc/shadow"
 - "/var/run/secrets/"
 - "/var/run/docker.sock"
 matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/kubelet" # legitimate kubelet access
 matchActions:
 - action: Post

The matchBinaries: NotIn is important: kubelet and other legitimate node agents access these paths constantly and would generate noise. We filter those out in the kernel.

In enforcement: swap Post for Override with argError: -1 (EPERM), so that the open fails but the offending process stays alive and produces the error so that tracing tools pick it up.

3. Unauthorised outbound network connections

Detecting outbound connections to destinations outside the corporate range. Useful for spotting data exfiltration or malware command-and-control.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-external-egress
 namespace: prod
spec:
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr" # destination NOT in these CIDRs
 values:
 - "10.0.0.0/8"
 - "192.168.0.0/16"
 - "172.16.0.0/12"
 matchActions:
 - action: Sigkill
 options:
 - name: policy-mode
 value: enforce

This kills any attempt at a TCP connection to an IP that is not in the corporate CIDRs, in the prod namespace. Cilium already does this with NetworkPolicy, but Tetragon has two complementary advantages:

It gives you the process that attempted the connection, not just “pod X tried to connect to Y”.
It also works for exotic protocols where NetworkPolicy is less expressive.

4. Container escape detection

Container escape is the operational nightmare: a process inside a container manages to break the isolation (via a kernel exploit, a badly set capability, a misconfigured hostPath mount) and gain access to the host. Three typical signals:

A namespace change on the process (it leaves the container’s pid namespace).
setns or unshare in non-init processes.
Access to /proc/1/root or /dev/ from a container.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-container-escape
spec:
 kprobes:
 - call: "__x64_sys_setns"
 syscall: true
 args:
 - index: 0
 type: int
 - index: 1
 type: int
 selectors:
 - matchNamespaces:
 - namespace: Pid
 operator: NotIn
 values: ["host_ns"] # only processes NOT in the host pid namespace
 matchActions:
 - action: Sigkill
 - call: "__x64_sys_unshare"
 syscall: true
 args:
 - index: 0
 type: int
 selectors:
 - matchNamespaceChanges:
 - unshare: true
 matchActions:
 - action: Post

__x64_sys_setns targeting the host namespace from a process in a container is practically always malicious (legitimate containers do not need this at runtime).

5. Cryptomining

Mining processes have fairly recognisable profiles:

Processes with names like xmrig, minerd, cgminer, or legitimate processes such as python running CPU-intensive scripts.
Outbound connections to known mining pools (a public list of IPs and domains).
Anomalous use of /dev/cpu_dma_latency to avoid throttling.

A combined policy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-cryptomining
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm
 selectors:
 - matchArgs:
 - index: 4
 operator: "Postfix"
 values:
 - "/xmrig"
 - "/minerd"
 - "/cgminer"
 matchActions:
 - action: Sigkill
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "DPort"
 values: ["3333", "5555", "7777", "14444"] # common pool ports
 matchActions:
 - action: Post  # log only

The double policy: kill binaries with the classic names (belt) and log connections to pool ports (braces), so you also get an alert when someone renames xmrig to nginx-helper or uses exotic ports.

6. Detecting rootkits and suspicious kernel modules

Modern rootkits load kernel modules to patch functions (hide processes, hide network connections, hide files). Detecting them:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: kernel-module-load
spec:
 kprobes:
 - call: "do_init_module"
 syscall: false
 args:
 - index: 0
 type: "string"
 selectors:
 - matchActions:
 - action: Post
 - call: "security_kernel_read_file"
 syscall: false
 args:
 - index: 0
 type: "file"
 - index: 1
 type: int
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values: ["READING_MODULE"]
 matchActions:
 - action: Post

In a “well configured” Kubernetes cluster no new kernel modules get loaded at runtime; any event here is highly suspicious. Combine with enforcement on machines where modules should be fixed: Sigkill for whoever tries to load one.

Bonus: detecting third-party modification of eBPF maps

As a 2025-2026 trend: loading malicious eBPF programs to hide presence. Tetragon can observe the bpf syscall:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-bpf-syscalls
spec:
 kprobes:
 - call: "__x64_sys_bpf"
 syscall: true
 args:
 - index: 0
 type: int  # bpf cmd
 - index: 1
 type: bpf_attr
 selectors:
 - matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/cilium-agent"
 - "/usr/bin/tetragon"
 - "/usr/bin/bpftool"
 matchActions:
 - action: Post

Any process that is not one of the legitimate agents loading eBPF programs: you want to know about it.

Comparison with Falco

Falco is the closest competitor: it is also runtime security for Kubernetes, also originally based on eBPF (and before that on kernel modules), also with declarative policies. Three years ago they were functionally similar. In 2026 the divergence is clear:

Dimension	Tetragon	Falco
Philosophy	Cilium-native, integrated	Standalone, generic
Filtering	In the kernel (eBPF)	Parsing in userspace
Typical overhead	<1% CPU	5-10% CPU
Enforcement	Yes, in-kernel (Sigkill, Override)	Not native (depends on plugins)
Race conditions	No (action atomic with the syscall)	Yes, in enforcement via plugins
False positive rate	Low (K8s context in the kernel)	Higher (parsing afterwards)
Detection latency	5-26 ms	~10 ms (more constant)
Ecosystem maturity	Young, growing	Mature, plenty of material
Community	Cilium / CNCF Incubating	CNCF Graduated
Integrations	Hubble native	Falcosidekick, many
CRDs per policy	TracingPolicy / Namespaced	No CRDs; rules in YAML

When to choose each one:

Tetragon if you already use Cilium, if you need enforcement in the kernel (not detection-only), if overhead matters to you (workloads with many syscalls), and if you value the Hubble integration. Container escape and cryptomining detection is where its advantage over Falco shows up most.
Falco if you want a tool independent of the CNI, if you need the catalogue of ready-made rules and the wide community, if your cluster is not Cilium, or if the integration with SIEMs and ready-made notifiers (Falcosidekick) saves you work.
Both if the organisation is large: Falco for breadth of detection, Tetragon for surgical enforcement on critical workloads. That is what you see most in companies that have spent years with Falco and add Tetragon for specific cases.

Hubble + Tetragon: unified observability

Hubble is Cilium’s traffic observability component: it shows L3-L7 flow logs with zero impact on latency. Tetragon exposes its events over gRPC with the same format and vocabulary as Hubble, which allows you to:

See them in the same UI (Hubble UI shows Tetragon events as one more “layer”).
Correlate network events (Hubble) with process events (Tetragon) on the same timeline.
Export them together to Loki/Tempo/SIEM as a single stream.

The key synergy: Hubble tells you “this pod made a TCP connection to 1.2.3.4:80”. Tetragon tells you “this pod ran curl 1.2.3.4 from a bash binary launched by pid 1234”. Together they give you the full story.

Deployment and operation

Helm

The canonical install with Helm:

helm repo add cilium https://helm.cilium.io
helm install tetragon cilium/tetragon \
 --namespace kube-system \
 --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
 --set tetragon.exportFileMaxSizeMB=50 \
 --set tetragon.exportFileRotationInterval=24h

Tetragon deploys its DaemonSet, its CRDs and a service for Hubble. By default, it exposes the events on the agent pod’s stdout (any cluster log aggregator picks them up).

The `tetra` CLI

Tetragon ships a CLI called tetra for interactive investigation:

# real-time stream of the node's events
tetra getevents -o compact --pods <pod-name>

# structured JSON events to process with jq
tetra getevents -o json --since 5m --namespace prod

# view loaded policies
tetra tracingpolicy list

It is the best tool for debugging policies in monitoring before moving them to enforce.

Exporting to a SIEM

Three usual routes:

stdout + log aggregator: the agent writes JSON to stdout, Fluent Bit/Vector picks it up and sends it to Splunk/Datadog/Elastic. Simple, works with any logging infrastructure.
gRPC streaming: for low-latency integrations. A gRPC consumer of your own or Hubble Relay.
File + rotation: for air-gapped environments or regulatory audits that require persistent logs with controlled rotation.

Performance

Published benchmarks consistently place Tetragon at <1% of the node’s CPU under real workloads, compared with 5-10% for Falco under the same workloads. The reason is the architectural separation: Tetragon filters in the kernel and only carries the events that actually matter to userspace; Falco carries every syscall to userspace and filters there. On clusters with thousands of pods making hundreds of thousands of syscalls per second, the difference shows up on the bill.

Common operational traps

Permanent `monitoring`

The biggest trap is never reaching enforce: starting well with policies in monitoring, collecting events, tuning selectors, and then never switching. The result: you have detection without prevention, exactly what Falco gave you without paying Tetragon’s complexity. If you are going to use Tetragon, plan the road to enforce for the critical policies.

Selectors that are too lax

A policy with a single matchActions: Post and no specific selectors generates events for every syscall of the chosen hook. On a serious node that means tens of thousands per second, which fill logs, saturate exporters and hide the signal in the noise. Always start with strict filters (matchBinaries, matchNamespaces, matchPIDs) and open up once you know what you are looking for.

A kernel that is too old

Tetragon needs modern eBPF features. Kernels < 5.4 do not have the large buffer support (needed for execve with complete argv). Kernels < 5.10 do not have many of the LSM hooks. Kernel 5.15+ is the recommended minimum for production and 6.1+ to have every feature.

Hooks on renamed kernel functions

kprobes are tied to kernel function names that can change between versions. A policy that uses __x64_sys_setns can fail silently on a kernel where the function is called __do_sys_setns. Solutions: use static tracepoints where they are available (more stable), or keep alternative policies with several call entries for compatibility.

`Sigkill` in critical namespaces

Applying Sigkill to processes in kube-system or cilium-system can break the cluster. Enforcement policies must explicitly exclude the platform namespaces with matchNamespaces Operator: NotIn, or limit the scope with TracingPolicyNamespaced to make sure they do not act on systems they should not.

Missing `rateLimit`

A policy without a rateLimit on Post can suffer a catastrophic fan-out if the condition is met millions of times in an instant (typical in attack loops or application bugs). The agent saturates, events are lost, logs overflow. Always put a sensible rateLimit on detection policies, especially on high-frequency hooks such as tcp_connect or execve.

What we have not covered (upcoming articles)

eBPF LSM hooks in depth: how they relate to SELinux/AppArmor and when Tetragon is the right tool versus classic MAC.
Hubble UI with a Tetragon overlay: configuring the UI to show process observability and network observability on the same timeline.
Integration with OPA/Kyverno: how Tetragon complements admission policy engines (Kyverno validates at admission; Tetragon validates at runtime).
Forensics with eBPF: combining Tetragon with tools such as Beyla or OpenTelemetry to trace the full chain of an incident from the initial connection to the final syscall.

References

Official documentation (May 2026):

Tetragon — official site — entry point.
Tetragon docs — Tracing Policy — conceptual reference.
Tetragon docs — Hook points — kprobes, tracepoints, uprobes, LSM, USDT.
Tetragon docs — Selectors — full filter reference.
Tetragon docs — Enforcement Mode.
Tetragon docs — Kubernetes Identity Aware Policies — TracingPolicyNamespaced.
Tetragon GitHub.

Comparisons and analysis:

Comparative Analysis of eBPF-Based Runtime Security Monitoring (SciTePress paper 2025) — a benchmark with independent numbers.
Best eBPF Security Solutions for Kubernetes (ARMO, 2026) — Falco vs Tetragon vs KubeArmor comparison.
Falco vs. Tetragon (Asim Mirza, Medium) — analysis with use cases.
Deep Dive into Tetragon (A Security Engineer) — a walk through the inside of the agent.
Tetragon Series, Part 2: Enforcing Sensitive File Access (Medium).

Ecosystem:

Cilium Hubble — network observability.
Falco — official site — the other big name in the field.
KubeArmor — the third option, with AppArmor + eBPF.

Cross-references:

Part 1 of the series: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack — the eBPF fundamentals we take as read here.
Kubernetes with Cilium BGP: services reachable without Ingress — the starting point of the Cilium ecosystem on this blog.

eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack

Tue, 19 May 2026 04:30:00 +0200

TL;DR

eBPF is a sandboxed virtual machine inside the Linux kernel that runs verified code at well-defined hooks: kprobes, tracepoints, socket events, network drivers. Before eBPF, changing kernel behaviour meant recompiling it or loading an arbitrary module; with eBPF, you load a small program that passes a formal verifier and runs at native speed with memory safety. In networking, this translates into the fact that the packet does not have to travel through the traditional TCP/IP stack: an eBPF program in the NIC driver (XDP) can drop, forward or rewrite the packet before the kernel has done its first alloc; a program on cgroup hooks (sock_ops) can redirect connections to another socket without the packet ever leaving the machine. Cilium is the CNI that has taken this to its logical conclusion: it replaces kube-proxy with pure eBPF (O(1) instead of the O(N) of iptables), routes pod-to-pod without VXLAN where it can, evaluates Network Policies with BPF maps, and since 1.16 it has remade its BGP control plane with a new set of CRDs, CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, CiliumBGPNodeConfigOverride, which replace the monolithic CiliumBGPPeeringPolicy that is already deprecated. This post goes down the three layers (basic eBPF → eBPF networking → Cilium) and ends with the operational CRDs.

The analogy: signed plugins for the kernel

Think of the browser. Twenty years ago, extending a browser meant compiling a native binary and loading it: any extension could crash it, corrupt memory, read your bank cookies. Today, extensions are JavaScript in a sandbox with a manifest that declares permissions, a runtime that enforces the isolation and a store that signs the code. The extension does not touch the browser binary; it lives in a controlled world and can only talk to the browser through defined APIs. Result: massive extensibility with a bounded attack surface.

eBPF is exactly that for the Linux kernel. Loading a classic .ko module means loading native code with full access to kernel memory: one bug and the system is gone. eBPF is a bytecode VM with a static verifier, a controlled allocator, JIT to native hardware after passing the verifier, and a set of kernel “helpers” it can call. The eBPF program can read the kernel memory the verifier allows it to read, and only that. It cannot enter infinite loops (the verifier demands that it terminate). It cannot jump to arbitrary addresses. It cannot dereference pointers without having validated them first. And, most importantly: it is user code, loaded at runtime, executing inside the kernel at native speed.

The consequences are visible from miles away. Before, observing traffic in production meant patching the kernel or loading a risky module. Today, bpftrace -e 'tracepoint:net:net_dev_xmit { @[args->dev->name] = count(); }' gives you a histogram of packets per interface in three lines and zero downtime. Before, replacing iptables with something faster meant rewriting the netfilter subsystem. Today, Cilium loads 60 KB of eBPF bytecode into XDP and unseats iptables with a hash map.

Basic eBPF: what it is and what it is not

The origin and the scope

The name comes from Berkeley Packet Filter, a 1992 idea (McCanne and Jacobson) for filtering packets with a mini-bytecode that tcpdump used internally. In 2014, Alexei Starovoitov renamed it eBPF and extended it enormously: 11 64-bit registers instead of 2 32-bit ones, a 512-byte stack, maps as structures shared with userspace, JIT to native hardware, and a far more sophisticated formal verifier. From being a packet filter, it became a generic kernel extensibility mechanism.

Today eBPF is used for four things:

Networking: XDP, TC, cgroup hooks, socket ops, lightweight tunnels.
Observability: kprobes, uprobes, tracepoints, USDT. The basis of projects like bpftrace, bcc, Pixie, Parca.
Security: BPF LSM (Linux Security Module in eBPF), syscall blocking with seccomp-bpf. Falco, Tetragon, Tracee.
Scheduling: sched_ext (kernel 6.12+), process schedulers written entirely in eBPF. Still at a very early stage.

The VM

An eBPF program is compiled from C (or Rust, or Go with cilium/ebpf) to eBPF bytecode, not to x86/arm64 directly. The kernel loader (via the bpf() syscall) passes that bytecode through the verifier:

It reconstructs the control flow graph.
It performs static analysis of every possible path: every instruction has to be reachable, every memory access has to be within known bounds, every pointer has to have been validated.
It rejects loops without a known upper bound. Recent kernels admit bounded loops (the bpf_loop helper), but the counter is always finite.
It rejects calls to helpers or kfuncs that the hook’s program type does not allow.

If the verifier accepts the program, the JIT translates it to the host’s native code (x86, arm64, etc.) and it stays attached to its hook. From then on it runs every time the hook’s event occurs, with no context switch to userspace, with no syscall cost. Latencies on the order of hundreds of nanoseconds per invocation.

Maps: the bridge to userspace

An isolated eBPF program is not much use. What makes it useful are maps: data structures shared between the kernel program and userspace. There are several types:

BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_LRU_HASH: hash tables with or without LRU eviction.
BPF_MAP_TYPE_ARRAY, BPF_MAP_TYPE_PERCPU_ARRAY: arrays, optionally per-CPU to avoid contention.
BPF_MAP_TYPE_RINGBUF, BPF_MAP_TYPE_PERF_EVENT_ARRAY: channels for streaming events to userspace.
BPF_MAP_TYPE_PROG_ARRAY: arrays of eBPF programs for tail calls (chaining programs without returning to the base kernel).

Userspace reads and writes these maps via bpf() syscalls; the kernel program reads and writes them directly. It is the basis of any eBPF system: the kernel program collects data into a map, the userspace daemon reads it. Cilium does exactly this: the userland agent (Go) manages the policy and translates it into map entries; the eBPF programs living in XDP/TC read the maps and apply the decisions.

CO-RE: compile once, run on any kernel

A classic nightmare of kernel modules: they are tied to the exact kernel version they were compiled against. Distributing a precompiled module for a fleet of machines with different distros was impossible.

eBPF solves this with CO-RE (Compile Once, Run Everywhere): the bytecode includes relocations that the loader resolves on each specific kernel by consulting BTF (BPF Type Format), a representation of the kernel’s struct layouts that the kernel itself publishes. Result: a single eBPF binary works on kernels 5.10, 5.15, 6.1 and 6.8 without recompiling, because the loader adjusts the struct access offsets at runtime.

This is what has allowed productive eBPF distributions to exist. Without CO-RE, every kernel would be a porting project.

eBPF in networking: the hooks that matter

Inside the Linux network subsystem, eBPF has several hooks. The ones relevant to CNIs:

XDP — eXpress Data Path

XDP is the earliest hook: it runs in the NIC driver, before the packet enters the kernel proper. There is no sk_buff (the struct the rest of the kernel uses to represent packets); there is only a pointer to a RAM buffer with the received bytes.

The actions an XDP program can return:

XDP_DROP: discard the packet immediately. The driver drops it and frees the buffer. Cost: nanoseconds. Use case: DDoS mitigation. Cloudflare processed more than 8 million packets/second per CPU with XDP for dropping SYN floods.
XDP_PASS: let the packet continue to the normal kernel. It becomes an sk_buff and enters the traditional stack.
XDP_TX: send it back out on the same interface after possible modifications. Useful for L4 load balancers that rewrite the destination and return it.
XDP_REDIRECT: send the packet to another interface or to a map (to forward to userspace via AF_XDP, or to another NIC, or to a pod’s veth).
XDP_ABORTED: error (increments a counter, drops).

Real use cases:

Cloudflare L3 DDoS protection: XDP rules that drop millions of packets/s.
Facebook Katran: an L4 load balancer that rewrites the destination IP and returns it on the same interface. Handles 10× more connections per server than classic IPVS.
Cilium XDP acceleration: Service load balancing at the lowest layer possible.

TC (Traffic Control) — clsact with BPF

XDP is very fast but limited: the packet does not have an sk_buff yet and many decisions (conntrack, NAT, encapsulation with metadata) are easier when it does. The TC clsact with BPF hook runs after building the sk_buff but before the routing and netfilter decisions. Actions:

TC_ACT_OK: the packet continues through the stack.
TC_ACT_SHOT: drop.
TC_ACT_REDIRECT: redirect to another interface.
TC_ACT_PIPE, TC_ACT_STOLEN: pipeline control for combining with other qdiscs.

Use cases:

Stateful network policy: Cilium evaluates L3-L7 policies in TC with the full sk_buff and conntrack available.
Marking and QoS: traffic marking so the scheduler applies priorities.
Overlay encapsulation: adding VXLAN/Geneve headers when the mode is tunnel.

XDP and TC combine: XDP for the cheap and early stuff (DDoS, simple LB), TC for what needs an skb and state.

Cgroup hooks: sock_ops and CGROUP_SOCK_ADDR

The most radical conceptual step: hooks that are not in the network layer but in the socket layer. Relevant types:

BPF_PROG_TYPE_CGROUP_SOCK_ADDR: invoked when a process in a cgroup calls connect(), bind(), sendto(). The eBPF program can rewrite the destination address before the connection goes out. This is what lets Cilium do Service load balancing without the packet entering the network stack: if the client tries to connect to 10.96.0.1:443 (a ClusterIP), an eBPF program on this hook rewrites the destination to the real IP of the backend pod before the syscall continues.
BPF_PROG_TYPE_SOCK_OPS: invoked on TCP events (creation, established, retransmission). It allows tuning socket parameters at runtime and, most importantly, pairing local sockets via bpf_sk_assign to shortcut without the packet travelling over the network.

This is the “third layer” of the bypass: it is not just faster, it is conceptually different. The packet is not built, not serialised, does not traverse the IP layer or the TCP layer. It is the difference between speeding up a road and discovering that for some journeys you do not need to take the car at all.

The long road: what the traditional TCP/IP stack looks like

To appreciate what eBPF saves, it is worth tracing a packet’s full journey through the Linux stack. Take the case “packet arrives on a NIC, goes to a local process”:

NIC (DMA into the driver's ring buffer)
↓
driver: napi_schedule, poll, allocates sk_buff
↓
[XDP hook] ← if there is an XDP program, it is decided here
↓
netif_receive_skb
↓
__netif_receive_skb_core
↓
[TC ingress clsact + BPF] ← if there is a TC ingress program
↓
packet_type handlers (IP, ARP...)
↓
ip_rcv → ip_rcv_core
↓
[netfilter NF_INET_PRE_ROUTING] ← iptables PREROUTING
↓
routing decision (FIB lookup)
↓
[netfilter NF_INET_LOCAL_IN] or [NF_INET_FORWARD]
↓
tcp_v4_rcv → tcp_v4_do_rcv
↓
tcp_rcv_established
↓
sk_data_ready
↓
process reads with recv()/read()

Every arrow is a function call with a measurable cost. Every netfilter hook walks all the registered iptables/nftables rules. With kube-proxy in iptables mode and 5,000 Services × 10 endpoints each, there are on the order of 150,000 rules evaluated sequentially at NF_INET_PRE_ROUTING. Published benchmarks show latencies of tens of microseconds per packet in large Kubernetes clusters in the netfilter step alone, before the application receives anything.

And that is the normal path. On the way out the same thing happens in reverse: tcp_sendmsg → ip_output → NF_INET_LOCAL_OUT → routing → NF_INET_POSTROUTING → dev_queue_xmit → driver → NIC.

How Cilium skips this stack

Cilium does not eliminate the TCP/IP stack; it is still there for the cases that need it. What it does is shortcuts at the points where it hurts.

Shortcut 1 — XDP for the Service datapath

For a cluster with 5,000 Services, kube-proxy iptables has an O(N) cost in evaluating rules (even with iptables-restore --noflush and tricks, it is still linear in the number of chains the packet traverses).

Cilium replaces it like this:

Every Service and its endpoints live in an eBPF hash map.
When a packet comes in destined for a ClusterIP, Cilium’s XDP program does an O(1) lookup in that map and obtains the backend endpoint.
It rewrites the destination and does XDP_TX (returns it on the same interface towards the backend) or XDP_REDIRECT (sends it to the corresponding local pod’s veth).

This means the cost does not grow with the number of Services. 100 Services or 100,000, constant lookup in the map. Published benchmarks show latency reductions of 30-50% in clusters with many Services compared with kube-proxy iptables, and of an order of magnitude compared with IPVS in some cases.

Shortcut 2 — socket-LB: the packet is never built

Cilium 1.6+ introduced socket-level load balancing, based on cgroup hooks. It works like this:

When a pod calls connect(10.96.0.1:443) (a Service’s ClusterIP), the syscall enters the kernel.
Before the kernel builds anything network-related, an eBPF program on CGROUP_SOCK_ADDR/connect4 intercepts it and rewrites the destination address to the real IP of the backend pod.
The kernel carries on with the connect as if the client had written 10.0.0.42:8080 directly.

Why does it matter? Because when the backend pod is on the same node, this shortcut turns a call that would have involved:

syscall connect → kernel stack → veth → bridge → veth → kernel stack → syscall accept

into:

syscall connect (with the destination rewritten) → direct loopback

The TCP/IP stack is literally avoided. There is no encapsulated packet, no journey through veth pairs, no netfilter. L7 pod-to-pod latencies on the same node drop to local communication levels (~5-15 µs instead of ~30-50 µs for services with kube-proxy iptables and traditional veth).

Shortcut 3 — pod-to-pod direct routing

The traditional overlay mode (Flannel, Calico VXLAN) encapsulates every pod-to-pod packet in VXLAN/Geneve. Every packet carries an extra 50-byte header, requires encap/decap, and consumes MTU.

Cilium supports direct routing: the pod CIDRs are advertised to the underlying fabric (with BGP, which is where the control plane we will look at comes in) and the physical routers route the pod-to-pod packets without encapsulating. The packet leaves a pod with its original IP as source and the destination pod’s IP as dest, the node’s NIC hands it to the network, the network routes it, it arrives at the destination node and is delivered to the pod. Zero encap, full MTU, minimal latency.

Cilium does this via eBPF programs in TC that rewrite the necessary headers and decide whether the packet goes via encap or direct according to the policy configured per node.

Shortcut 4 — Network Policy in TC with maps

Network Policies in classic CNIs are usually translated into iptables rules, another factor that explodes linearly. Cilium evaluates them in eBPF programs that read identity maps: every workload has a numeric identifier computed from its labels, and the policy is a map (src_identity, dst_identity, port, proto) → allow|deny. One hash map lookup per packet.

This also enables Cilium’s L7 policies (HTTP, gRPC, Kafka filtering): the eBPF program recognises the L7 handshake, selectively redirects to the embedded Envoy proxy (which lives as a sidecar of the datapath, not as a pod sidecar) and only on that subset does it pay the cost of the L7 proxy. All the L3/L4 traffic stays on the eBPF fast path.

Cilium: the architecture

Cilium combines two planes:

Agent (Go): lives as a DaemonSet on every node. It is the “slow” part: it translates the intent expressed in CRDs (CiliumNetworkPolicy, CiliumBGPClusterConfig, etc.) into entries in eBPF maps. It talks to the Kubernetes API server to discover endpoints, services, pods. It embeds a GoBGP for the BGP control plane. It embeds an Envoy for L7 policies.
Datapath (eBPF): the programs loaded into XDP, TC, cgroup hooks. They are the “fast” part: they see every packet, read the maps the agent maintains, and decide in nanoseconds.

This separation is what makes Cilium operationally comfortable: the intent is expressed in YAML, the agent materialises it into maps, the maps are read by the datapath. If the agent goes down temporarily, the datapath keeps working with the last loaded configuration. As in any well-built control/data plane system.

BGP Control Plane v2: the CRDs you have to know

Cilium has had BGP support for several years. The first version used a single monolithic CRD, CiliumBGPPeeringPolicy, which mixed node configuration, peers, timers and advertisements into a single object. Since Cilium 1.16 there is BGP Control Plane v2, which breaks that configuration into separate CRDs with clear responsibilities. CiliumBGPPeeringPolicy (the cilium.io/v2alpha1 API) is deprecated and migration warnings appear in the operator logs if you still use it.

The new CRDs (the cilium.io/v2 API):

1. `CiliumBGPClusterConfig`

Defines BGP instances and the peers they connect to, from the cluster’s perspective. Which nodes apply this configuration is selected with a nodeSelector.

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
 name: cilium-bgp-cluster
spec:
 nodeSelector:
 matchLabels:
 bgp-policy: rack-1  # only nodes with this label
 bgpInstances:
 - name: instance-65000
 localASN: 65000
 peers:
 - name: top-of-rack-1a
 peerASN: 64512
 peerAddress: 10.0.1.1
 peerConfigRef:
 name: tor-shared-config  # → reference to CiliumBGPPeerConfig
 - name: top-of-rack-1b
 peerASN: 64512
 peerAddress: 10.0.1.2
 peerConfigRef:
 name: tor-shared-config

A BGP instance is the abstraction “this node takes part in BGP with this local ASN and these peers”. Several can coexist on the same node (multi-instance for multi-VRF).

2. `CiliumBGPPeerConfig`

Defines the shared parameters of the peering: timers, address families, transport, MD5 password, graceful restart, etc. It is referenced from CiliumBGPClusterConfig via peerConfigRef. This avoids repeating the same configuration for every peer when there are dozens of them.

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
 name: tor-shared-config
spec:
 timers:
 holdTimeSeconds: 30
 keepAliveTimeSeconds: 10
 connectRetryTimeSeconds: 5
 gracefulRestart:
 enabled: true
 restartTimeSeconds: 120
 families:
 - afi: ipv4
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp  # → binds to CiliumBGPAdvertisement
 - afi: ipv6
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp
 authentication:
 password:
 name: bgp-md5-secret  # Secret with the MD5 password
 key: password

A single CiliumBGPPeerConfig can be referenced by many different peers. You change timers or families in one place.

3. `CiliumBGPAdvertisement`

Declares which prefixes are advertised: the node’s pod CIDRs, the ClusterIPs and ExternalIPs of Services, the IPs assigned by CiliumLoadBalancerIPPool for type=LoadBalancer Services. They are bound to CiliumBGPPeerConfig via labels.

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
 name: services-and-pods
 labels:
 advertise: bgp  # ← the label the PeerConfig matches
spec:
 advertisements:
 - advertisementType: PodCIDR  # advertises the node's pod CIDR
 attributes:
 communities:
 standard:
 - "65000:100"
 - advertisementType: Service  # advertises ClusterIPs / LoadBalancer IPs
 service:
 addresses:
 - LoadBalancerIP
 - ClusterIP
 - ExternalIP
 selector:
 matchLabels:
 bgp-advertise: "true" # only Services with this label
 attributes:
 communities:
 standard:
 - "65000:200"
 localPreference: 200

The granularity is very fine: you can advertise different types of prefixes with different BGP communities, different local-preference, different path attributes, and filter Services with label selectors. This was literally impossible with CiliumBGPPeeringPolicy v1.

4. `CiliumBGPNodeConfig` (auto-generated)

This CRD is not configured by hand. The Cilium operator generates one per node from the CiliumBGPClusterConfig that applies to that node. It is the materialised per-node state that each node’s agent reads to bring up its peerings. If you want to see what BGP configuration is actually running on a node, kubectl get ciliumbgpnodeconfig <nodename> -o yaml shows you.

5. `CiliumBGPNodeConfigOverride`

Optional. It allows overriding the generated configuration for a specific node when you need something non-standard. Use cases:

Pinning the BGP router-id to a specific IP (useful when the node has several interfaces).
Specifying the peer’s local address when there are several outgoing interfaces.
Changing timers only for one problematic node.

apiVersion: cilium.io/v2
kind: CiliumBGPNodeConfigOverride
metadata:
 name: node-rack1-master01  # the name must match the node's
spec:
 bgpInstances:
 - name: instance-65000
 routerID: 10.0.1.10 # router-id override
 peers:
 - name: top-of-rack-1a
 localAddress: 10.0.1.10 # specific local interface

Relationship diagram

CiliumLoadBalancerIPPool: it complements, it is not BGP

Although it is not strictly a BGP CRD, it is worth mentioning: CiliumLoadBalancerIPPool is the CRD that supplies IPs to type=LoadBalancer Services. It defines a range (10.20.0.0/24, for example) that Cilium assigns automatically to LoadBalancer Services. Combined with a CiliumBGPAdvertisement that advertises LoadBalancerIP, it gives the complete cycle: new Service → IP assigned from the pool → BGP advertisement to the routers → IP routable from the corporate network, with no external balancer.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
 name: lb-pool-rack1
spec:
 blocks:
 - start: "10.20.0.10"
 stop: "10.20.0.250"
 serviceSelector:
 matchLabels:
 lb-pool: rack1

Full manifest: pod CIDRs + LoadBalancer Services advertised to a redundant ToR pair

A realistic example of a cluster with two top-of-rack switches as BGP peers, both in the same AS (64512), Cilium in AS 65000:

# 1. CiliumBGPPeerConfig — shared config for both ToRs
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
 name: tor-peers
spec:
 timers:
 holdTimeSeconds: 30
 keepAliveTimeSeconds: 10
 gracefulRestart:
 enabled: true
 restartTimeSeconds: 120
 families:
 - afi: ipv4
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp

# 2. CiliumBGPAdvertisement — what gets advertised
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
 name: pods-and-lb
 labels:
 advertise: bgp
spec:
 advertisements:
 - advertisementType: PodCIDR
 - advertisementType: Service
 service:
 addresses: [LoadBalancerIP]
 selector:
 matchExpressions:
 - { key: io.kubernetes.service.namespace, operator: NotIn, values: [kube-system] }

# 3. CiliumBGPClusterConfig — which nodes talk to which peers
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
 name: cluster-bgp
spec:
 nodeSelector:
 matchLabels:
 bgp: enabled
 bgpInstances:
 - name: instance-65000
 localASN: 65000
 peers:
 - name: tor-a
 peerASN: 64512
 peerAddress: 10.0.1.1
 peerConfigRef: { name: tor-peers }
 - name: tor-b
 peerASN: 64512
 peerAddress: 10.0.1.2
 peerConfigRef: { name: tor-peers }

# 4. CiliumLoadBalancerIPPool — range of LB IPs
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
 name: lb-corporate
spec:
 blocks:
 - cidr: "10.20.0.0/24"

Four objects. Before, in v1, it was a single CiliumBGPPeeringPolicy that mixed everything together and turned out to be hard to maintain in clusters of more than 5 nodes with heterogeneous configuration. The new separation is longer but clearly factorable: one PeerConfig per peer type, one Advertisement per advertisement policy, one ClusterConfig connecting nodes with peers.

Common operational traps

`routingMode: tunnel` mode with BGP

BGP only makes sense with direct routing (routingMode: native). If you have tunnel mode (VXLAN/Geneve) and configure BGP, you will advertise pod CIDRs but the packets will still go out encapsulated, producing confusing behaviour (sometimes via tunnel, sometimes direct depending on routes). Configure routingMode: native and disable the tunnel.

eBPF host routing vs `kube-proxy replacement`

They are two different things. kubeProxyReplacement: true enables the replacement of kube-proxy (the Services). bpf.hostRouting: true enables the host’s iptables bypass (the node’s routing decisions are made with eBPF instead of the traditional FIB). The second needs kernel 5.10+ with all the bpf features enabled; if you do not have that kernel, it falls back to legacy mode and the performance is only “almost as good”.

Aggressive BGP timers over flapping NICs

With holdTimeSeconds: 9 / keepAliveSeconds: 3, a NIC that blinks for 5 seconds breaks the BGP session and all the advertised routes disappear from the fabric. That node’s pods become unreachable until the session is re-established. For clusters on hardware with suspect NICs, use the conservative values (holdTime: 30, keepAlive: 10) and consider graceful restart explicitly (it is already in the example above).

Advertising ClusterIP to the corporate network

Advertising ClusterIP to external routers is rarely what you want: these are internal Service IPs, not designed to be reached from outside the cluster. For external exposure, use LoadBalancerIP from a CiliumLoadBalancerIPPool. Advertising ClusterIP only makes sense in very specific topologies (multi-cluster mesh with shared service discovery).

Mixing v2alpha1 (`CiliumBGPPeeringPolicy`) and v2 (`CiliumBGPClusterConfig`)

It does not work well. The operator emits warnings in the logs about the use of the deprecated API, and conflicts between what the peering policy defines and what the cluster config defines can produce strange states. Migrate from one to the other in a single pass; do not run both.

MD5 password and MTU

If you configure an MD5 password in CiliumBGPPeerConfig.authentication, the TCP header is larger. On links with a tight MTU (1500 - 50 for the upstream fabric’s VXLAN, for example), the BGP handshake can fragment and die silently. Either use MTU 9000 between nodes and ToR, or make sure the MSS values are negotiated correctly.

What we have not covered

Cilium Cluster Mesh: federation of several Cilium clusters so their Services can see each other. It fits with BGP when you want native routing between clusters; it has its own CRDs.
L7 Policies and the embedded Envoy: HTTP/gRPC/Kafka policy. Another layer of eBPF + proxy that deserves its own post.
Hubble: eBPF-based traffic observability that Cilium exposes. Flow log dashboards with zero impact on latency.
Transparent WireGuard: pod-to-pod encryption without sidecars, controlled by Cilium via eBPF redirect to a kernel WireGuard dataplane.
Gateway API in Cilium: the successor to Ingress, with first-class support since Cilium 1.16+.
eBPF for LLM serving: the natural connection with the previous inference series. There is recent work using eBPF for multi-tenant fairness on GPUs and for token tracking; paper territory, not production yet.

In other words, there is material left for another three posts in today’s series. Let us take them in order.

References

Conceptual and project:

eBPF.io — canonical documentation of the eBPF ecosystem.
The BPF Compiler Collection (bcc) and bpftrace — tools to get started with.
eBPF en 2026: How Extended Berkeley Packet Filter Became the Engine of Linux Observability and Networking — state of the art.

XDP, TC and firewalling:

Getting started with XDP and eBPF (Red Hat docs).
Full Guide to BPF Firewalls: XDP, tc, and eBPF Integration (Medium, 2025).
Cloudflare blog: XDP for DDoS mitigation.
Facebook Katran (GitHub) — L4 LB with XDP, code and paper.

Cilium:

Cilium documentation — always the first port of call.
Kubernetes Without kube-proxy — the official guide to the replacement.
Cilium BGP Control Plane Resources (docs) — reference for the v2 CRDs.
Configuring Cilium BGP Control Plane (OneUptime blog, mar 2026) — walkthrough.
A Guide to BGP Control Plane and Cluster Mesh in Cilium Networking (Sigrid Jin, Medium) — a deeper post with use cases.

Cross-references:

Previous post on this blog: Kubernetes with Cilium BGP: services reachable without Ingress — the first step, with version v1 (which now needs migrating).
Series on LLM inference: KV cache, vLLM on Kubernetes, PagedAttention deep dive, LLM K8s Operators — where the fast network (which we will see in the following posts of this series) determines real performance.

Ebpf on lo0 — Tech Blog

How we use our infrastructure to meet defence standards with AI

The map of standards in the sector

Requirement: coalition interoperability (FMN, MIP/JC3IEDM)

Requirement: do not trust by default, Zero Trust and verified access (DoD ZT RA, NIST 800-171)

Requirement: data-centric security and labelling (STANAG 4774/4778)

Requirement: segregation by mission and level (multitenancy, NIST 800-171, ENS)

Requirement: confidentiality, encryption and exfiltration prevention (NIST 800-171, ENS, Zero Trust)

Requirement: verifiable software supply chain (Iron Bank/STIG, SBOM, CMMC)

Requirement: sovereignty and disconnected or air-gapped operation (classified networks, DDIL)

Requirement: resilience and operation at the tactical edge (DDIL)

Requirement: robustness and protection of the AI system itself (PRU reliability)

Requirement: reliable execution of mission workloads and efficiency (availability, sustainability)

Requirement: explainability, traceability and governability (NATO PRU, NIST AU)

Requirement: continuous accreditation and reproducible operation (cATO, CMMC, ENS)

The piece that ties it all together: ISO/IEC 42001 and NATO’s principles

A view of the sector as of June 2026

Summary: requirement by requirement

Sources

Measuring energy in production: Kepler, DCGM and the practical stack (accuracy and overhead)

What this article covers

The measurement stack on Kubernetes

DCGM exporter: the base of GPU power

Kepler in depth: eBPF and the sources

Deployment: a DaemonSet on top of Prometheus

PromQL queries: energy per pod

A calibration procedure

Accuracy: estimation vs power meter

Overhead: the cost of measuring

Zeus: for the bench and for optimisation, not for continuous production

CodeCarbon and Scaphandre in practice

Choosing the stack by objective

From figure to action

From energy to carbon: the final layer

Energy efficiency KPIs to watch

Example: energy attribution on a multi-tenant cluster

Energy and carbon-aware scheduling

Deployment checklist

Energy by phase: prefill vs decode

State of the art: the LLM resolution gap

Limits and traps (data-driven)

Closing

See also

Sources

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

TL;DR

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

A1 — The per-project .ai-jail file

A2 — Permission allowlists with –bootstrap

A3 — Lockdown for anything you do not trust

A4 — The safety net: git with no push

Track B — Cluster (RKE2 with Cilium + Tetragon)

B0 — The pod baseline

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

B2 — Tetragon, observation phase (Post)

B3 — Tetragon, enforcement phase (Sigkill)

The client ↔ cluster equivalence table

Checklist of gotchas

See also

References

The contractor with the master key: isolating AI agents from workstation to cluster

TL;DR

The analogy: the contractor with the master key

The threat model: what a runaway agent can do

Two domains, one policy

The client: isolating the agent on the workstation

The cluster: isolating the agent in production

The landscape table

Extrapolate, do not copy

What no sandbox solves

See also

References

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

TL;DR

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

The gap: why the Topology Manager does not look at the NIC

The network datapath under NUMA: IRQ, softirq and DMA

`stdiocap`: capturing the stdio of local MCP servers