Cilium on lo0 — Tech Blog

Hardening and secrets in the sovereign LLM stack: defence in depth

Thu, 11 Jun 2026 11:00:00 +0000

Part of the operational series on squeezing a generic on-premise 4×H100 SXM 80GB LLM cluster. The sibling pieces: the document ingestion from PDF to indexed chunk that fills the vector database, the embeddings and rerankers service with TEI in production that feeds it, and, the most directly coupled to this post, the GitOps of the inference stack with Flux, because GitOps and secrets share a chicken-and-egg problem we solve here. Assembling all of this into a conversational assistant (LibreChat + LiteLLM + RAG) is covered by another post in the series, still in draft.

TL;DR

A sovereign LLM inference stack is, at minimum, six services: a gateway (LiteLLM or equivalent), an inference engine (vLLM), an embeddings/rerankers service (TEI), a vector database, a state database (conversations, users) and a front end (LibreChat). Six services are six attack surfaces, and the assistant “works” as soon as the gateway returns tokens, long before it is hardened. This post walks through defence in depth layer by layer:

Secrets: never in the clear in git. sealed-secrets (asymmetric encryption, controller in the cluster) vs SOPS/age (file encryption, simple, pure GitOps) vs External Secrets Operator + Vault (dynamic secrets, native rotation). GitOps’s chicken-and-egg problem.
Network: default-deny NetworkPolicy + explicit allow; L3/L4 with the standard NetworkPolicy, L7 and DNS-based egress with Cilium (eBPF); egress control so the data does not leave the perimeter; internal mTLS.
Pod security: runAsNonRoot, readOnlyRootFilesystem, dropped capabilities, seccomp RuntimeDefault, restricted Pod Security Standards, nothing privileged.
Supply chain: pin by digest, scan with Trivy, sign with cosign, admission control that rejects anything unsigned.
AuthN/Z: LiteLLM virtual keys, OIDC/LDAP in the front end, Kubernetes RBAC with least privilege.
Runtime: detection and, optionally, enforcement with Tetragon.
Data at rest: storage encryption, credentials for the vector database and the state database.

The thesis: hardening reduces the blast radius, it does not eliminate it. That is why you prioritise by impact, and the first job on a 4×H100 is the gateway↔vector store pair and egress-deny.

The analogy: the sovereign office

Picture an office holding sensitive documentation: the corpus that feeds the RAG, the users’ conversations, the engine’s credentials. It is not enough for the street door to lock. Real security is defence in depth: several layers, each assuming the previous one can fail.

The mapping is direct:

The building’s perimeter is egress control: making sure a leaked secret or a compromised process cannot exfiltrate the corpus to an external server. It is the most underrated layer and the first one I put in place in a sovereign deployment.
Access control at every interior door is the default-deny NetworkPolicy: the front end does not talk to the vector database directly because it has no reason to; only the strictly necessary pairs are open.
The fire partitions are pod isolation (restricted) and namespaces: if one service catches fire, the fire does not jump to the one next door.
The doorman who checks credentials is the admission control that verifies each image’s cosign signature before letting it into the cluster.
The cameras are Tetragon/Falco: they record what each process did and detect (or kill) anything anomalous.
The safe with a rotated key is the secrets: encrypted at rest, out of git, and rotated so a stolen key expires.

No layer is sufficient on its own. The building is secure because crossing them all is expensive.

Layer 1 — Secrets: GitOps’s chicken-and-egg problem

The stack is deployed by GitOps: the sibling piece on Flux reconciles the state declared in git against the cluster. That is excellent for manifests, but there is a foundational problem: the inference engine needs the Hugging Face token to download the model, the state database needs its password, the gateway needs its master key. Those secrets cannot go into git in the clear. Anyone with read access to the repo, and in an organisation with generous read that is a lot of people, plus every repo backup, plus every fork, would read them.

The chicken and the egg: GitOps wants all state to be in git, but secrets cannot be in git in the clear. The solution comes in three families:

Family A — sealed-secrets (asymmetric encryption, controller in the cluster)

Bitnami Sealed Secrets is a controller running in the cluster plus a client CLI, kubeseal. It uses asymmetric cryptography: there is a key pair. The public one is held by developers and is used to encrypt; the private one lives only in the cluster’s controller and is used to decrypt. The flow:

The developer takes a normal Secret and encrypts it with kubeseal, which obtains the controller’s public key. Result: a SealedSecret resource.
That SealedSecret is committed to git in the clear — it is encrypted, it is not readable.
The controller in the cluster detects it, decrypts it with its private key and creates the real Kubernetes Secret in the target namespace.

The key property: because the private key never leaves the cluster, neither the developer nor anyone with access to git can decrypt. And the encryption includes the namespace name: a SealedSecret sealed for inferencia cannot be moved to front and decrypted there — it behaves as if each namespace had its own key. The controller also manages rotation of the sealing keys, labelling them as active or compromised.

Advantage: it fits pure GitOps perfectly, the encrypted secret lives with the rest of the state. Limitation: the secret, once decrypted, ends up as a normal Kubernetes Secret — it sits in etcd, and etcd has to be encrypted separately (we will see that in layer 7).

Family B — SOPS + age (file encryption, simple)

SOPS encrypts the whole YAML/JSON file (or only its values) and leaves it in git encrypted, decrypting it only at deployment time. It supports cloud KMS (AWS/GCP/Azure), PGP and, the relevant one for on-premise sovereignty, age, a simple serverless offline encryption scheme. The GitOps operator (Flux brings native SOPS integration) decrypts on reconciliation using the age key stored in the cluster.

Advantage: simple, scriptable, with no server to maintain; the operational weight falls on safeguarding the age key. It is the usual recommendation for small teams or for getting started. Limitation: rotation is manual (re-encrypt everything with the new key) and there are no dynamic secrets.

Family C — External Secrets Operator + Vault (dynamic secrets, native rotation)

The External Secrets Operator (ESO) keeps no secrets in git at all. All that goes into git is an ExternalSecret: a reference saying “the password field comes from path secret/data/vectordb in such-and-such a SecretStore”. The operator reads from an external store, typically HashiCorp Vault, and synchronises the Kubernetes Secret. Vault keeps the secrets in its own encrypted backend and serves them over an authenticated API, so that they never reside in git and, with its dynamic engines, it can issue short-lived database credentials that expire on their own.

Advantage: native rotation, dynamic secrets, centralised auditing, a single point of governance. Limitation: you have to operate Vault (sealing/unsealing, high availability, access policy), which is real work.

How to choose, and rotation

Criterion	sealed-secrets	SOPS + age	ESO + Vault
Secret in git	encrypted	encrypted	reference only
Extra server	lightweight controller	none	Vault (heavy)
Rotation	automatic sealing keys	manual (re-encrypt)	native / dynamic
Dynamic (short-lived) secrets	no	no	yes
Fit with pure GitOps	excellent	excellent	good (references)
Operational curve	low	very low	high

For a sovereign 4×H100 getting started, SOPS/age or sealed-secrets cover 90% at minimal cost. When the system grows and requirements for frequent rotation and auditing appear, typical in ENS High Category scenarios, you migrate to ESO + Vault. The rotation rule: a secret that never rotates is a secret that, once leaked, is leaked forever. Rotation does not prevent the leak; it bounds its window of validity. We will come back to that when we talk about blast radius.

Layer 2 — Network: default-deny, egress and the arithmetic of the surface

The arithmetic of communication pairs

Without a network policy, in Kubernetes every pod can talk to every pod. That is the default posture, and it is the worst one for a sovereign system. With $N$ services, the number of possible ordered communication pairs (who-can-call-whom) is:

$$P_{\text{abierto}} = N \cdot (N-1) \approx N^2$$

With our $N = 8$ components (gateway, vLLM, TEI, vector database, state database, front end, plus the secrets controller and the observability one), that is $8 \cdot 7 = 56$ possible directed pairs. Fifty-six paths along which a compromised service could pivot laterally.

Now we apply a default-deny NetworkPolicy: nothing talks to anything except what is explicitly allowed. The real whitelist of an LLM stack is small. The strictly necessary pairs:

front end → gateway
gateway → vLLM
gateway → TEI
gateway → vector database
gateway → state database
front end → state database (sessions/users)

That is $E = 6$ edges. The communication surface falls from $56$ to $6$:

$$\frac{E}{P_{\text{abierto}}} = \frac{6}{56} \approx 0.107$$

Almost 89% of the possible paths are closed. The front end can no longer touch vLLM or TEI or the vector database directly; if somebody compromises the front end, they have no network route to the corpus. This is the difference between $\sim N^2$ and a whitelist $E \ll N^2$.

Standard L3/L4, L7 and egress with Cilium

Kubernetes’ native NetworkPolicy operates at L3/L4: it selects pods by label and allows/denies by port and protocol. That covers the bulk of the whitelist. But it has limits: it does not understand DNS or HTTP. This is where Cilium comes in, applying policies in the kernel via eBPF and extending them with CiliumNetworkPolicy:

Egress by FQDN/DNS: instead of pinning IPs (which change), allow egress only to huggingface.co so vLLM can download the model and block the rest. Critical for egress control.
L7: allow only certain HTTP methods/paths between the gateway and an internal service.
Enforcement modes: in the default mode, an endpoint with no policy selecting it has everything open; in always mode, everything is denied until a policy explicitly opens it. For a sovereign system, always + default-deny is the target posture.

Egress control is the sovereign piece par excellence. An internal service being unable to open connections to the internet except to a handful of explicit destinations means that, even if vLLM or the gateway is compromised, the corpus cannot leave the perimeter. The RAG is the asset of value; egress-deny is what stops it leaking out.

Internal mTLS

The NetworkPolicy says who can talk to whom, but it neither encrypts nor authenticates east-west traffic. For that, mTLS (mutual TLS: client and server authenticate each other). Cilium offers native mutual authentication; alternatives such as Linkerd or Istio give you a full service mesh. For a six-service stack, Istio’s full mesh is usually over-engineering; Cilium’s mutual authentication or Linkerd (lighter) is proportionate. The effect: an attacker on the cluster network cannot impersonate the gateway or eavesdrop on internal traffic in the clear.

Layer 3 — Pod security: restricted, and nothing privileged

The Pod Security Standards define three profiles: privileged (no restrictions), baseline (the reasonable minimum) and restricted (hardened). The target for the whole LLM stack is restricted, applied via Pod Security Admission with a namespace label in enforce mode. The restricted profile requires:

runAsNonRoot: true and a runAsUser other than 0. No container runs as root.
readOnlyRootFilesystem: true. The root filesystem is read-only; anything that needs to write goes to an emptyDir or an explicit volume. An attacker cannot leave persistent binaries in the container.
drop: ["ALL"] of Linux capabilities. No capabilities that are not needed.
seccomp RuntimeDefault, which applies the runtime’s profile and blocks dangerous syscalls.
allowPrivilegeEscalation: false, nothing privileged, no host namespaces.

The nuance with GPUs: pods using an H100 load NVIDIA’s device plugin, which historically tempted people to relax the securityContext. You do not need to run the inference container as privileged in order to use the GPU; device access is managed by the plugin, and the vLLM pod can and should run restricted. This is a fire partition: if vLLM is compromised, the attacker has a non-root process, with no capabilities, a read-only FS and trimmed syscalls — a terrible starting point for escalation.

Layer 4 — Image supply chain

The container image is third-party code running with access to your data. Three composed controls:

Pin by digest, not by tag. vllm/vllm-openai:latest is a moving target; @sha256:... is immutable. Pinning the digest guarantees that what was deployed is exactly what was audited.
Scanning with Trivy. Before promoting an image, Trivy enumerates its CVEs and composes the SBOM (component inventory). The pipeline fails if there are unmitigated critical vulnerabilities.
Signing with cosign and verification at admission. The Sigstore project (cosign for signing, Fulcio as a CA for ephemeral certificates via OIDC, Rekor as an immutable transparency log) allows the image to be signed. And Sigstore’s policy-controller, or Kyverno, is an admission controller that verifies the signature before admitting the pod: an image without a valid signature does not enter the cluster.

This is the doorman from the analogy: he checks each image’s credential (signature) at the door. A poisoned image uploaded to a registry, or a hijacked tag, stays outside because it carries no signature from the trusted issuer.

Layer 5 — AuthN/Z: virtual keys, OIDC and minimal RBAC

Three identity control points, from the outside in:

Front end (OIDC/LDAP). The human user authenticates against the corporate identity provider. The front end does not invent its own user system; it delegates to OIDC. This gives SSO, MFA and centralised revocation.
Gateway (LiteLLM virtual keys). The gateway issues virtual keys: each team, application or user has its own key with a budget, a rate limit and permitted models. The gateway’s master key, and the real API keys towards the engine, are never seen by the client; clients only handle their virtual key, revocable individually. If a virtual key leaks, that one and only that one is revoked.
Kubernetes RBAC with least privilege. Each service’s ServiceAccount has exactly the permissions it needs. The vLLM pod does not need to list Secret objects in other namespaces or create pods. Restrictive RBAC means a stolen service account token opens very little.

Layer 6 — Runtime: the cameras with Tetragon

The previous layers are preventive. What is missing is detection: what if something, despite everything, runs where it should not? Tetragon is observable security and runtime enforcement over eBPF, Kubernetes-aware. It hooks kernel events — process_exec, tcp_connect, security_file_open — with typical overhead below 1%, and it can move from observing (recording the event) to enforcement (killing the process or cutting the connection in the kernel, Sigkill).

The operational rule, which I set out in detail in the bubblewrap + Tetragon runbook: adopt first, block later. First you deploy in observation mode to build a baseline of the stack’s normal behaviour — which binaries vLLM executes, what the gateway connects to — without false positives. Only afterwards do you promote the clear rules to enforcement: kill any process that tries to read secret paths, or cut every tcp_connect to destinations outside the egress whitelist. Falco is the detection-only alternative over eBPF; Tetragon adds enforcement. These cameras also produce the audit evidence: what each service ran and which attempt was blocked.

Layer 7 — Data at rest

The last thing we protect is data standing still:

Storage encryption. The persistent volumes, where the vector database and the state database live, on top of disk encryption (LUKS/dm-crypt) or Ceph-level encryption. A disk stolen from the datacenter does not reveal the corpus.
etcd encryption. Remember that Kubernetes Secret objects, once decrypted by sealed-secrets or ESO, are normal objects in etcd. You have to enable etcd encryption at rest, or the secret sits in the clear in the control plane.
Database credentials. The vector database’s password and the state database’s are first-class secrets (layer 1), never embedded in the manifest or in environment variables in the clear in git.

Exposure table: service × port × external egress?

This table is the input for writing the NetworkPolicy resources. The egress column is the one that decides what leaves the perimeter.

Service	Internal port	Permitted callers	External egress?
front end	3080	(human ingress via OIDC)	No
gateway (LiteLLM)	4000	front end	No
vLLM	8000	gateway	Only `huggingface.co` for the initial download; zero in operation
TEI (embeddings/reranker)	8080	gateway	Only model download; zero in operation
vector database	6333	gateway	No
state database	5432	gateway, front end	No
secrets controller	—	(control plane)	No
observability	9090	internal scraping	No

The reading: no service needs external egress in normal operation. vLLM and TEI need it once to pull the model, and that permission can be temporary or restricted by FQDN to huggingface.co. Everything else is total egress-deny. If a service starts attempting outbound connections this table does not contemplate, Tetragon records it and, in enforcement, cuts it.

Blast radius: a leaked secret, with and without defence

The blast radius measures how much damage a compromise does. Let us model it for the realistic worst case: the state database’s credential leaks.

Without hardening (flat network, unrotated secret, no egress-deny, no detection):

The secret gives access to the state database from any pod (flat network → 56 open pairs).
The secret does not rotate → it is valid indefinitely; the exploitation window is $\infty$ until somebody notices.
Without egress-deny, the attacker dumps the whole database to an external server.
Without Tetragon, nobody finds out until the public incident.
Blast radius: the entire state database, exfiltrated, undetected, for an indefinite time.

With hardening (default-deny, ESO + Vault with dynamic credentials, egress-deny, Tetragon in enforcement):

The secret is only usable from the pod that has a network route to the state database (1-2 pairs, not 56).
With Vault’s dynamic credentials, the secret expires — say within a window $T$ of hours; past $T$, it is worthless.
Egress-deny prevents the dump to the internet: the attacker can read, but cannot take anything out.
Tetragon records the anomalous access and, in enforcement, kills the process attempting the outbound connection.

The qualitative reduction is enormous, but let us be quantitative about the window. If a static secret is valid forever and a rotated one with period $T$ is valid for at most $T$, and compromises arrive at rate $\lambda$, the expected number of live and exploitable secrets at any given instant goes from growing without bound to being capped at $\lambda \cdot T$. With rotation every 24 h ($T = 1$ day) against “never”, a given secret’s window falls from months to a day: a reduction of one to two orders of magnitude in temporal exposure. Combined with the reduction in network pairs ($56 \to \sim 2$, a factor of $\sim 28\times$) and egress-deny (from exfiltration possible to impossible by the direct route), the blast radius is drastically reduced.

But let us be honest: it does not reach zero. The attacker with the credential can read the state database during the window $T$ from the compromised pod. Hardening turned “indefinite, silent catastrophe” into “a bounded, detected incident with no exfiltration by the direct route”. That is exactly what defence in depth promises: not invulnerability, but that the cost of a full compromise is high and its radius small.

The ENS / NIS2 angle

These layers are not voluntary hygiene: they implement specific controls. The mapping is developed in detail in the technical controls ENS × ISO 42001 × EU AI Act post; here, the actionable summary:

Hardening layer	ENS measure (RD 311/2022)	NIS2 / framework link
Encrypted secrets + rotation	`op.exp.11` (cryptographic keys), `mp.info.3` (encryption)	Credential management; in High Cat., HSM
Default-deny NetworkPolicy + segmentation	`mp.com.1` (perimeter), `mp.com.4` (flow separation)	NIS2 art. 21: network security measures
Egress control	`mp.com.1` + `op.mon.1`	Exfiltration prevention
Internal mTLS	`mp.com.2-3` (confidentiality/integrity in transit)	Mandatory TLS
Restricted Pod Security	`op.exp.2` (hardened configuration)	Configuration hardening (CIS)
Trivy + cosign + admission	`op.exp.6` (malicious code), `op.ext.3` (supply chain)	NIS2 supply chain
OIDC + virtual keys + RBAC	`op.acc.1-2-5` (identification, access, authentication)	MFA in High Cat.
Tetragon runtime	`op.mon.1` (intrusion detection)	Monitoring and response
Encryption at rest + etcd	`mp.info.3` (encryption), `mp.si` (media)	Data at rest

The honest note for auditing: hardening reduces risk, it does not eliminate it, and maturity is demonstrated by prioritising by impact. A competent auditor does not want to see nine half-finished layers; they want to see that egress-deny on the critical asset (the corpus) and secrets management are solid before mTLS is perfect between low-sensitivity services. For the management and governance context, see also ISO/IEC 42001 as an AIMS and the EU AI Act mapping onto the architecture.

Applied to the generic 4×H100 cluster

In a real deployment on 4×H100 SXM 80GB, you do not harden the nine layers at once. The order, prioritised by impact:

Egress-deny on vLLM, TEI and the vector database, first. It is the barrier that stops the corpus leaving the perimeter, and it goes in as soon as the services start. Allow egress by FQDN to huggingface.co only during the model’s initial download; afterwards, zero. It is the layer with the highest return per hour invested.
Secrets for the inference engine and the databases, out of git. The Hugging Face token vLLM uses to download the model, the vector database’s and the state database’s passwords, the gateway’s master key: into sealed-secrets or, if there is already a rotation requirement, into ESO + Vault. Never in values.yaml in the clear.
Default-deny NetworkPolicy + the 6-edge whitelist. The gateway and the vector database are the most exposed services — the gateway because it receives all the traffic, the vector database because it holds the embedded RAG. Closing everything that is not on the whitelist cuts lateral pivoting from $\sim N^2$ to the 6 real edges.
restricted Pod Security in the inference namespace, including the vLLM pod with a GPU (it needs no privileges to use the H100).
cosign + admission so only signed images get in; Trivy in the GitOps pipeline of Flux.
Tetragon in observation, baseline, and then enforcement over the egress of the sensitive pods.
Encryption at rest for the databases’ volumes and for etcd.

The gateway and the vector database are the two I harden first inside the whitelist: the gateway for being the exposed face, the vector database for holding the asset that egress-deny protects. The rest is built on top, layer by layer, always assuming the previous one can fail.

What hardening does NOT solve

To close honestly, what these layers do not cover and what needs other pieces:

Prompt-level attacks (jailbreak, indirect injection via the RAG corpus): that is the job of guardrails and LLM Guard, not of NetworkPolicy.
Agents with legitimate permissions doing something harmful: the runtime isolation from the agent isolation post bounds what an agent can touch, but a granted permission is a usable permission.
The human factor: a well-managed secret shared over Slack is still leaked. Rotation bounds the window, it does not eliminate the mistake.
Zero-day vulnerabilities in the components themselves: Trivy detects what is known; the unknown gets through until the CVE is published.

Hardening is a multiplier on the cost of attacking, not an impassable wall. Its value lies in making an individual compromise bounded, detected and with no way out — and that, for a sovereign system holding sensitive data, is exactly the goal.

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

Sat, 06 Jun 2026 12:00:00 +0200

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

Disaggregated serving. The KV-cache that travels between the prefill pool and the decode pool moves over RDMA. These are large transfers, sensitive to latency and bandwidth, which in multi-node setups go out through the NIC.
Multi-node NCCL collectives. When tensor/pipeline parallelism crosses the node boundary, the all-reduce/all-gather operations no longer go over NVLink but over GPUDirect RDMA on the NIC.

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Three kernel facts the analogy compresses:

The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 piece	What it does	What Cilium eBPF puts in
`kube-proxy` (iptables/IPVS)	Service balancing	LB in eBPF; with `kubeProxyReplacement=true`, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)	VXLAN overlay + NetworkPolicy	native datapath (`routingMode=native`), L3/L4 and L7 NetworkPolicy in eBPF
veth per pod	namespace interface pair	netkit (kernel ≥6.8): namespace overhead ~0
host iptables traversal	netfilter hooks	eBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus = 2-31,34-63 # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33 # kubelet housekeeping (post 3)
IRQ affinity = 0-1,32-33 # NIC IRQs → housekeeping ONLY (this post)
 # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
 --namespace kube-system \
 --set routingMode=native \
 --set bpf.datapathMode=netkit \  # namespace overhead ~0 (kernel >=6.8)
 --set bpf.masquerade=true \
 --set kubeProxyReplacement=true \  # replaces RKE2's kube-proxy
 --set enableIPv4BIGTCP=true \  # 192k super-packets (mlx5/ice NIC)
 --set enableIPv6BIGTCP=true \
 --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
 --set bpf.mapDynamicSizeRatio=0.08 \
 --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
 name: gpu-nic-numa-aligned
spec:
 spec:
 devices:
 requests:
 - name: rdma-nic
 deviceClassName: dra.net  # NICs published by DRANET
 constraints:
 - requests: ["rdma-nic"]
 matchAttribute: "dra.net/pcieRoot" # same PCIe root as the GPU
 # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

References

Cilium, Tuning Guide (netkit, host-routing, BIG TCP, XDP, pin the IRQs and kill irqbalance): https://docs.cilium.io/en/stable/operations/performance/tuning/.
Cilium 1.19 (February 2026), Cilium at Ten Years — hardening of encryption, policies and observability: https://www.infoq.com/news/2026/02/cilium-119/.
Isovalent, Cilium 1.18 (IPv6, encrypted overlay, ingress bandwidth, policy perf): https://isovalent.com/blog/post/cilium-1-18/.
RKE2, Network Options (Canal by default; Cilium with kube-proxy replacement): https://docs.rke2.io/networking/basic_network_options.
Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/.
Kubernetes blog, v1.36: More Drivers, New Features, and the Next Era of DRA (May 2026): https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/.
DRANET (kubernetes-sigs), the network DRA driver and the paper The Kubernetes Network Driver Model (+59.6% all_gather / +58.1% all_reduce): https://github.com/kubernetes-sigs/dranet.
AKS Engineering, Optimizing RDMA performance for AI workloads on AKS with DRANET (April 2026): https://blog.aks.azure.com/2026/04/01/dranet-rdma-optimization-for-ai-on-aks.
Linux network tuning — IRQ affinity, RSS/RPS/RFS and NUMA softirq: https://andreaskaris.github.io/blog/networking/rss-irq-affinity-and-rps/.

Hubble: eBPF network observability, state of the art 2026 and the new frontier with AI agents

Tue, 19 May 2026 06:00:00 +0200

TL;DR

Hubble is Cilium’s native network observability, built on the same eBPF programs Cilium uses for enforcement. It does not duplicate the datapath or instrument the kernel its own way: it listens to the hooks Cilium already has and produces structured flow logs with Kubernetes context included, covering pod, namespace, labels, service, policy verdict and L7 payload where applicable. It is what happens when someone decides that tcpdump with grep does not scale to 10,000 pods and builds a distributed system of their own (a Hubble server per node, plus Hubble Relay as the aggregator, plus a CLI and a UI) with practically zero overhead, because the capture was already happening. In 2026 it is at version 1.19.3 (April 2026), with Cilium 1.19 marking the project’s tenth anniversary; IP options tracing has arrived, along with filtering by encryption status, the drop event tagged with the exact NetworkPolicy that caused it (direct attribution), a stabilised field mask API, and the first wave of ML anomaly detection applied to flows for predictive security in IoT/5G clusters. And, most interesting of all for 2026: a new frontier appears where the same eBPF observes AI agents such as Claude Code, Gemini CLI and MCP agents, intercepting SSL/TLS and stdio without instrumenting the code, which turns the Cilium + Hubble + Tetragon + AgentSight stack into a complete set of tools for understanding what an agentic system does inside a cluster.

This article is part 3 of the eBPF series. Part 1: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack. Part 2: Tetragon: Cilium’s security cousin that sees every syscall in the kernel. Here we complete the observability quadrant: network with Hubble, process with Tetragon, AI agent with AgentSight.

The analogy: tcpdump that speaks Kubernetes

If you have administered networks over the last twenty years, tcpdump and Wireshark have been your daily bread. They capture packets on an interface, parse them, and let you filter with tcp.port == 443 and host 10.0.0.5. They work, they have been working since the 90s, and they are the first thing you open when something smells wrong.

Now stick tcpdump on a Kubernetes cluster with 10,000 pods. The problems show up in order:

One tcpdump session per node. You wanted to “see the traffic between the frontend and the API”; you need SSH to every node, tcpdump on every NIC, synchronised timestamps, manual aggregation.
There is no K8s context. You see a packet from 10.244.5.7 to 10.244.8.42. Which pod was that? Which namespace? Which label? You have to correlate with kubectl get pod -A -o wide every single time.
No L7 understanding. You see a POST to HTTPS, and you cannot know the method and path because it is encrypted on the wire. If there is mTLS between pods, worse.
High cost: full packet capture with a copy to userspace slows down the datapath. Under dense traffic, you notice it.

Hubble is tcpdump redesigned for all of that. It reuses the eBPF programs that are already processing every packet (Cilium puts them there for enforcement) and, while they take their allow/deny decision, they emit a flow event with all the context: source and destination pod identity, namespace, labels, protocol, verdict, and, if Cilium has done L7 parsing via Envoy, the HTTP method, path, status code, DNS query, Kafka topic. That event travels through a ringbuffer to userspace, it is received by the Hubble server that lives inside the node’s Cilium agent, and it is exposed over gRPC. A separate service, Hubble Relay, aggregates the streams from all nodes and gives you a single cluster-wide API. On top of that: a CLI (hubble) and a web UI with a real-time service graph.

Zero additional copies. Zero duplicated parsing. And the result is understandable by anyone who knows what a Pod is.

Architecture: four pieces visible from outside

Hubble is made up of four logical components, all optional depending on what you want to do:

1. Hubble Server (embedded in every Cilium agent)

It lives inside the Cilium agent process (it is not a separate binary). Each node exposes a gRPC endpoint locally on the Unix socket /var/run/cilium/hubble.sock. The server listens to the events the eBPF programs emit to the ringbuffer, enriches them with Kubernetes metadata (which the agent already has in memory), and makes them available to consumers.

Enabling it: --set hubble.enabled=true in Cilium’s Helm chart. By default, the server is only accessible locally; if you want to consume it from another node, it has to be exposed (which is what Hubble Relay does).

2. Hubble Relay (aggregator)

It is a separate Deployment (typically 1 replica, scalable) that connects to every Hubble server in the cluster and aggregates their streams into a single API. When your CLI or UI asks for “the last 1000 flows in the cluster”, the Relay collects them in parallel from all nodes and returns the union.

Enabling it: --set hubble.relay.enabled=true. Without the Relay you only see the traffic of the node you are connected to, which is useful for local debugging but not for a cluster-wide view.

3. Hubble CLI (`hubble`)

A Go binary that speaks gRPC with the Relay (or with a local Hubble server). It supports two main modes:

hubble observe: a real-time stream of flows, with very expressive filters (by namespace, pod, port, verdict, protocol, label).
hubble status: the state of the Hubble cluster (how many nodes connected, lag, flow rate).

And the equivalent of tcpdump’s pcap dump: hubble observe --output jsonpb > flows.json to process afterwards with jq or other tools.

4. Hubble UI

A web frontend that connects to Hubble Relay and shows:

A service graph in real time (which Pod talks to which Service, which protocols it uses, which verdict).
A filterable list of flows.
L7 details where there are any (HTTP method/path/status, DNS query/response).

Enabling it: --set hubble.ui.enabled=true. Useful for presentations to non-CLI teams; it does not replace the CLI for serious debugging.

What you see: the Hubble flow log from the inside

A Hubble flow in JSON format looks roughly like this (simplified):

{
 "time": "2026-05-19T03:12:45.182Z",
 "verdict": "FORWARDED",
 "source": {
 "ID": 5482,
 "identity": 24871,
 "namespace": "prod-api",
 "labels": ["app=checkout", "team=payments"],
 "pod_name": "checkout-7c9f-x8j2",
 "workloads": [{"name": "checkout", "kind": "Deployment"}]
 },
 "destination": {
 "ID": 12041,
 "identity": 18356,
 "namespace": "prod-db",
 "labels": ["app=postgres", "tier=primary"],
 "pod_name": "postgres-0"
 },
 "Type": "L3_L4",
 "l4": {
 "TCP": {
 "source_port": 41982,
 "destination_port": 5432,
 "flags": {"SYN": true}
 }
 },
 "node_name": "rke2-worker-03",
 "Summary": "TCP Flags: SYN"
}

When L7 parsing is active (via embedded Envoy or Hubble’s lightweight parser), the same flow adds:

"l7": {
 "type": "REQUEST",
 "http": {
 "code": 200,
 "method": "GET",
 "url": "/api/v1/cart/items",
 "protocol": "HTTP/1.1",
 "headers": [{"key": "user-agent", "value": "checkout/1.4.2"}]
 }
}

The protocols natively supported for L7 parsing:

HTTP/1.1 and HTTP/2 (including gRPC over HTTP/2).
DNS (queries and responses, with domains, types, response codes).
Kafka (topics, API keys).
TLS handshake (SNI, not the encrypted payload by default).
MySQL, Cassandra (with optional modules).

For HTTP and gRPC, Cilium can enable the embedded Envoy proxy for the flows you want to inspect (not all of them; it is selective via CiliumNetworkPolicy with L7 rules). Without Envoy there is lightweight parsing, but less detailed.

Verdict and drop attribution

Every flow has a verdict: FORWARDED, DROPPED, ERROR, AUDIT, REDIRECTED, TRACED, TRANSLATED. For the DROPPED case, Hubble includes a structured reason (drop_reason) and, since Cilium 1.19, the exact NetworkPolicy that caused it.

That last point changes day-to-day operations. Before, when a pod could not talk to another one, the debug flow was:

See the dropped flow in Hubble.
Look at every CiliumNetworkPolicy in the namespace.
Reason manually about which one, with which labels, is blocking it.

With Cilium 1.19’s attribution, the policy_match_info field tells you directly “it was dropped by the frontend-egress policy, rule 3”. You go from “Sherlock Holmes for 20 minutes” to “kubectl get -o yaml of that specific policy”.

Prometheus metrics and Grafana dashboards

Hubble also exposes aggregated metrics in Prometheus format, separate from the gRPC flow stream. Enabling it: --set hubble.metrics.enabled=true (Helm) plus a list of the set you want to export.

The usual metric groups:

flow: total flows by verdict, source/dest, protocol.
http: requests by method, response code, latency (histogram).
dns: queries, response codes, top-N domains.
tcp: handshakes, retransmissions, congestion window.
drop: drops by reason, with NetworkPolicy attribution.
port-distribution: histogram of active ports.
policy: hits by policy and verdict.

These metrics have rich K8s labels (source_workload, destination_workload, namespace, and so on) that make them pivotable in Grafana. There are prebuilt dashboards on Grafana Labs covering the common cases; importing one and having immediate visibility costs five minutes.

Cost: metrics with many K8s labels can blow up cardinality in Prometheus. For large clusters (>1,000 pods), it is worth reviewing which set you export and using drop rules in Prometheus to keep a lid on it.

Deployment: Helm on one screen

The canonical Cilium install with full Hubble:

# values.yaml
hubble:
 enabled: true
 metrics:
 enabled:
 - dns:query;ignoreAAAA
 - drop
 - tcp
 - flow
 - port-distribution
 - icmp
 - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
 serviceMonitor:
 enabled: true # auto-discovered by kube-prometheus-stack
 relay:
 enabled: true
 rollOutPods: true
 ui:
 enabled: true
 rollOutPods: true
 ingress:
 enabled: true
 className: cilium
 hosts:
 - hubble.example.local

And the install itself:

helm upgrade --install cilium cilium/cilium -n kube-system -f values.yaml

After installing, validate with cilium status (the Cilium CLI) that the Hubble section shows OK, and try your first flow with:

cilium hubble observe --namespace prod-api --pod checkout-7c9f-x8j2

State of the art in 2026

Cilium 1.19 was released in February 2026, marking the project’s tenth anniversary. Hubble reached version 1.19.3 on 22 April 2026. The relevant news:

Direct attribution of drops to a NetworkPolicy

Already covered above; it is probably the most valuable operational change in the release. Any dropped flow carries the name, namespace and specific rule of the policy responsible. It is also available via Prometheus metrics, which allows alerts of the form “policy X is dropping >N requests/second”.

Tracing with IP options

Hubble can now trace individual packets with IP options enabled. It is a mechanism similar to traceroute, but at L3: you put a mark on the packet and Cilium reports it every time the packet crosses a node or an eBPF decision. Useful for debugging multi-cluster paths, fabric mesh, or NetworkPolicies applied at different layers.

Filtering by encryption status

A new CLI flag: hubble observe --encryption-status=encrypted (or unencrypted). Useful for validating deployments with WireGuard or IPsec enabled pod-to-pod: you confirm that the traffic that should be encrypted is, and you spot regressions quickly.

Hubble field mask API stabilised

The field_mask lets you ask for only the parts of the flow you care about, hugely reducing bandwidth and processing when all you need is, say, source/dest and verdict. It used to be experimental; now it is stable and it is default-on in the CLI.

AI-driven anomaly detection (predictive security)

This is the most talked-about addition of 2026. Cilium 1.19 adds hooks so that an external consumer, typically an ML system, can process flows in streaming and detect statistical anomalies: pods that suddenly talk to new destinations, latency spikes on an API, odd DNS sequences. The detection part happens outside the Cilium agent (nobody wants heavy ML in the datapath), but Cilium exposes the flows with the pre-computed features the model needs. The published use cases focus on IoT and 5G, where traffic is high in volume and low in variety, ideal conditions for anomaly detection.

Scaling to 10,000+ pods

Cilium 1.19 has done serious work on scalability: Hubble Relay can now aggregate streams from hundreds of nodes without saturating; the default field_mask reduces inter-node bandwidth; and flows can be sampled under high load if your use is statistical analysis rather than forensic debugging.

Cilium 1.20 in development

Cilium 1.20 is in the development branch. The most relevant bits for Hubble:

Unification of preferIpv6: the hubble.preferIpv6 flag was deprecated in favour of the global preferIpv6 that applies to every Cilium component.
tetragon-python SDK: although it belongs to Tetragon, not Hubble, it sets a trend, namely eBPF policies written in Python instead of YAML. Hubble will probably follow a similar path.

The new frontier: eBPF and AI agents

So far, the classic Hubble content. But there is a 2026 twist worth covering, because it closes the loop with the other series on this blog.

When a Kubernetes cluster starts running AI agents, such as Claude Code, Gemini CLI, or LangGraph-based agents calling APIs and MCP servers, the observability problem changes shape. It is no longer enough to know “which pod talked to which pod” (that is Hubble) or “which process ran what” (that is Tetragon). You need to know:

Which external APIs the agent is calling and with which prompts.
Which MCP tools it is invoking, with which arguments.
How many tokens it consumes, which model it picks, how much it costs.
Whether the agent drifts from expected behaviour (out-of-policy queries, jailbreak attempts, secret leakage).

The traditional solutions, instrumenting the agent’s code with OpenTelemetry or parsing structured logs, do not work well when the agent is a third-party binary (Anthropic’s Claude Code, Google’s Gemini CLI) or when the MCP servers live in other languages with stdio as their transport.

AgentSight: zero-instrumentation for LLM agents

AgentSight (a project from the eunomia-bpf group, the same ecosystem behind several high-profile eBPF runtimes) attacks this problem with the same philosophy as Hubble: do not instrument; listen. It puts eBPF hooks at two critical points:

uprobes on SSL/TLS libraries (libssl, boringssl, rustls). It captures the plaintext before encryption on send and after decryption on recv. For an HTTP call to https://api.anthropic.com/v1/messages, AgentSight sees the full JSON of the prompt and the response without decrypting anything in transit, simply because it has reached the syscall level before the TLS layer does its work.
stdiocap BPF: it captures read, write and dup on a process’s stdin/stdout/stderr file descriptors. This is what makes it possible to observe MCP servers that speak stdio with their client, the usual pattern for local MCP servers. You capture the JSON-RPC going back and forth without either the client or the server knowing.

Reported overhead: <3% CPU, comparable to Hubble in its own regime.

How it fits with Hubble and Tetragon

The three complement each other cleanly:

Hubble tells you: “the agent’s pod opened a TCP connection to api.anthropic.com:443 with verdict ALLOW”.
Tetragon tells you: “the claude-code process with PID 1843 did a connect() to that IP” (plus the binary, the arguments, the pod namespace).
AgentSight tells you: “the HTTPS content of that connection was a prompt messages=[{role:'user', content:'analyze this repo and modify the firewall config'}] and the response included a tool call to read_file with argument /etc/passwd”.

It is the difference between flow, process and semantics. For a security team that wants to watch AI agents in production, all three are necessary. For anyone who wants to understand cost, all three are useful (Hubble for network latency, Tetragon for resource usage, AgentSight for tokens and chosen model).

Emerging use cases

The patterns consolidating in 2026:

Agent audit trail: recording every LLM call and every tool call for compliance, especially in regulated sectors.
Jailbreak and prompt injection detection: applying rules over the prompts captured by AgentSight (similar to Tetragon’s TracingPolicy, but over semantic content).
Cost accountability: seeing which team/agent consumes which tokens, without instrumenting.
Replay and debugging: reproducing an agent’s reasoning in production without asking it to run again (which is non-deterministic).

It is a young field, AgentSight is months old rather than years, but the “eBPF as zero-instrumentation observability” pattern is very clearly spreading beyond network and process. The coming year will see consolidation and, probably, native integration with Hubble.

Common Hubble use cases

Back to Hubble proper, the cases in which any organisation deploys it:

1. NetworkPolicy debugging

The classic use: “this pod cannot reach this Service”. Without Hubble, it meant SSH, tcpdump, comparing rules. With Hubble:

hubble observe --from-pod prod-api/checkout --to-pod prod-db/postgres --verdict DROPPED

If there are drops, you see the policy responsible (Cilium 1.19+). If there are no drops, the problem is not policy: it is DNS, routing or the target service.

2. Auditing inter-namespace communication

For compliance: validating that isolated namespaces are not communicating against what was declared.

hubble observe --from-namespace prod-payments --to-namespace 'NOT prod-db' --output json

3. Exfiltration detection

Outbound traffic to suspicious public destinations. Hubble detects them by IP/SNI, not by payload (which is encrypted):

hubble observe --to-fqdn 'NOT *.example.com' --to-fqdn 'NOT *.internal' --protocol tcp

Combined with Prometheus metrics and Grafana alerts, this gives you an exfiltration radar at zero cost.

4. Real-time service SLOs

The hubble:http:response_time_seconds metrics with source_workload, destination_workload, method and status_code labels allow SLO dashboards without needing to instrument the apps. The SRE sees the p95 latency of checkout → catalog directly.

5. Performance debugging

hubble:tcp:retransmissions_total and hubble:tcp:flags_total{flag="RST"} are early signals of network problems. A rise correlated with a latency regression points you at something in the infrastructure (NIC, switch, MTU) before you go down to investigate the app.

6. Post-incident forensics

Configuring Hubble to export flows to persistent storage (via OTLP to Tempo/Loki, or hubble observe --output jsonpb to S3) gives you forensic capability: if at T+30 days you detect that something was wrong at T, you can reconstruct the traffic.

Hubble and the rest of the observability stack

Hubble does not replace Prometheus, Loki, Tempo or Jaeger; it complements them:

Prometheus: receives Hubble’s aggregated metrics. Hubble exports a native Prometheus endpoint.
Loki: receives the structured flow logs if you export them as logs. Hubble has no native Loki exporter, but a Fluent Bit with an OTLP plugin or a custom one bridges it easily.
Tempo / Jaeger: the Cilium Operator has an OTLP exporter of flows in trace format (each HTTP/gRPC flow is a span). It integrates with Tempo or any other OTLP tracing backend.
Grafana: there are already public Hubble dashboards. Combined with Prometheus, Loki and Tempo, they give you a unified panel: metrics, logs, traces, all correlated by K8s labels.

The full-stack pile seen in production in 2026 (described in Building a Production eBPF Observability & Security Stack for Kubernetes in 2026):

Data: Cilium + Hubble (network), Tetragon (process), AgentSight (AI agent).
Pipeline: OTLP Collector as the single router.
Storage: Prometheus (metrics), Loki (logs), Tempo (traces).
UI: Grafana with domain-specific dashboards.
Alerting: AlertManager with rules over the Hubble + Tetragon metrics.

Comparison with the alternatives

System	Layer	Focus	Model
Hubble	L3-L7 network	K8s cluster with Cilium	eBPF, pull metrics, push flows gRPC
GKE Dataplane v2 obs	L3-L7 network	GKE managed	eBPF (Cilium-based, managed)
Tigera Calico Whisker	L3-L7 network	Cluster with Calico	eBPF + pcap, own UI
Tetragon	Process/syscall	K8s cluster	eBPF, push events gRPC
Falco	Process/syscall	K8s cluster	eBPF in userspace or kernel module
AgentSight	LLM agent	Agentic systems	eBPF (SSL uprobes + stdio)
Beyla (Grafana)	Application	App L7 + tracing	eBPF (uprobes on libs)
Pixie	App + system	Broad cluster visibility	eBPF + PXL script
Parca	CPU/mem profiling	Performance	eBPF profile sampling

If your CNI is Cilium, Hubble is the natural entry point and it does not compete with the rest: it complements them. For Calico clusters, Whisker is the equivalent. For profiling, Parca. For AI agents, AgentSight. The era of “one tool for everything” is passing: the modern stack combines several specialised pieces, all eBPF-based, exposed via OTLP.

Operational traps

Cardinality in Prometheus

Hubble metrics with all the K8s labels can blow up Prometheus. Measure cardinality before exporting everything. The most prolific metrics are flow and httpV2; start with drop and port-distribution and add the rest incrementally.

L7 visibility costs CPU

Enabling L7 parsing via embedded Envoy adds load to the agent (not to the base datapath, but yes to the node’s envoy proxy). For heavy HTTP traffic, measure. For flows where you only need L4, leave Envoy disabled.

Hubble Relay without HA

A single Relay replica is a single point of failure for the CLI and the UI (not for the local agent, which keeps working). For production, deploy with replicas: 2+ and topologySpreadConstraints so both do not go down together.

Encryption status reporting depends on the Cilium config

The new --encryption-status filter only gives real data if Cilium has encryption enabled (WireGuard or IPsec). Without that, everything is unencrypted and the filter contributes nothing.

UI exposed without auth

Hubble UI has no native auth. If you expose it through Ingress, there has to be authentication in front of it: OIDC via oauth2-proxy, mTLS, IP allowlist. It is not optional.

Storage that does not scale

If you keep flows for days for forensics, the volume is serious. For a cluster with 100 active pods, easily 1-10 GB/day of flow logs. Plan the lifecycle (compaction, retention, cold storage) before enabling it.

What we have not covered

Mesh / multi-cluster Hubble: aggregating flows from several Cilium clusters into a single Relay. Use case: a cross-cluster view, debugging a distributed service mesh.
hubble export: local persistence on the agent’s disk for forensics with low retention.
Anomaly detection with your own models: how to connect the gRPC stream to a custom ML consumer.
AgentSight in depth: the project deserves its own article. Next instalment.
eBPF for LLM serving profiling: how to measure vLLM’s TTFT, TPOT and throughput without instrumenting, using uprobes on libcudart.

References

Hubble and Cilium:

Hubble GitHub — main repo.
Hubble — Network Observability (Cilium docs) — official reference.
Cilium 1.19 release notes (InfoQ, feb 2026) — tenth anniversary and 1.19 news.
Cilium releases — all releases.
Hubble L7 HTTP Metrics — Grafana dashboard 19423 — ready to import.
End‑to‑end L7 Visibility with Cilium Hubble (cloud-cod.com, mar 2026).
Cilium Hubble Observability Platform Internal Analysis (Young-ju).
CiliumNetworkPolicy Python Hubble: L7 Visibility 2026 — one of the threads on the Python SDK.

State of the art 2026 and the full stack:

eBPF + AI agents:

AgentSight (GitHub eunomia-bpf) — the referenced project.
Harnessing eBPF for High‑Performance LLM Workloads (Klizo Solutions).

Cross-references:

Part 1: eBPF from zero to Cilium.
Part 2: Tetragon: Cilium’s security cousin.
LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators — where the traffic Hubble observes carries the prompts AgentSight inspects.

Tetragon: Cilium's security cousin that sees every syscall in the kernel

Tue, 19 May 2026 05:00:00 +0200

TL;DR

Tetragon is the runtime security and observability engine that the Cilium project published as a companion to the CNI. Its job is not to route packets, Cilium is already there for that, but to observe what happens inside the node’s processes in real time: which binary runs in each pod, which files it opens, which syscalls it invokes, which capabilities it asks for, which network connections it establishes, which kernel modules get loaded. It does this by loading eBPF programs into the kernel’s hook points (kprobes, tracepoints, uprobes, LSM hooks) and filtering the relevant events inside the kernel itself with a declarative language expressed as a CRD (TracingPolicy and TracingPolicyNamespaced). The result is a stream of events enriched with Kubernetes metadata (pod, namespace, labels) that costs less than 1% of CPU and, the thing that sets Tetragon apart from the competition, can block actions inside the kernel, killing the process with SIGKILL or overwriting a syscall’s return value, before they finish executing, with no race conditions. Against Falco (which parses syscalls in userspace, 5-10% overhead, detection-only), Tetragon is “cheaper and with enforcement”; against the bare kernel, it is “a declarative layer your operations colleague can read”. This article is the extensive introduction you need to take it on seriously: architecture, all the hooks and selectors, the operating modes, a guide to use cases (exec auditing, sensitive file access, container escape, cryptomining, rootkit detection, network observability) and the traps you see in production.

This article is part 2 of the eBPF series. Part 1, eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack, covered basic eBPF, the networking hooks (XDP, TC, sock_ops), how Cilium implements the datapath and the BGP Control Plane v2 CRDs. Here we take those same eBPF hooks and use them for something different: observing and, if needed, stopping what the cluster’s processes do.

The analogy: auditd on steroids in eBPF

Anyone who has spent a few years administering Linux has used auditd. It is the classic kernel subsystem for auditing syscalls: you configure a rule with auditctl (for instance, “monitor any open on /etc/shadow”) and the kernel sends events to a userspace daemon that persists them. It works, but it has two limitations that weigh heavily on modern Kubernetes clusters:

No Kubernetes context. auditd reports processes by PID and UID. Knowing which pod, which namespace, which image, which labels, the information that actually matters when responding to an incident, requires correlating afterwards with data from cri-o or containerd. It is operationally miserable.
No granular enforcement. auditd can generate events, but it cannot take the decision to kill the offending process before the syscall finishes. You leave that to a higher layer that reads the events, processes them and kills the process… if it gets there in time. A race by design.

Tetragon is auditd on steroids: the same conceptual ideas, hooks on syscalls and events to userspace, but implemented with modern eBPF, with filtering inside the kernel so you do not pay the cost of waking the daemon for every irrelevant syscall, with Kubernetes metadata injected by an agent that knows the cluster, and with actions executed inside the kernel itself without waiting for userspace to decide. If the rule says “kill any process that opens /etc/shadow from the prod namespace”, the decision is taken in the kernel kprobe and SIGKILL is delivered before the open completes. There is no race; there is no window between detection and action.

What Tetragon is, architecturally

Tetragon is an agent deployed as a DaemonSet (one pod per node) and a set of CRDs that define the policies to apply. The agent has four responsibilities:

Load eBPF programs into the hook points the active TracingPolicies demand.
Maintain a cache of Kubernetes metadata (pods, namespaces, labels) by reading the API server, so it can enrich every event with the right context.
Collect the events the eBPF programs emit (via ring buffers) and serialise them.
Export the events to configurable destinations: stdout JSON (typical in sidecars or log-collection agents), gRPC streaming (to consume them from Hubble or another consumer), a file, or Fluentd/Loki/SIEM.

The eBPF programs are not written by the user. Tetragon generates the bytecode from the TracingPolicies: it reads the declarative policy, decides which hooks to attack, which arguments to read from the kernel, which filters to apply inline and which actions to execute. The user only writes YAML.

This separation of declarative policy → generated eBPF bytecode is what makes Tetragon usable. Writing eBPF programs by hand is a specialist’s job; writing a TracingPolicy is the job of an SRE with a good example in front of them.

The two CRDs: TracingPolicy and TracingPolicyNamespaced

Tetragon exposes exactly two main CRDs:

TracingPolicy (cluster-scoped, cilium.io/v1alpha1): applies to the whole cluster, every node, every pod. Suitable for platform policies (the whole cluster must be audited the same way): for example, “log every execve in every pod” or “kill any process that tries to load a kernel module”.
TracingPolicyNamespaced (namespaced, same group and version): defined inside a namespace and applied only to the pods of that namespace. Suitable for policies with per-tenant autonomy: for example, “in the prod-payments namespace, kill any outbound connect to an IP outside the corporate range”.

Both CRDs have exactly the same internal structure. The difference is one of scope. The distinction was introduced precisely to allow multi-tenancy: the central security team defines cluster-wide TracingPolicy objects and each tenant can add its own with TracingPolicyNamespaced without needing cluster-admin permissions.

Anatomy of a TracingPolicy

A policy is made up of:

Hook points: which kernel events to observe.
Arguments: which data to read when the hook fires.
Selectors: filters evaluated inside the kernel to discard irrelevant events and, optionally, execute actions when they match.

Supported hook points

The official documentation lists five families of hook points:

kprobes: hook a kernel function. Syscalls are a particular case (when syscall: true) because their ABI differs from that of internal functions. Typical examples: sys_open, sys_openat, sys_connect, tcp_connect, do_mount, commit_creds. It is the most versatile hook and the one used 80% of the time.
tracepoints: hook static tracepoints compiled into the kernel. More stable across kernel versions than kprobes (they do not depend on function names that can change). Examples: syscalls/sys_enter_openat, sched/sched_process_exec.
uprobes: hook functions in userspace libraries or binaries. They serve to observe runtime primitives such as libssl functions, libc, the Go runtime, the JVM.
USDT tracepoints (User Statically Defined Tracepoints): static tracepoints defined in userspace binaries (like those MySQL, PostgreSQL and OpenJDK expose). Useful for application observability.
lsmHooks (LSM, Linux Security Module): hooks of the LSM subsystem, where SELinux/AppArmor plug in. They allow security policies very similar to traditional MAC but programmable with eBPF. Example: file_open, inode_unlink, socket_bind.

Arguments

Every hook can read the arguments of the function it is attached to. The supported types cover the primitives (int, uint64, bool, string, char_buf) and higher abstractions (file, path, sock, linux_binprm, capability, bpf_attr, cred). The high-level types are pointers to kernel structures that Tetragon knows how to parse; instead of having to read an offset, you write type: file and Tetragon gives you the full path of the descriptor’s file.

There is an important capability detail: on kernels ≥ 5.4, Tetragon can read up to 327,360 bytes of an argument if the large buffers flag is enabled. That is the difference between being able to audit execve with all of its long argv complete versus truncating them at 256 bytes and losing context.

Selectors: filtering in the kernel

Selectors are what make Tetragon cheap. Without them, every syscall on the node would fire an event that would travel kernel → ring buffer → agent → processed → filtered → discarded. With selectors, the filtering happens inside the eBPF program itself, in the kernel, and only the events that matter reach userspace.

The available selectors include:

matchArgs: filters by an argument’s value. Operators: Equal, NotEqual, Prefix, Postfix, GreaterThan, LessThan, Mask, SPort (source port), DPort (dest port), Family (AF_INET vs AF_INET6), State (socket state).
matchPIDs: filters by PID; useful for targeted observation.
matchBinaries: filters by the binary executing the syscall (absolute path), with Operator: In, NotIn, Prefix. Essential for avoiding noise from legitimate system processes.
matchNamespaces: filters by Linux namespace (Pid, Mnt, Net, Ipc, Cgroup, User). It allows policies specific to processes in containers versus the host.
matchCapabilities: filters by the process’s effective capabilities. Block actions requiring CAP_SYS_ADMIN that run in pods that should not have them.
matchNamespaceChanges: detects namespace changes (typical of container escape).
matchCapabilityChanges: detects capability changes (privilege escalation).
matchActions: the actions executed when all the preceding matchers hit.

Actions: from a simple Post to Sigkill

When a selector matches, an action is executed. Tetragon defines several:

Post: emits an event to userspace (the observability case). It supports rateLimit to avoid flooding the agent if the condition fires a thousand times per second. The syntax accepts 5 for 5 seconds, 5m for 5 minutes, 1h for 1 hour.
Sigkill: sends SIGKILL to the offending process from inside the kernel, before the syscall completes. This is the only thing that guarantees enforcement without a race.
Override: overwrites the syscall’s return value. Useful for making the process believe the syscall failed (Override -EPERM) without killing it. A better experience for apps that can handle errors; worse for apps that assume success.
Signal: sends any arbitrary signal (not just SIGKILL).
NoPost: does not emit an event, useful when combined with another selector that does emit and you only want the action without duplicated telemetry.
FollowFD and UnfollowFD: mark a file descriptor to follow its lifecycle and enrich subsequent events with the original path. Useful for auditing “which process read this file after opening it”.
TrackSock and UntrackSock: the same for sockets.
GetUrl and DnsLookup: make HTTP requests or DNS resolutions from the kernel. Designed for integrations with external systems (security webhooks, IP reputation lookups).
NotifyEnforcer and CleanupEnforcerNotification: communication with Tetragon’s enforcement subsystem for complex actions.

Modes: detection vs enforcement

A policy can be declared in one of two explicit modes:

enforce: enforcement actions (Sigkill, Override, Signal) are active. This is production.
monitoring: enforcement actions are ignored; only Post events are emitted. This is the “let us see what would happen if this were switched on” mode, critical for testing policies without breaking applications.

Control is done with the spec.options[].name: policy-mode field and value: monitoring or enforce. It is the best practice: start in monitoring, collect events for days, tune the selectors until no false positives come out, and only then switch to enforce.

Full example: blocking writes to `/etc/passwd` in the prod namespace

A realistic policy, commented line by line:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-passwd-write
 namespace: prod
spec:
 kprobes:
 - call: "fd_install"
 syscall: false # kernel function, not a syscall
 args:
 - index: 0
 type: int  # file descriptor
 - index: 1
 type: "file" # struct file*
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values:
 - "/etc/passwd"
 matchActions:
 - action: Sigkill  # kills the process
 rateLimit: "1m" # max once per minute
 options:
 - name: policy-mode
 value: enforce  # enforcement mode active

fd_install runs every time a process obtains a new file descriptor; the second argument is the file’s file struct. Tetragon knows how to resolve it to its absolute path. The matchArgs compares that path with /etc/passwd. If it matches, Sigkill kills the process before the descriptor even becomes usable. rateLimit: 1m stops the agent from saturating if a malicious application tries it in a loop.

Common use cases

Now to real use. These are the six cases that show up in any serious Tetragon deployment in 2026.

1. Execution auditing (`execve`)

The most basic use case and, even so, the most valuable. Which binaries are running in each pod? In a container that is supposed to run only nginx, suddenly seeing an sh or a wget is almost always a red flag.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-execve
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm  # struct linux_binprm*
 selectors:
 - matchActions:
 - action: Post  # events only, no enforcement

With no filters: every execve in the cluster generates an event. With K8s metadata, the event includes pod, namespace, container, image, labels. You turn it into a stream of events towards your SIEM and set up rules: “alert if I see sh, bash, nc, curl, wget or python running in any pod of the prod-api namespace”.

An enforcement variant: instead of Post, use matchBinaries with Operator: NotIn and a whitelist, plus Sigkill if the binary is not on the list. A very rigid box, but effective in pods that are “single-binary” (like a Go microservice).

2. Sensitive file access

Detecting (or blocking) reads and writes on critical files: /etc/shadow, /etc/kubernetes/, Secret mounts, /var/run/docker.sock, /proc/*/cmdline.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: sensitive-file-access
spec:
 kprobes:
 - call: "security_file_open" # LSM-ish via kprobe
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/etc/shadow"
 - "/var/run/secrets/"
 - "/var/run/docker.sock"
 matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/kubelet" # legitimate kubelet access
 matchActions:
 - action: Post

The matchBinaries: NotIn is important: kubelet and other legitimate node agents access these paths constantly and would generate noise. We filter those out in the kernel.

In enforcement: swap Post for Override with argError: -1 (EPERM), so that the open fails but the offending process stays alive and produces the error so that tracing tools pick it up.

3. Unauthorised outbound network connections

Detecting outbound connections to destinations outside the corporate range. Useful for spotting data exfiltration or malware command-and-control.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-external-egress
 namespace: prod
spec:
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr" # destination NOT in these CIDRs
 values:
 - "10.0.0.0/8"
 - "192.168.0.0/16"
 - "172.16.0.0/12"
 matchActions:
 - action: Sigkill
 options:
 - name: policy-mode
 value: enforce

This kills any attempt at a TCP connection to an IP that is not in the corporate CIDRs, in the prod namespace. Cilium already does this with NetworkPolicy, but Tetragon has two complementary advantages:

It gives you the process that attempted the connection, not just “pod X tried to connect to Y”.
It also works for exotic protocols where NetworkPolicy is less expressive.

4. Container escape detection

Container escape is the operational nightmare: a process inside a container manages to break the isolation (via a kernel exploit, a badly set capability, a misconfigured hostPath mount) and gain access to the host. Three typical signals:

A namespace change on the process (it leaves the container’s pid namespace).
setns or unshare in non-init processes.
Access to /proc/1/root or /dev/ from a container.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-container-escape
spec:
 kprobes:
 - call: "__x64_sys_setns"
 syscall: true
 args:
 - index: 0
 type: int
 - index: 1
 type: int
 selectors:
 - matchNamespaces:
 - namespace: Pid
 operator: NotIn
 values: ["host_ns"] # only processes NOT in the host pid namespace
 matchActions:
 - action: Sigkill
 - call: "__x64_sys_unshare"
 syscall: true
 args:
 - index: 0
 type: int
 selectors:
 - matchNamespaceChanges:
 - unshare: true
 matchActions:
 - action: Post

__x64_sys_setns targeting the host namespace from a process in a container is practically always malicious (legitimate containers do not need this at runtime).

5. Cryptomining

Mining processes have fairly recognisable profiles:

Processes with names like xmrig, minerd, cgminer, or legitimate processes such as python running CPU-intensive scripts.
Outbound connections to known mining pools (a public list of IPs and domains).
Anomalous use of /dev/cpu_dma_latency to avoid throttling.

A combined policy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-cryptomining
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm
 selectors:
 - matchArgs:
 - index: 4
 operator: "Postfix"
 values:
 - "/xmrig"
 - "/minerd"
 - "/cgminer"
 matchActions:
 - action: Sigkill
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "DPort"
 values: ["3333", "5555", "7777", "14444"] # common pool ports
 matchActions:
 - action: Post  # log only

The double policy: kill binaries with the classic names (belt) and log connections to pool ports (braces), so you also get an alert when someone renames xmrig to nginx-helper or uses exotic ports.

6. Detecting rootkits and suspicious kernel modules

Modern rootkits load kernel modules to patch functions (hide processes, hide network connections, hide files). Detecting them:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: kernel-module-load
spec:
 kprobes:
 - call: "do_init_module"
 syscall: false
 args:
 - index: 0
 type: "string"
 selectors:
 - matchActions:
 - action: Post
 - call: "security_kernel_read_file"
 syscall: false
 args:
 - index: 0
 type: "file"
 - index: 1
 type: int
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values: ["READING_MODULE"]
 matchActions:
 - action: Post

In a “well configured” Kubernetes cluster no new kernel modules get loaded at runtime; any event here is highly suspicious. Combine with enforcement on machines where modules should be fixed: Sigkill for whoever tries to load one.

Bonus: detecting third-party modification of eBPF maps

As a 2025-2026 trend: loading malicious eBPF programs to hide presence. Tetragon can observe the bpf syscall:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-bpf-syscalls
spec:
 kprobes:
 - call: "__x64_sys_bpf"
 syscall: true
 args:
 - index: 0
 type: int  # bpf cmd
 - index: 1
 type: bpf_attr
 selectors:
 - matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/cilium-agent"
 - "/usr/bin/tetragon"
 - "/usr/bin/bpftool"
 matchActions:
 - action: Post

Any process that is not one of the legitimate agents loading eBPF programs: you want to know about it.

Comparison with Falco

Falco is the closest competitor: it is also runtime security for Kubernetes, also originally based on eBPF (and before that on kernel modules), also with declarative policies. Three years ago they were functionally similar. In 2026 the divergence is clear:

Dimension	Tetragon	Falco
Philosophy	Cilium-native, integrated	Standalone, generic
Filtering	In the kernel (eBPF)	Parsing in userspace
Typical overhead	<1% CPU	5-10% CPU
Enforcement	Yes, in-kernel (Sigkill, Override)	Not native (depends on plugins)
Race conditions	No (action atomic with the syscall)	Yes, in enforcement via plugins
False positive rate	Low (K8s context in the kernel)	Higher (parsing afterwards)
Detection latency	5-26 ms	~10 ms (more constant)
Ecosystem maturity	Young, growing	Mature, plenty of material
Community	Cilium / CNCF Incubating	CNCF Graduated
Integrations	Hubble native	Falcosidekick, many
CRDs per policy	TracingPolicy / Namespaced	No CRDs; rules in YAML

When to choose each one:

Tetragon if you already use Cilium, if you need enforcement in the kernel (not detection-only), if overhead matters to you (workloads with many syscalls), and if you value the Hubble integration. Container escape and cryptomining detection is where its advantage over Falco shows up most.
Falco if you want a tool independent of the CNI, if you need the catalogue of ready-made rules and the wide community, if your cluster is not Cilium, or if the integration with SIEMs and ready-made notifiers (Falcosidekick) saves you work.
Both if the organisation is large: Falco for breadth of detection, Tetragon for surgical enforcement on critical workloads. That is what you see most in companies that have spent years with Falco and add Tetragon for specific cases.

Hubble + Tetragon: unified observability

Hubble is Cilium’s traffic observability component: it shows L3-L7 flow logs with zero impact on latency. Tetragon exposes its events over gRPC with the same format and vocabulary as Hubble, which allows you to:

See them in the same UI (Hubble UI shows Tetragon events as one more “layer”).
Correlate network events (Hubble) with process events (Tetragon) on the same timeline.
Export them together to Loki/Tempo/SIEM as a single stream.

The key synergy: Hubble tells you “this pod made a TCP connection to 1.2.3.4:80”. Tetragon tells you “this pod ran curl 1.2.3.4 from a bash binary launched by pid 1234”. Together they give you the full story.

Deployment and operation

Helm

The canonical install with Helm:

helm repo add cilium https://helm.cilium.io
helm install tetragon cilium/tetragon \
 --namespace kube-system \
 --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
 --set tetragon.exportFileMaxSizeMB=50 \
 --set tetragon.exportFileRotationInterval=24h

Tetragon deploys its DaemonSet, its CRDs and a service for Hubble. By default, it exposes the events on the agent pod’s stdout (any cluster log aggregator picks them up).

The `tetra` CLI

Tetragon ships a CLI called tetra for interactive investigation:

# real-time stream of the node's events
tetra getevents -o compact --pods <pod-name>

# structured JSON events to process with jq
tetra getevents -o json --since 5m --namespace prod

# view loaded policies
tetra tracingpolicy list

It is the best tool for debugging policies in monitoring before moving them to enforce.

Exporting to a SIEM

Three usual routes:

stdout + log aggregator: the agent writes JSON to stdout, Fluent Bit/Vector picks it up and sends it to Splunk/Datadog/Elastic. Simple, works with any logging infrastructure.
gRPC streaming: for low-latency integrations. A gRPC consumer of your own or Hubble Relay.
File + rotation: for air-gapped environments or regulatory audits that require persistent logs with controlled rotation.

Performance

Published benchmarks consistently place Tetragon at <1% of the node’s CPU under real workloads, compared with 5-10% for Falco under the same workloads. The reason is the architectural separation: Tetragon filters in the kernel and only carries the events that actually matter to userspace; Falco carries every syscall to userspace and filters there. On clusters with thousands of pods making hundreds of thousands of syscalls per second, the difference shows up on the bill.

Common operational traps

Permanent `monitoring`

The biggest trap is never reaching enforce: starting well with policies in monitoring, collecting events, tuning selectors, and then never switching. The result: you have detection without prevention, exactly what Falco gave you without paying Tetragon’s complexity. If you are going to use Tetragon, plan the road to enforce for the critical policies.

Selectors that are too lax

A policy with a single matchActions: Post and no specific selectors generates events for every syscall of the chosen hook. On a serious node that means tens of thousands per second, which fill logs, saturate exporters and hide the signal in the noise. Always start with strict filters (matchBinaries, matchNamespaces, matchPIDs) and open up once you know what you are looking for.

A kernel that is too old

Tetragon needs modern eBPF features. Kernels < 5.4 do not have the large buffer support (needed for execve with complete argv). Kernels < 5.10 do not have many of the LSM hooks. Kernel 5.15+ is the recommended minimum for production and 6.1+ to have every feature.

Hooks on renamed kernel functions

kprobes are tied to kernel function names that can change between versions. A policy that uses __x64_sys_setns can fail silently on a kernel where the function is called __do_sys_setns. Solutions: use static tracepoints where they are available (more stable), or keep alternative policies with several call entries for compatibility.

`Sigkill` in critical namespaces

Applying Sigkill to processes in kube-system or cilium-system can break the cluster. Enforcement policies must explicitly exclude the platform namespaces with matchNamespaces Operator: NotIn, or limit the scope with TracingPolicyNamespaced to make sure they do not act on systems they should not.

Missing `rateLimit`

A policy without a rateLimit on Post can suffer a catastrophic fan-out if the condition is met millions of times in an instant (typical in attack loops or application bugs). The agent saturates, events are lost, logs overflow. Always put a sensible rateLimit on detection policies, especially on high-frequency hooks such as tcp_connect or execve.

What we have not covered (upcoming articles)

eBPF LSM hooks in depth: how they relate to SELinux/AppArmor and when Tetragon is the right tool versus classic MAC.
Hubble UI with a Tetragon overlay: configuring the UI to show process observability and network observability on the same timeline.
Integration with OPA/Kyverno: how Tetragon complements admission policy engines (Kyverno validates at admission; Tetragon validates at runtime).
Forensics with eBPF: combining Tetragon with tools such as Beyla or OpenTelemetry to trace the full chain of an incident from the initial connection to the final syscall.

References

Official documentation (May 2026):

Tetragon — official site — entry point.
Tetragon docs — Tracing Policy — conceptual reference.
Tetragon docs — Hook points — kprobes, tracepoints, uprobes, LSM, USDT.
Tetragon docs — Selectors — full filter reference.
Tetragon docs — Enforcement Mode.
Tetragon docs — Kubernetes Identity Aware Policies — TracingPolicyNamespaced.
Tetragon GitHub.

Comparisons and analysis:

Comparative Analysis of eBPF-Based Runtime Security Monitoring (SciTePress paper 2025) — a benchmark with independent numbers.
Best eBPF Security Solutions for Kubernetes (ARMO, 2026) — Falco vs Tetragon vs KubeArmor comparison.
Falco vs. Tetragon (Asim Mirza, Medium) — analysis with use cases.
Deep Dive into Tetragon (A Security Engineer) — a walk through the inside of the agent.
Tetragon Series, Part 2: Enforcing Sensitive File Access (Medium).

Ecosystem:

Cilium Hubble — network observability.
Falco — official site — the other big name in the field.
KubeArmor — the third option, with AppArmor + eBPF.

Cross-references:

Part 1 of the series: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack — the eBPF fundamentals we take as read here.
Kubernetes with Cilium BGP: services reachable without Ingress — the starting point of the Cilium ecosystem on this blog.

eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack

Tue, 19 May 2026 04:30:00 +0200

TL;DR

eBPF is a sandboxed virtual machine inside the Linux kernel that runs verified code at well-defined hooks: kprobes, tracepoints, socket events, network drivers. Before eBPF, changing kernel behaviour meant recompiling it or loading an arbitrary module; with eBPF, you load a small program that passes a formal verifier and runs at native speed with memory safety. In networking, this translates into the fact that the packet does not have to travel through the traditional TCP/IP stack: an eBPF program in the NIC driver (XDP) can drop, forward or rewrite the packet before the kernel has done its first alloc; a program on cgroup hooks (sock_ops) can redirect connections to another socket without the packet ever leaving the machine. Cilium is the CNI that has taken this to its logical conclusion: it replaces kube-proxy with pure eBPF (O(1) instead of the O(N) of iptables), routes pod-to-pod without VXLAN where it can, evaluates Network Policies with BPF maps, and since 1.16 it has remade its BGP control plane with a new set of CRDs, CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, CiliumBGPNodeConfigOverride, which replace the monolithic CiliumBGPPeeringPolicy that is already deprecated. This post goes down the three layers (basic eBPF → eBPF networking → Cilium) and ends with the operational CRDs.

The analogy: signed plugins for the kernel

Think of the browser. Twenty years ago, extending a browser meant compiling a native binary and loading it: any extension could crash it, corrupt memory, read your bank cookies. Today, extensions are JavaScript in a sandbox with a manifest that declares permissions, a runtime that enforces the isolation and a store that signs the code. The extension does not touch the browser binary; it lives in a controlled world and can only talk to the browser through defined APIs. Result: massive extensibility with a bounded attack surface.

eBPF is exactly that for the Linux kernel. Loading a classic .ko module means loading native code with full access to kernel memory: one bug and the system is gone. eBPF is a bytecode VM with a static verifier, a controlled allocator, JIT to native hardware after passing the verifier, and a set of kernel “helpers” it can call. The eBPF program can read the kernel memory the verifier allows it to read, and only that. It cannot enter infinite loops (the verifier demands that it terminate). It cannot jump to arbitrary addresses. It cannot dereference pointers without having validated them first. And, most importantly: it is user code, loaded at runtime, executing inside the kernel at native speed.

The consequences are visible from miles away. Before, observing traffic in production meant patching the kernel or loading a risky module. Today, bpftrace -e 'tracepoint:net:net_dev_xmit { @[args->dev->name] = count(); }' gives you a histogram of packets per interface in three lines and zero downtime. Before, replacing iptables with something faster meant rewriting the netfilter subsystem. Today, Cilium loads 60 KB of eBPF bytecode into XDP and unseats iptables with a hash map.

Basic eBPF: what it is and what it is not

The origin and the scope

The name comes from Berkeley Packet Filter, a 1992 idea (McCanne and Jacobson) for filtering packets with a mini-bytecode that tcpdump used internally. In 2014, Alexei Starovoitov renamed it eBPF and extended it enormously: 11 64-bit registers instead of 2 32-bit ones, a 512-byte stack, maps as structures shared with userspace, JIT to native hardware, and a far more sophisticated formal verifier. From being a packet filter, it became a generic kernel extensibility mechanism.

Today eBPF is used for four things:

Networking: XDP, TC, cgroup hooks, socket ops, lightweight tunnels.
Observability: kprobes, uprobes, tracepoints, USDT. The basis of projects like bpftrace, bcc, Pixie, Parca.
Security: BPF LSM (Linux Security Module in eBPF), syscall blocking with seccomp-bpf. Falco, Tetragon, Tracee.
Scheduling: sched_ext (kernel 6.12+), process schedulers written entirely in eBPF. Still at a very early stage.

The VM

An eBPF program is compiled from C (or Rust, or Go with cilium/ebpf) to eBPF bytecode, not to x86/arm64 directly. The kernel loader (via the bpf() syscall) passes that bytecode through the verifier:

It reconstructs the control flow graph.
It performs static analysis of every possible path: every instruction has to be reachable, every memory access has to be within known bounds, every pointer has to have been validated.
It rejects loops without a known upper bound. Recent kernels admit bounded loops (the bpf_loop helper), but the counter is always finite.
It rejects calls to helpers or kfuncs that the hook’s program type does not allow.

If the verifier accepts the program, the JIT translates it to the host’s native code (x86, arm64, etc.) and it stays attached to its hook. From then on it runs every time the hook’s event occurs, with no context switch to userspace, with no syscall cost. Latencies on the order of hundreds of nanoseconds per invocation.

Maps: the bridge to userspace

An isolated eBPF program is not much use. What makes it useful are maps: data structures shared between the kernel program and userspace. There are several types:

BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_LRU_HASH: hash tables with or without LRU eviction.
BPF_MAP_TYPE_ARRAY, BPF_MAP_TYPE_PERCPU_ARRAY: arrays, optionally per-CPU to avoid contention.
BPF_MAP_TYPE_RINGBUF, BPF_MAP_TYPE_PERF_EVENT_ARRAY: channels for streaming events to userspace.
BPF_MAP_TYPE_PROG_ARRAY: arrays of eBPF programs for tail calls (chaining programs without returning to the base kernel).

Userspace reads and writes these maps via bpf() syscalls; the kernel program reads and writes them directly. It is the basis of any eBPF system: the kernel program collects data into a map, the userspace daemon reads it. Cilium does exactly this: the userland agent (Go) manages the policy and translates it into map entries; the eBPF programs living in XDP/TC read the maps and apply the decisions.

CO-RE: compile once, run on any kernel

A classic nightmare of kernel modules: they are tied to the exact kernel version they were compiled against. Distributing a precompiled module for a fleet of machines with different distros was impossible.

eBPF solves this with CO-RE (Compile Once, Run Everywhere): the bytecode includes relocations that the loader resolves on each specific kernel by consulting BTF (BPF Type Format), a representation of the kernel’s struct layouts that the kernel itself publishes. Result: a single eBPF binary works on kernels 5.10, 5.15, 6.1 and 6.8 without recompiling, because the loader adjusts the struct access offsets at runtime.

This is what has allowed productive eBPF distributions to exist. Without CO-RE, every kernel would be a porting project.

eBPF in networking: the hooks that matter

Inside the Linux network subsystem, eBPF has several hooks. The ones relevant to CNIs:

XDP — eXpress Data Path

XDP is the earliest hook: it runs in the NIC driver, before the packet enters the kernel proper. There is no sk_buff (the struct the rest of the kernel uses to represent packets); there is only a pointer to a RAM buffer with the received bytes.

The actions an XDP program can return:

XDP_DROP: discard the packet immediately. The driver drops it and frees the buffer. Cost: nanoseconds. Use case: DDoS mitigation. Cloudflare processed more than 8 million packets/second per CPU with XDP for dropping SYN floods.
XDP_PASS: let the packet continue to the normal kernel. It becomes an sk_buff and enters the traditional stack.
XDP_TX: send it back out on the same interface after possible modifications. Useful for L4 load balancers that rewrite the destination and return it.
XDP_REDIRECT: send the packet to another interface or to a map (to forward to userspace via AF_XDP, or to another NIC, or to a pod’s veth).
XDP_ABORTED: error (increments a counter, drops).

Real use cases:

Cloudflare L3 DDoS protection: XDP rules that drop millions of packets/s.
Facebook Katran: an L4 load balancer that rewrites the destination IP and returns it on the same interface. Handles 10× more connections per server than classic IPVS.
Cilium XDP acceleration: Service load balancing at the lowest layer possible.

TC (Traffic Control) — clsact with BPF

XDP is very fast but limited: the packet does not have an sk_buff yet and many decisions (conntrack, NAT, encapsulation with metadata) are easier when it does. The TC clsact with BPF hook runs after building the sk_buff but before the routing and netfilter decisions. Actions:

TC_ACT_OK: the packet continues through the stack.
TC_ACT_SHOT: drop.
TC_ACT_REDIRECT: redirect to another interface.
TC_ACT_PIPE, TC_ACT_STOLEN: pipeline control for combining with other qdiscs.

Use cases:

Stateful network policy: Cilium evaluates L3-L7 policies in TC with the full sk_buff and conntrack available.
Marking and QoS: traffic marking so the scheduler applies priorities.
Overlay encapsulation: adding VXLAN/Geneve headers when the mode is tunnel.

XDP and TC combine: XDP for the cheap and early stuff (DDoS, simple LB), TC for what needs an skb and state.

Cgroup hooks: sock_ops and CGROUP_SOCK_ADDR

The most radical conceptual step: hooks that are not in the network layer but in the socket layer. Relevant types:

BPF_PROG_TYPE_CGROUP_SOCK_ADDR: invoked when a process in a cgroup calls connect(), bind(), sendto(). The eBPF program can rewrite the destination address before the connection goes out. This is what lets Cilium do Service load balancing without the packet entering the network stack: if the client tries to connect to 10.96.0.1:443 (a ClusterIP), an eBPF program on this hook rewrites the destination to the real IP of the backend pod before the syscall continues.
BPF_PROG_TYPE_SOCK_OPS: invoked on TCP events (creation, established, retransmission). It allows tuning socket parameters at runtime and, most importantly, pairing local sockets via bpf_sk_assign to shortcut without the packet travelling over the network.

This is the “third layer” of the bypass: it is not just faster, it is conceptually different. The packet is not built, not serialised, does not traverse the IP layer or the TCP layer. It is the difference between speeding up a road and discovering that for some journeys you do not need to take the car at all.

The long road: what the traditional TCP/IP stack looks like

To appreciate what eBPF saves, it is worth tracing a packet’s full journey through the Linux stack. Take the case “packet arrives on a NIC, goes to a local process”:

NIC (DMA into the driver's ring buffer)
↓
driver: napi_schedule, poll, allocates sk_buff
↓
[XDP hook] ← if there is an XDP program, it is decided here
↓
netif_receive_skb
↓
__netif_receive_skb_core
↓
[TC ingress clsact + BPF] ← if there is a TC ingress program
↓
packet_type handlers (IP, ARP...)
↓
ip_rcv → ip_rcv_core
↓
[netfilter NF_INET_PRE_ROUTING] ← iptables PREROUTING
↓
routing decision (FIB lookup)
↓
[netfilter NF_INET_LOCAL_IN] or [NF_INET_FORWARD]
↓
tcp_v4_rcv → tcp_v4_do_rcv
↓
tcp_rcv_established
↓
sk_data_ready
↓
process reads with recv()/read()

Every arrow is a function call with a measurable cost. Every netfilter hook walks all the registered iptables/nftables rules. With kube-proxy in iptables mode and 5,000 Services × 10 endpoints each, there are on the order of 150,000 rules evaluated sequentially at NF_INET_PRE_ROUTING. Published benchmarks show latencies of tens of microseconds per packet in large Kubernetes clusters in the netfilter step alone, before the application receives anything.

And that is the normal path. On the way out the same thing happens in reverse: tcp_sendmsg → ip_output → NF_INET_LOCAL_OUT → routing → NF_INET_POSTROUTING → dev_queue_xmit → driver → NIC.

How Cilium skips this stack

Cilium does not eliminate the TCP/IP stack; it is still there for the cases that need it. What it does is shortcuts at the points where it hurts.

Shortcut 1 — XDP for the Service datapath

For a cluster with 5,000 Services, kube-proxy iptables has an O(N) cost in evaluating rules (even with iptables-restore --noflush and tricks, it is still linear in the number of chains the packet traverses).

Cilium replaces it like this:

Every Service and its endpoints live in an eBPF hash map.
When a packet comes in destined for a ClusterIP, Cilium’s XDP program does an O(1) lookup in that map and obtains the backend endpoint.
It rewrites the destination and does XDP_TX (returns it on the same interface towards the backend) or XDP_REDIRECT (sends it to the corresponding local pod’s veth).

This means the cost does not grow with the number of Services. 100 Services or 100,000, constant lookup in the map. Published benchmarks show latency reductions of 30-50% in clusters with many Services compared with kube-proxy iptables, and of an order of magnitude compared with IPVS in some cases.

Shortcut 2 — socket-LB: the packet is never built

Cilium 1.6+ introduced socket-level load balancing, based on cgroup hooks. It works like this:

When a pod calls connect(10.96.0.1:443) (a Service’s ClusterIP), the syscall enters the kernel.
Before the kernel builds anything network-related, an eBPF program on CGROUP_SOCK_ADDR/connect4 intercepts it and rewrites the destination address to the real IP of the backend pod.
The kernel carries on with the connect as if the client had written 10.0.0.42:8080 directly.

Why does it matter? Because when the backend pod is on the same node, this shortcut turns a call that would have involved:

syscall connect → kernel stack → veth → bridge → veth → kernel stack → syscall accept

into:

syscall connect (with the destination rewritten) → direct loopback

The TCP/IP stack is literally avoided. There is no encapsulated packet, no journey through veth pairs, no netfilter. L7 pod-to-pod latencies on the same node drop to local communication levels (~5-15 µs instead of ~30-50 µs for services with kube-proxy iptables and traditional veth).

Shortcut 3 — pod-to-pod direct routing

The traditional overlay mode (Flannel, Calico VXLAN) encapsulates every pod-to-pod packet in VXLAN/Geneve. Every packet carries an extra 50-byte header, requires encap/decap, and consumes MTU.

Cilium supports direct routing: the pod CIDRs are advertised to the underlying fabric (with BGP, which is where the control plane we will look at comes in) and the physical routers route the pod-to-pod packets without encapsulating. The packet leaves a pod with its original IP as source and the destination pod’s IP as dest, the node’s NIC hands it to the network, the network routes it, it arrives at the destination node and is delivered to the pod. Zero encap, full MTU, minimal latency.

Cilium does this via eBPF programs in TC that rewrite the necessary headers and decide whether the packet goes via encap or direct according to the policy configured per node.

Shortcut 4 — Network Policy in TC with maps

Network Policies in classic CNIs are usually translated into iptables rules, another factor that explodes linearly. Cilium evaluates them in eBPF programs that read identity maps: every workload has a numeric identifier computed from its labels, and the policy is a map (src_identity, dst_identity, port, proto) → allow|deny. One hash map lookup per packet.

This also enables Cilium’s L7 policies (HTTP, gRPC, Kafka filtering): the eBPF program recognises the L7 handshake, selectively redirects to the embedded Envoy proxy (which lives as a sidecar of the datapath, not as a pod sidecar) and only on that subset does it pay the cost of the L7 proxy. All the L3/L4 traffic stays on the eBPF fast path.

Cilium: the architecture

Cilium combines two planes:

Agent (Go): lives as a DaemonSet on every node. It is the “slow” part: it translates the intent expressed in CRDs (CiliumNetworkPolicy, CiliumBGPClusterConfig, etc.) into entries in eBPF maps. It talks to the Kubernetes API server to discover endpoints, services, pods. It embeds a GoBGP for the BGP control plane. It embeds an Envoy for L7 policies.
Datapath (eBPF): the programs loaded into XDP, TC, cgroup hooks. They are the “fast” part: they see every packet, read the maps the agent maintains, and decide in nanoseconds.

This separation is what makes Cilium operationally comfortable: the intent is expressed in YAML, the agent materialises it into maps, the maps are read by the datapath. If the agent goes down temporarily, the datapath keeps working with the last loaded configuration. As in any well-built control/data plane system.

BGP Control Plane v2: the CRDs you have to know

Cilium has had BGP support for several years. The first version used a single monolithic CRD, CiliumBGPPeeringPolicy, which mixed node configuration, peers, timers and advertisements into a single object. Since Cilium 1.16 there is BGP Control Plane v2, which breaks that configuration into separate CRDs with clear responsibilities. CiliumBGPPeeringPolicy (the cilium.io/v2alpha1 API) is deprecated and migration warnings appear in the operator logs if you still use it.

The new CRDs (the cilium.io/v2 API):

1. `CiliumBGPClusterConfig`

Defines BGP instances and the peers they connect to, from the cluster’s perspective. Which nodes apply this configuration is selected with a nodeSelector.

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
 name: cilium-bgp-cluster
spec:
 nodeSelector:
 matchLabels:
 bgp-policy: rack-1  # only nodes with this label
 bgpInstances:
 - name: instance-65000
 localASN: 65000
 peers:
 - name: top-of-rack-1a
 peerASN: 64512
 peerAddress: 10.0.1.1
 peerConfigRef:
 name: tor-shared-config  # → reference to CiliumBGPPeerConfig
 - name: top-of-rack-1b
 peerASN: 64512
 peerAddress: 10.0.1.2
 peerConfigRef:
 name: tor-shared-config

A BGP instance is the abstraction “this node takes part in BGP with this local ASN and these peers”. Several can coexist on the same node (multi-instance for multi-VRF).

2. `CiliumBGPPeerConfig`

Defines the shared parameters of the peering: timers, address families, transport, MD5 password, graceful restart, etc. It is referenced from CiliumBGPClusterConfig via peerConfigRef. This avoids repeating the same configuration for every peer when there are dozens of them.

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
 name: tor-shared-config
spec:
 timers:
 holdTimeSeconds: 30
 keepAliveTimeSeconds: 10
 connectRetryTimeSeconds: 5
 gracefulRestart:
 enabled: true
 restartTimeSeconds: 120
 families:
 - afi: ipv4
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp  # → binds to CiliumBGPAdvertisement
 - afi: ipv6
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp
 authentication:
 password:
 name: bgp-md5-secret  # Secret with the MD5 password
 key: password

A single CiliumBGPPeerConfig can be referenced by many different peers. You change timers or families in one place.

3. `CiliumBGPAdvertisement`

Declares which prefixes are advertised: the node’s pod CIDRs, the ClusterIPs and ExternalIPs of Services, the IPs assigned by CiliumLoadBalancerIPPool for type=LoadBalancer Services. They are bound to CiliumBGPPeerConfig via labels.

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
 name: services-and-pods
 labels:
 advertise: bgp  # ← the label the PeerConfig matches
spec:
 advertisements:
 - advertisementType: PodCIDR  # advertises the node's pod CIDR
 attributes:
 communities:
 standard:
 - "65000:100"
 - advertisementType: Service  # advertises ClusterIPs / LoadBalancer IPs
 service:
 addresses:
 - LoadBalancerIP
 - ClusterIP
 - ExternalIP
 selector:
 matchLabels:
 bgp-advertise: "true" # only Services with this label
 attributes:
 communities:
 standard:
 - "65000:200"
 localPreference: 200

The granularity is very fine: you can advertise different types of prefixes with different BGP communities, different local-preference, different path attributes, and filter Services with label selectors. This was literally impossible with CiliumBGPPeeringPolicy v1.

4. `CiliumBGPNodeConfig` (auto-generated)

This CRD is not configured by hand. The Cilium operator generates one per node from the CiliumBGPClusterConfig that applies to that node. It is the materialised per-node state that each node’s agent reads to bring up its peerings. If you want to see what BGP configuration is actually running on a node, kubectl get ciliumbgpnodeconfig <nodename> -o yaml shows you.

5. `CiliumBGPNodeConfigOverride`

Optional. It allows overriding the generated configuration for a specific node when you need something non-standard. Use cases:

Pinning the BGP router-id to a specific IP (useful when the node has several interfaces).
Specifying the peer’s local address when there are several outgoing interfaces.
Changing timers only for one problematic node.

apiVersion: cilium.io/v2
kind: CiliumBGPNodeConfigOverride
metadata:
 name: node-rack1-master01  # the name must match the node's
spec:
 bgpInstances:
 - name: instance-65000
 routerID: 10.0.1.10 # router-id override
 peers:
 - name: top-of-rack-1a
 localAddress: 10.0.1.10 # specific local interface

Relationship diagram

CiliumLoadBalancerIPPool: it complements, it is not BGP

Although it is not strictly a BGP CRD, it is worth mentioning: CiliumLoadBalancerIPPool is the CRD that supplies IPs to type=LoadBalancer Services. It defines a range (10.20.0.0/24, for example) that Cilium assigns automatically to LoadBalancer Services. Combined with a CiliumBGPAdvertisement that advertises LoadBalancerIP, it gives the complete cycle: new Service → IP assigned from the pool → BGP advertisement to the routers → IP routable from the corporate network, with no external balancer.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
 name: lb-pool-rack1
spec:
 blocks:
 - start: "10.20.0.10"
 stop: "10.20.0.250"
 serviceSelector:
 matchLabels:
 lb-pool: rack1

Full manifest: pod CIDRs + LoadBalancer Services advertised to a redundant ToR pair

A realistic example of a cluster with two top-of-rack switches as BGP peers, both in the same AS (64512), Cilium in AS 65000:

# 1. CiliumBGPPeerConfig — shared config for both ToRs
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
 name: tor-peers
spec:
 timers:
 holdTimeSeconds: 30
 keepAliveTimeSeconds: 10
 gracefulRestart:
 enabled: true
 restartTimeSeconds: 120
 families:
 - afi: ipv4
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp

# 2. CiliumBGPAdvertisement — what gets advertised
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
 name: pods-and-lb
 labels:
 advertise: bgp
spec:
 advertisements:
 - advertisementType: PodCIDR
 - advertisementType: Service
 service:
 addresses: [LoadBalancerIP]
 selector:
 matchExpressions:
 - { key: io.kubernetes.service.namespace, operator: NotIn, values: [kube-system] }

# 3. CiliumBGPClusterConfig — which nodes talk to which peers
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
 name: cluster-bgp
spec:
 nodeSelector:
 matchLabels:
 bgp: enabled
 bgpInstances:
 - name: instance-65000
 localASN: 65000
 peers:
 - name: tor-a
 peerASN: 64512
 peerAddress: 10.0.1.1
 peerConfigRef: { name: tor-peers }
 - name: tor-b
 peerASN: 64512
 peerAddress: 10.0.1.2
 peerConfigRef: { name: tor-peers }

# 4. CiliumLoadBalancerIPPool — range of LB IPs
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
 name: lb-corporate
spec:
 blocks:
 - cidr: "10.20.0.0/24"

Four objects. Before, in v1, it was a single CiliumBGPPeeringPolicy that mixed everything together and turned out to be hard to maintain in clusters of more than 5 nodes with heterogeneous configuration. The new separation is longer but clearly factorable: one PeerConfig per peer type, one Advertisement per advertisement policy, one ClusterConfig connecting nodes with peers.

Common operational traps

`routingMode: tunnel` mode with BGP

BGP only makes sense with direct routing (routingMode: native). If you have tunnel mode (VXLAN/Geneve) and configure BGP, you will advertise pod CIDRs but the packets will still go out encapsulated, producing confusing behaviour (sometimes via tunnel, sometimes direct depending on routes). Configure routingMode: native and disable the tunnel.

eBPF host routing vs `kube-proxy replacement`

They are two different things. kubeProxyReplacement: true enables the replacement of kube-proxy (the Services). bpf.hostRouting: true enables the host’s iptables bypass (the node’s routing decisions are made with eBPF instead of the traditional FIB). The second needs kernel 5.10+ with all the bpf features enabled; if you do not have that kernel, it falls back to legacy mode and the performance is only “almost as good”.

Aggressive BGP timers over flapping NICs

With holdTimeSeconds: 9 / keepAliveSeconds: 3, a NIC that blinks for 5 seconds breaks the BGP session and all the advertised routes disappear from the fabric. That node’s pods become unreachable until the session is re-established. For clusters on hardware with suspect NICs, use the conservative values (holdTime: 30, keepAlive: 10) and consider graceful restart explicitly (it is already in the example above).

Advertising ClusterIP to the corporate network

Advertising ClusterIP to external routers is rarely what you want: these are internal Service IPs, not designed to be reached from outside the cluster. For external exposure, use LoadBalancerIP from a CiliumLoadBalancerIPPool. Advertising ClusterIP only makes sense in very specific topologies (multi-cluster mesh with shared service discovery).

Mixing v2alpha1 (`CiliumBGPPeeringPolicy`) and v2 (`CiliumBGPClusterConfig`)

It does not work well. The operator emits warnings in the logs about the use of the deprecated API, and conflicts between what the peering policy defines and what the cluster config defines can produce strange states. Migrate from one to the other in a single pass; do not run both.

MD5 password and MTU

If you configure an MD5 password in CiliumBGPPeerConfig.authentication, the TCP header is larger. On links with a tight MTU (1500 - 50 for the upstream fabric’s VXLAN, for example), the BGP handshake can fragment and die silently. Either use MTU 9000 between nodes and ToR, or make sure the MSS values are negotiated correctly.

What we have not covered

Cilium Cluster Mesh: federation of several Cilium clusters so their Services can see each other. It fits with BGP when you want native routing between clusters; it has its own CRDs.
L7 Policies and the embedded Envoy: HTTP/gRPC/Kafka policy. Another layer of eBPF + proxy that deserves its own post.
Hubble: eBPF-based traffic observability that Cilium exposes. Flow log dashboards with zero impact on latency.
Transparent WireGuard: pod-to-pod encryption without sidecars, controlled by Cilium via eBPF redirect to a kernel WireGuard dataplane.
Gateway API in Cilium: the successor to Ingress, with first-class support since Cilium 1.16+.
eBPF for LLM serving: the natural connection with the previous inference series. There is recent work using eBPF for multi-tenant fairness on GPUs and for token tracking; paper territory, not production yet.

In other words, there is material left for another three posts in today’s series. Let us take them in order.

References

Conceptual and project:

eBPF.io — canonical documentation of the eBPF ecosystem.
The BPF Compiler Collection (bcc) and bpftrace — tools to get started with.
eBPF en 2026: How Extended Berkeley Packet Filter Became the Engine of Linux Observability and Networking — state of the art.

XDP, TC and firewalling:

Getting started with XDP and eBPF (Red Hat docs).
Full Guide to BPF Firewalls: XDP, tc, and eBPF Integration (Medium, 2025).
Cloudflare blog: XDP for DDoS mitigation.
Facebook Katran (GitHub) — L4 LB with XDP, code and paper.

Cilium:

Cilium documentation — always the first port of call.
Kubernetes Without kube-proxy — the official guide to the replacement.
Cilium BGP Control Plane Resources (docs) — reference for the v2 CRDs.
Configuring Cilium BGP Control Plane (OneUptime blog, mar 2026) — walkthrough.
A Guide to BGP Control Plane and Cluster Mesh in Cilium Networking (Sigrid Jin, Medium) — a deeper post with use cases.

Cross-references:

Previous post on this blog: Kubernetes with Cilium BGP: services reachable without Ingress — the first step, with version v1 (which now needs migrating).
Series on LLM inference: KV cache, vLLM on Kubernetes, PagedAttention deep dive, LLM K8s Operators — where the fast network (which we will see in the following posts of this series) determines real performance.

Kubernetes with Cilium BGP: services reachable without Ingress

Sun, 08 Mar 2026 00:00:00 +0000

One of the advantages of using Cilium as the CNI in Kubernetes is its native BGP support. It lets you advertise ClusterIPs and LoadBalancer IPs directly to the LAN router, making services reachable without needing Ingress or NodePort.

The problem

In a standard Kubernetes cluster, pods and services live on internal networks that are not reachable from outside the cluster. Getting to them requires NodePort, Ingress or an external LoadBalancer.

With Cilium BGP, the pod CIDRs and service CIDRs are advertised over BGP to the upstream router, making the whole cluster network routable from the LAN.

Work in progress — full configurations for RKE2 + CRS327 coming soon.

Cilium on lo0 — Tech Blog

Hardening and secrets in the sovereign LLM stack: defence in depth

TL;DR

The analogy: the sovereign office

Layer 1 — Secrets: GitOps’s chicken-and-egg problem

Family A — sealed-secrets (asymmetric encryption, controller in the cluster)

Family B — SOPS + age (file encryption, simple)

Family C — External Secrets Operator + Vault (dynamic secrets, native rotation)

How to choose, and rotation

Layer 2 — Network: default-deny, egress and the arithmetic of the surface

The arithmetic of communication pairs

Standard L3/L4, L7 and egress with Cilium

Internal mTLS

Layer 3 — Pod security: restricted, and nothing privileged

Layer 4 — Image supply chain

Layer 5 — AuthN/Z: virtual keys, OIDC and minimal RBAC

Layer 6 — Runtime: the cameras with Tetragon

Layer 7 — Data at rest

Exposure table: service × port × external egress?

Blast radius: a leaked secret, with and without defence

The ENS / NIS2 angle

Applied to the generic 4×H100 cluster

What hardening does NOT solve

See also

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

TL;DR

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

The gap: why the Topology Manager does not look at the NIC

The network datapath under NUMA: IRQ, softirq and DMA

One number, with its caveat

What RKE2’s Cilium eBPF replaces (and why it touches this story)

Cilium performance profile (state 1.19, kernel ≥6.8)

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

How it connects with the rest of the stack

Traps and things that are not what they look like

Conclusion

See also

References

Hubble: eBPF network observability, state of the art 2026 and the new frontier with AI agents

TL;DR

The analogy: tcpdump that speaks Kubernetes

Architecture: four pieces visible from outside

1. Hubble Server (embedded in every Cilium agent)

2. Hubble Relay (aggregator)

3. Hubble CLI (hubble)

4. Hubble UI

What you see: the Hubble flow log from the inside

Verdict and drop attribution

Prometheus metrics and Grafana dashboards

Deployment: Helm on one screen

State of the art in 2026

Direct attribution of drops to a NetworkPolicy

Tracing with IP options

Filtering by encryption status

Hubble field mask API stabilised

AI-driven anomaly detection (predictive security)

Scaling to 10,000+ pods

Cilium 1.20 in development

The new frontier: eBPF and AI agents

AgentSight: zero-instrumentation for LLM agents

How it fits with Hubble and Tetragon

Emerging use cases

Common Hubble use cases

1. NetworkPolicy debugging

2. Auditing inter-namespace communication

3. Exfiltration detection

4. Real-time service SLOs

5. Performance debugging

6. Post-incident forensics

Hubble and the rest of the observability stack

Comparison with the alternatives

Operational traps

Cardinality in Prometheus

L7 visibility costs CPU

Hubble Relay without HA

Encryption status reporting depends on the Cilium config

UI exposed without auth

Storage that does not scale

What we have not covered

3. Hubble CLI (`hubble`)

Full example: blocking writes to `/etc/passwd` in the prod namespace

1. Execution auditing (`execve`)

The `tetra` CLI

Permanent `monitoring`

`Sigkill` in critical namespaces

Missing `rateLimit`

1. `CiliumBGPClusterConfig`

2. `CiliumBGPPeerConfig`

3. `CiliumBGPAdvertisement`

4. `CiliumBGPNodeConfig` (auto-generated)

5. `CiliumBGPNodeConfigOverride`

`routingMode: tunnel` mode with BGP

eBPF host routing vs `kube-proxy replacement`