Kubernetes on lo0 — Tech Blog

Hubble: eBPF network observability, state of the art 2026 and the new frontier with AI agents

Tue, 19 May 2026 06:00:00 +0200

TL;DR

Hubble is Cilium’s native network observability, built on the same eBPF programs Cilium uses for enforcement. It does not duplicate the datapath or instrument the kernel its own way: it listens to the hooks Cilium already has and produces structured flow logs with Kubernetes context included, covering pod, namespace, labels, service, policy verdict and L7 payload where applicable. It is what happens when someone decides that tcpdump with grep does not scale to 10,000 pods and builds a distributed system of their own (a Hubble server per node, plus Hubble Relay as the aggregator, plus a CLI and a UI) with practically zero overhead, because the capture was already happening. In 2026 it is at version 1.19.3 (April 2026), with Cilium 1.19 marking the project’s tenth anniversary; IP options tracing has arrived, along with filtering by encryption status, the drop event tagged with the exact NetworkPolicy that caused it (direct attribution), a stabilised field mask API, and the first wave of ML anomaly detection applied to flows for predictive security in IoT/5G clusters. And, most interesting of all for 2026: a new frontier appears where the same eBPF observes AI agents such as Claude Code, Gemini CLI and MCP agents, intercepting SSL/TLS and stdio without instrumenting the code, which turns the Cilium + Hubble + Tetragon + AgentSight stack into a complete set of tools for understanding what an agentic system does inside a cluster.

This article is part 3 of the eBPF series. Part 1: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack. Part 2: Tetragon: Cilium’s security cousin that sees every syscall in the kernel. Here we complete the observability quadrant: network with Hubble, process with Tetragon, AI agent with AgentSight.

The analogy: tcpdump that speaks Kubernetes

If you have administered networks over the last twenty years, tcpdump and Wireshark have been your daily bread. They capture packets on an interface, parse them, and let you filter with tcp.port == 443 and host 10.0.0.5. They work, they have been working since the 90s, and they are the first thing you open when something smells wrong.

Now stick tcpdump on a Kubernetes cluster with 10,000 pods. The problems show up in order:

One tcpdump session per node. You wanted to “see the traffic between the frontend and the API”; you need SSH to every node, tcpdump on every NIC, synchronised timestamps, manual aggregation.
There is no K8s context. You see a packet from 10.244.5.7 to 10.244.8.42. Which pod was that? Which namespace? Which label? You have to correlate with kubectl get pod -A -o wide every single time.
No L7 understanding. You see a POST to HTTPS, and you cannot know the method and path because it is encrypted on the wire. If there is mTLS between pods, worse.
High cost: full packet capture with a copy to userspace slows down the datapath. Under dense traffic, you notice it.

Hubble is tcpdump redesigned for all of that. It reuses the eBPF programs that are already processing every packet (Cilium puts them there for enforcement) and, while they take their allow/deny decision, they emit a flow event with all the context: source and destination pod identity, namespace, labels, protocol, verdict, and, if Cilium has done L7 parsing via Envoy, the HTTP method, path, status code, DNS query, Kafka topic. That event travels through a ringbuffer to userspace, it is received by the Hubble server that lives inside the node’s Cilium agent, and it is exposed over gRPC. A separate service, Hubble Relay, aggregates the streams from all nodes and gives you a single cluster-wide API. On top of that: a CLI (hubble) and a web UI with a real-time service graph.

Zero additional copies. Zero duplicated parsing. And the result is understandable by anyone who knows what a Pod is.

Architecture: four pieces visible from outside

Hubble is made up of four logical components, all optional depending on what you want to do:

1. Hubble Server (embedded in every Cilium agent)

It lives inside the Cilium agent process (it is not a separate binary). Each node exposes a gRPC endpoint locally on the Unix socket /var/run/cilium/hubble.sock. The server listens to the events the eBPF programs emit to the ringbuffer, enriches them with Kubernetes metadata (which the agent already has in memory), and makes them available to consumers.

Enabling it: --set hubble.enabled=true in Cilium’s Helm chart. By default, the server is only accessible locally; if you want to consume it from another node, it has to be exposed (which is what Hubble Relay does).

2. Hubble Relay (aggregator)

It is a separate Deployment (typically 1 replica, scalable) that connects to every Hubble server in the cluster and aggregates their streams into a single API. When your CLI or UI asks for “the last 1000 flows in the cluster”, the Relay collects them in parallel from all nodes and returns the union.

Enabling it: --set hubble.relay.enabled=true. Without the Relay you only see the traffic of the node you are connected to, which is useful for local debugging but not for a cluster-wide view.

3. Hubble CLI (`hubble`)

A Go binary that speaks gRPC with the Relay (or with a local Hubble server). It supports two main modes:

hubble observe: a real-time stream of flows, with very expressive filters (by namespace, pod, port, verdict, protocol, label).
hubble status: the state of the Hubble cluster (how many nodes connected, lag, flow rate).

And the equivalent of tcpdump’s pcap dump: hubble observe --output jsonpb > flows.json to process afterwards with jq or other tools.

4. Hubble UI

A web frontend that connects to Hubble Relay and shows:

A service graph in real time (which Pod talks to which Service, which protocols it uses, which verdict).
A filterable list of flows.
L7 details where there are any (HTTP method/path/status, DNS query/response).

Enabling it: --set hubble.ui.enabled=true. Useful for presentations to non-CLI teams; it does not replace the CLI for serious debugging.

What you see: the Hubble flow log from the inside

A Hubble flow in JSON format looks roughly like this (simplified):

{
 "time": "2026-05-19T03:12:45.182Z",
 "verdict": "FORWARDED",
 "source": {
 "ID": 5482,
 "identity": 24871,
 "namespace": "prod-api",
 "labels": ["app=checkout", "team=payments"],
 "pod_name": "checkout-7c9f-x8j2",
 "workloads": [{"name": "checkout", "kind": "Deployment"}]
 },
 "destination": {
 "ID": 12041,
 "identity": 18356,
 "namespace": "prod-db",
 "labels": ["app=postgres", "tier=primary"],
 "pod_name": "postgres-0"
 },
 "Type": "L3_L4",
 "l4": {
 "TCP": {
 "source_port": 41982,
 "destination_port": 5432,
 "flags": {"SYN": true}
 }
 },
 "node_name": "rke2-worker-03",
 "Summary": "TCP Flags: SYN"
}

When L7 parsing is active (via embedded Envoy or Hubble’s lightweight parser), the same flow adds:

"l7": {
 "type": "REQUEST",
 "http": {
 "code": 200,
 "method": "GET",
 "url": "/api/v1/cart/items",
 "protocol": "HTTP/1.1",
 "headers": [{"key": "user-agent", "value": "checkout/1.4.2"}]
 }
}

The protocols natively supported for L7 parsing:

HTTP/1.1 and HTTP/2 (including gRPC over HTTP/2).
DNS (queries and responses, with domains, types, response codes).
Kafka (topics, API keys).
TLS handshake (SNI, not the encrypted payload by default).
MySQL, Cassandra (with optional modules).

For HTTP and gRPC, Cilium can enable the embedded Envoy proxy for the flows you want to inspect (not all of them; it is selective via CiliumNetworkPolicy with L7 rules). Without Envoy there is lightweight parsing, but less detailed.

Verdict and drop attribution

Every flow has a verdict: FORWARDED, DROPPED, ERROR, AUDIT, REDIRECTED, TRACED, TRANSLATED. For the DROPPED case, Hubble includes a structured reason (drop_reason) and, since Cilium 1.19, the exact NetworkPolicy that caused it.

That last point changes day-to-day operations. Before, when a pod could not talk to another one, the debug flow was:

See the dropped flow in Hubble.
Look at every CiliumNetworkPolicy in the namespace.
Reason manually about which one, with which labels, is blocking it.

With Cilium 1.19’s attribution, the policy_match_info field tells you directly “it was dropped by the frontend-egress policy, rule 3”. You go from “Sherlock Holmes for 20 minutes” to “kubectl get -o yaml of that specific policy”.

Prometheus metrics and Grafana dashboards

Hubble also exposes aggregated metrics in Prometheus format, separate from the gRPC flow stream. Enabling it: --set hubble.metrics.enabled=true (Helm) plus a list of the set you want to export.

The usual metric groups:

flow: total flows by verdict, source/dest, protocol.
http: requests by method, response code, latency (histogram).
dns: queries, response codes, top-N domains.
tcp: handshakes, retransmissions, congestion window.
drop: drops by reason, with NetworkPolicy attribution.
port-distribution: histogram of active ports.
policy: hits by policy and verdict.

These metrics have rich K8s labels (source_workload, destination_workload, namespace, and so on) that make them pivotable in Grafana. There are prebuilt dashboards on Grafana Labs covering the common cases; importing one and having immediate visibility costs five minutes.

Cost: metrics with many K8s labels can blow up cardinality in Prometheus. For large clusters (>1,000 pods), it is worth reviewing which set you export and using drop rules in Prometheus to keep a lid on it.

Deployment: Helm on one screen

The canonical Cilium install with full Hubble:

# values.yaml
hubble:
 enabled: true
 metrics:
 enabled:
 - dns:query;ignoreAAAA
 - drop
 - tcp
 - flow
 - port-distribution
 - icmp
 - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
 serviceMonitor:
 enabled: true # auto-discovered by kube-prometheus-stack
 relay:
 enabled: true
 rollOutPods: true
 ui:
 enabled: true
 rollOutPods: true
 ingress:
 enabled: true
 className: cilium
 hosts:
 - hubble.example.local

And the install itself:

helm upgrade --install cilium cilium/cilium -n kube-system -f values.yaml

After installing, validate with cilium status (the Cilium CLI) that the Hubble section shows OK, and try your first flow with:

cilium hubble observe --namespace prod-api --pod checkout-7c9f-x8j2

State of the art in 2026

Cilium 1.19 was released in February 2026, marking the project’s tenth anniversary. Hubble reached version 1.19.3 on 22 April 2026. The relevant news:

Direct attribution of drops to a NetworkPolicy

Already covered above; it is probably the most valuable operational change in the release. Any dropped flow carries the name, namespace and specific rule of the policy responsible. It is also available via Prometheus metrics, which allows alerts of the form “policy X is dropping >N requests/second”.

Tracing with IP options

Hubble can now trace individual packets with IP options enabled. It is a mechanism similar to traceroute, but at L3: you put a mark on the packet and Cilium reports it every time the packet crosses a node or an eBPF decision. Useful for debugging multi-cluster paths, fabric mesh, or NetworkPolicies applied at different layers.

Filtering by encryption status

A new CLI flag: hubble observe --encryption-status=encrypted (or unencrypted). Useful for validating deployments with WireGuard or IPsec enabled pod-to-pod: you confirm that the traffic that should be encrypted is, and you spot regressions quickly.

Hubble field mask API stabilised

The field_mask lets you ask for only the parts of the flow you care about, hugely reducing bandwidth and processing when all you need is, say, source/dest and verdict. It used to be experimental; now it is stable and it is default-on in the CLI.

AI-driven anomaly detection (predictive security)

This is the most talked-about addition of 2026. Cilium 1.19 adds hooks so that an external consumer, typically an ML system, can process flows in streaming and detect statistical anomalies: pods that suddenly talk to new destinations, latency spikes on an API, odd DNS sequences. The detection part happens outside the Cilium agent (nobody wants heavy ML in the datapath), but Cilium exposes the flows with the pre-computed features the model needs. The published use cases focus on IoT and 5G, where traffic is high in volume and low in variety, ideal conditions for anomaly detection.

Scaling to 10,000+ pods

Cilium 1.19 has done serious work on scalability: Hubble Relay can now aggregate streams from hundreds of nodes without saturating; the default field_mask reduces inter-node bandwidth; and flows can be sampled under high load if your use is statistical analysis rather than forensic debugging.

Cilium 1.20 in development

Cilium 1.20 is in the development branch. The most relevant bits for Hubble:

Unification of preferIpv6: the hubble.preferIpv6 flag was deprecated in favour of the global preferIpv6 that applies to every Cilium component.
tetragon-python SDK: although it belongs to Tetragon, not Hubble, it sets a trend, namely eBPF policies written in Python instead of YAML. Hubble will probably follow a similar path.

The new frontier: eBPF and AI agents

So far, the classic Hubble content. But there is a 2026 twist worth covering, because it closes the loop with the other series on this blog.

When a Kubernetes cluster starts running AI agents, such as Claude Code, Gemini CLI, or LangGraph-based agents calling APIs and MCP servers, the observability problem changes shape. It is no longer enough to know “which pod talked to which pod” (that is Hubble) or “which process ran what” (that is Tetragon). You need to know:

Which external APIs the agent is calling and with which prompts.
Which MCP tools it is invoking, with which arguments.
How many tokens it consumes, which model it picks, how much it costs.
Whether the agent drifts from expected behaviour (out-of-policy queries, jailbreak attempts, secret leakage).

The traditional solutions, instrumenting the agent’s code with OpenTelemetry or parsing structured logs, do not work well when the agent is a third-party binary (Anthropic’s Claude Code, Google’s Gemini CLI) or when the MCP servers live in other languages with stdio as their transport.

AgentSight: zero-instrumentation for LLM agents

AgentSight (a project from the eunomia-bpf group, the same ecosystem behind several high-profile eBPF runtimes) attacks this problem with the same philosophy as Hubble: do not instrument; listen. It puts eBPF hooks at two critical points:

uprobes on SSL/TLS libraries (libssl, boringssl, rustls). It captures the plaintext before encryption on send and after decryption on recv. For an HTTP call to https://api.anthropic.com/v1/messages, AgentSight sees the full JSON of the prompt and the response without decrypting anything in transit, simply because it has reached the syscall level before the TLS layer does its work.
stdiocap BPF: it captures read, write and dup on a process’s stdin/stdout/stderr file descriptors. This is what makes it possible to observe MCP servers that speak stdio with their client, the usual pattern for local MCP servers. You capture the JSON-RPC going back and forth without either the client or the server knowing.

Reported overhead: <3% CPU, comparable to Hubble in its own regime.

How it fits with Hubble and Tetragon

The three complement each other cleanly:

Hubble tells you: “the agent’s pod opened a TCP connection to api.anthropic.com:443 with verdict ALLOW”.
Tetragon tells you: “the claude-code process with PID 1843 did a connect() to that IP” (plus the binary, the arguments, the pod namespace).
AgentSight tells you: “the HTTPS content of that connection was a prompt messages=[{role:'user', content:'analyze this repo and modify the firewall config'}] and the response included a tool call to read_file with argument /etc/passwd”.

It is the difference between flow, process and semantics. For a security team that wants to watch AI agents in production, all three are necessary. For anyone who wants to understand cost, all three are useful (Hubble for network latency, Tetragon for resource usage, AgentSight for tokens and chosen model).

Emerging use cases

The patterns consolidating in 2026:

Agent audit trail: recording every LLM call and every tool call for compliance, especially in regulated sectors.
Jailbreak and prompt injection detection: applying rules over the prompts captured by AgentSight (similar to Tetragon’s TracingPolicy, but over semantic content).
Cost accountability: seeing which team/agent consumes which tokens, without instrumenting.
Replay and debugging: reproducing an agent’s reasoning in production without asking it to run again (which is non-deterministic).

It is a young field, AgentSight is months old rather than years, but the “eBPF as zero-instrumentation observability” pattern is very clearly spreading beyond network and process. The coming year will see consolidation and, probably, native integration with Hubble.

Common Hubble use cases

Back to Hubble proper, the cases in which any organisation deploys it:

1. NetworkPolicy debugging

The classic use: “this pod cannot reach this Service”. Without Hubble, it meant SSH, tcpdump, comparing rules. With Hubble:

hubble observe --from-pod prod-api/checkout --to-pod prod-db/postgres --verdict DROPPED

If there are drops, you see the policy responsible (Cilium 1.19+). If there are no drops, the problem is not policy: it is DNS, routing or the target service.

2. Auditing inter-namespace communication

For compliance: validating that isolated namespaces are not communicating against what was declared.

hubble observe --from-namespace prod-payments --to-namespace 'NOT prod-db' --output json

3. Exfiltration detection

Outbound traffic to suspicious public destinations. Hubble detects them by IP/SNI, not by payload (which is encrypted):

hubble observe --to-fqdn 'NOT *.example.com' --to-fqdn 'NOT *.internal' --protocol tcp

Combined with Prometheus metrics and Grafana alerts, this gives you an exfiltration radar at zero cost.

4. Real-time service SLOs

The hubble:http:response_time_seconds metrics with source_workload, destination_workload, method and status_code labels allow SLO dashboards without needing to instrument the apps. The SRE sees the p95 latency of checkout → catalog directly.

5. Performance debugging

hubble:tcp:retransmissions_total and hubble:tcp:flags_total{flag="RST"} are early signals of network problems. A rise correlated with a latency regression points you at something in the infrastructure (NIC, switch, MTU) before you go down to investigate the app.

6. Post-incident forensics

Configuring Hubble to export flows to persistent storage (via OTLP to Tempo/Loki, or hubble observe --output jsonpb to S3) gives you forensic capability: if at T+30 days you detect that something was wrong at T, you can reconstruct the traffic.

Hubble and the rest of the observability stack

Hubble does not replace Prometheus, Loki, Tempo or Jaeger; it complements them:

Prometheus: receives Hubble’s aggregated metrics. Hubble exports a native Prometheus endpoint.
Loki: receives the structured flow logs if you export them as logs. Hubble has no native Loki exporter, but a Fluent Bit with an OTLP plugin or a custom one bridges it easily.
Tempo / Jaeger: the Cilium Operator has an OTLP exporter of flows in trace format (each HTTP/gRPC flow is a span). It integrates with Tempo or any other OTLP tracing backend.
Grafana: there are already public Hubble dashboards. Combined with Prometheus, Loki and Tempo, they give you a unified panel: metrics, logs, traces, all correlated by K8s labels.

The full-stack pile seen in production in 2026 (described in Building a Production eBPF Observability & Security Stack for Kubernetes in 2026):

Data: Cilium + Hubble (network), Tetragon (process), AgentSight (AI agent).
Pipeline: OTLP Collector as the single router.
Storage: Prometheus (metrics), Loki (logs), Tempo (traces).
UI: Grafana with domain-specific dashboards.
Alerting: AlertManager with rules over the Hubble + Tetragon metrics.

Comparison with the alternatives

System	Layer	Focus	Model
Hubble	L3-L7 network	K8s cluster with Cilium	eBPF, pull metrics, push flows gRPC
GKE Dataplane v2 obs	L3-L7 network	GKE managed	eBPF (Cilium-based, managed)
Tigera Calico Whisker	L3-L7 network	Cluster with Calico	eBPF + pcap, own UI
Tetragon	Process/syscall	K8s cluster	eBPF, push events gRPC
Falco	Process/syscall	K8s cluster	eBPF in userspace or kernel module
AgentSight	LLM agent	Agentic systems	eBPF (SSL uprobes + stdio)
Beyla (Grafana)	Application	App L7 + tracing	eBPF (uprobes on libs)
Pixie	App + system	Broad cluster visibility	eBPF + PXL script
Parca	CPU/mem profiling	Performance	eBPF profile sampling

If your CNI is Cilium, Hubble is the natural entry point and it does not compete with the rest: it complements them. For Calico clusters, Whisker is the equivalent. For profiling, Parca. For AI agents, AgentSight. The era of “one tool for everything” is passing: the modern stack combines several specialised pieces, all eBPF-based, exposed via OTLP.

Operational traps

Cardinality in Prometheus

Hubble metrics with all the K8s labels can blow up Prometheus. Measure cardinality before exporting everything. The most prolific metrics are flow and httpV2; start with drop and port-distribution and add the rest incrementally.

L7 visibility costs CPU

Enabling L7 parsing via embedded Envoy adds load to the agent (not to the base datapath, but yes to the node’s envoy proxy). For heavy HTTP traffic, measure. For flows where you only need L4, leave Envoy disabled.

Hubble Relay without HA

A single Relay replica is a single point of failure for the CLI and the UI (not for the local agent, which keeps working). For production, deploy with replicas: 2+ and topologySpreadConstraints so both do not go down together.

Encryption status reporting depends on the Cilium config

The new --encryption-status filter only gives real data if Cilium has encryption enabled (WireGuard or IPsec). Without that, everything is unencrypted and the filter contributes nothing.

UI exposed without auth

Hubble UI has no native auth. If you expose it through Ingress, there has to be authentication in front of it: OIDC via oauth2-proxy, mTLS, IP allowlist. It is not optional.

Storage that does not scale

If you keep flows for days for forensics, the volume is serious. For a cluster with 100 active pods, easily 1-10 GB/day of flow logs. Plan the lifecycle (compaction, retention, cold storage) before enabling it.

What we have not covered

Mesh / multi-cluster Hubble: aggregating flows from several Cilium clusters into a single Relay. Use case: a cross-cluster view, debugging a distributed service mesh.
hubble export: local persistence on the agent’s disk for forensics with low retention.
Anomaly detection with your own models: how to connect the gRPC stream to a custom ML consumer.
AgentSight in depth: the project deserves its own article. Next instalment.
eBPF for LLM serving profiling: how to measure vLLM’s TTFT, TPOT and throughput without instrumenting, using uprobes on libcudart.

References

Hubble and Cilium:

Hubble GitHub — main repo.
Hubble — Network Observability (Cilium docs) — official reference.
Cilium 1.19 release notes (InfoQ, feb 2026) — tenth anniversary and 1.19 news.
Cilium releases — all releases.
Hubble L7 HTTP Metrics — Grafana dashboard 19423 — ready to import.
End‑to‑end L7 Visibility with Cilium Hubble (cloud-cod.com, mar 2026).
Cilium Hubble Observability Platform Internal Analysis (Young-ju).
CiliumNetworkPolicy Python Hubble: L7 Visibility 2026 — one of the threads on the Python SDK.

State of the art 2026 and the full stack:

eBPF + AI agents:

AgentSight (GitHub eunomia-bpf) — the referenced project.
Harnessing eBPF for High‑Performance LLM Workloads (Klizo Solutions).

Cross-references:

Part 1: eBPF from zero to Cilium.
Part 2: Tetragon: Cilium’s security cousin.
LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators — where the traffic Hubble observes carries the prompts AgentSight inspects.

Tetragon: Cilium's security cousin that sees every syscall in the kernel

Tue, 19 May 2026 05:00:00 +0200

TL;DR

Tetragon is the runtime security and observability engine that the Cilium project published as a companion to the CNI. Its job is not to route packets, Cilium is already there for that, but to observe what happens inside the node’s processes in real time: which binary runs in each pod, which files it opens, which syscalls it invokes, which capabilities it asks for, which network connections it establishes, which kernel modules get loaded. It does this by loading eBPF programs into the kernel’s hook points (kprobes, tracepoints, uprobes, LSM hooks) and filtering the relevant events inside the kernel itself with a declarative language expressed as a CRD (TracingPolicy and TracingPolicyNamespaced). The result is a stream of events enriched with Kubernetes metadata (pod, namespace, labels) that costs less than 1% of CPU and, the thing that sets Tetragon apart from the competition, can block actions inside the kernel, killing the process with SIGKILL or overwriting a syscall’s return value, before they finish executing, with no race conditions. Against Falco (which parses syscalls in userspace, 5-10% overhead, detection-only), Tetragon is “cheaper and with enforcement”; against the bare kernel, it is “a declarative layer your operations colleague can read”. This article is the extensive introduction you need to take it on seriously: architecture, all the hooks and selectors, the operating modes, a guide to use cases (exec auditing, sensitive file access, container escape, cryptomining, rootkit detection, network observability) and the traps you see in production.

This article is part 2 of the eBPF series. Part 1, eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack, covered basic eBPF, the networking hooks (XDP, TC, sock_ops), how Cilium implements the datapath and the BGP Control Plane v2 CRDs. Here we take those same eBPF hooks and use them for something different: observing and, if needed, stopping what the cluster’s processes do.

The analogy: auditd on steroids in eBPF

Anyone who has spent a few years administering Linux has used auditd. It is the classic kernel subsystem for auditing syscalls: you configure a rule with auditctl (for instance, “monitor any open on /etc/shadow”) and the kernel sends events to a userspace daemon that persists them. It works, but it has two limitations that weigh heavily on modern Kubernetes clusters:

No Kubernetes context. auditd reports processes by PID and UID. Knowing which pod, which namespace, which image, which labels, the information that actually matters when responding to an incident, requires correlating afterwards with data from cri-o or containerd. It is operationally miserable.
No granular enforcement. auditd can generate events, but it cannot take the decision to kill the offending process before the syscall finishes. You leave that to a higher layer that reads the events, processes them and kills the process… if it gets there in time. A race by design.

Tetragon is auditd on steroids: the same conceptual ideas, hooks on syscalls and events to userspace, but implemented with modern eBPF, with filtering inside the kernel so you do not pay the cost of waking the daemon for every irrelevant syscall, with Kubernetes metadata injected by an agent that knows the cluster, and with actions executed inside the kernel itself without waiting for userspace to decide. If the rule says “kill any process that opens /etc/shadow from the prod namespace”, the decision is taken in the kernel kprobe and SIGKILL is delivered before the open completes. There is no race; there is no window between detection and action.

What Tetragon is, architecturally

Tetragon is an agent deployed as a DaemonSet (one pod per node) and a set of CRDs that define the policies to apply. The agent has four responsibilities:

Load eBPF programs into the hook points the active TracingPolicies demand.
Maintain a cache of Kubernetes metadata (pods, namespaces, labels) by reading the API server, so it can enrich every event with the right context.
Collect the events the eBPF programs emit (via ring buffers) and serialise them.
Export the events to configurable destinations: stdout JSON (typical in sidecars or log-collection agents), gRPC streaming (to consume them from Hubble or another consumer), a file, or Fluentd/Loki/SIEM.

The eBPF programs are not written by the user. Tetragon generates the bytecode from the TracingPolicies: it reads the declarative policy, decides which hooks to attack, which arguments to read from the kernel, which filters to apply inline and which actions to execute. The user only writes YAML.

This separation of declarative policy → generated eBPF bytecode is what makes Tetragon usable. Writing eBPF programs by hand is a specialist’s job; writing a TracingPolicy is the job of an SRE with a good example in front of them.

The two CRDs: TracingPolicy and TracingPolicyNamespaced

Tetragon exposes exactly two main CRDs:

TracingPolicy (cluster-scoped, cilium.io/v1alpha1): applies to the whole cluster, every node, every pod. Suitable for platform policies (the whole cluster must be audited the same way): for example, “log every execve in every pod” or “kill any process that tries to load a kernel module”.
TracingPolicyNamespaced (namespaced, same group and version): defined inside a namespace and applied only to the pods of that namespace. Suitable for policies with per-tenant autonomy: for example, “in the prod-payments namespace, kill any outbound connect to an IP outside the corporate range”.

Both CRDs have exactly the same internal structure. The difference is one of scope. The distinction was introduced precisely to allow multi-tenancy: the central security team defines cluster-wide TracingPolicy objects and each tenant can add its own with TracingPolicyNamespaced without needing cluster-admin permissions.

Anatomy of a TracingPolicy

A policy is made up of:

Hook points: which kernel events to observe.
Arguments: which data to read when the hook fires.
Selectors: filters evaluated inside the kernel to discard irrelevant events and, optionally, execute actions when they match.

Supported hook points

The official documentation lists five families of hook points:

kprobes: hook a kernel function. Syscalls are a particular case (when syscall: true) because their ABI differs from that of internal functions. Typical examples: sys_open, sys_openat, sys_connect, tcp_connect, do_mount, commit_creds. It is the most versatile hook and the one used 80% of the time.
tracepoints: hook static tracepoints compiled into the kernel. More stable across kernel versions than kprobes (they do not depend on function names that can change). Examples: syscalls/sys_enter_openat, sched/sched_process_exec.
uprobes: hook functions in userspace libraries or binaries. They serve to observe runtime primitives such as libssl functions, libc, the Go runtime, the JVM.
USDT tracepoints (User Statically Defined Tracepoints): static tracepoints defined in userspace binaries (like those MySQL, PostgreSQL and OpenJDK expose). Useful for application observability.
lsmHooks (LSM, Linux Security Module): hooks of the LSM subsystem, where SELinux/AppArmor plug in. They allow security policies very similar to traditional MAC but programmable with eBPF. Example: file_open, inode_unlink, socket_bind.

Arguments

Every hook can read the arguments of the function it is attached to. The supported types cover the primitives (int, uint64, bool, string, char_buf) and higher abstractions (file, path, sock, linux_binprm, capability, bpf_attr, cred). The high-level types are pointers to kernel structures that Tetragon knows how to parse; instead of having to read an offset, you write type: file and Tetragon gives you the full path of the descriptor’s file.

There is an important capability detail: on kernels ≥ 5.4, Tetragon can read up to 327,360 bytes of an argument if the large buffers flag is enabled. That is the difference between being able to audit execve with all of its long argv complete versus truncating them at 256 bytes and losing context.

Selectors: filtering in the kernel

Selectors are what make Tetragon cheap. Without them, every syscall on the node would fire an event that would travel kernel → ring buffer → agent → processed → filtered → discarded. With selectors, the filtering happens inside the eBPF program itself, in the kernel, and only the events that matter reach userspace.

The available selectors include:

matchArgs: filters by an argument’s value. Operators: Equal, NotEqual, Prefix, Postfix, GreaterThan, LessThan, Mask, SPort (source port), DPort (dest port), Family (AF_INET vs AF_INET6), State (socket state).
matchPIDs: filters by PID; useful for targeted observation.
matchBinaries: filters by the binary executing the syscall (absolute path), with Operator: In, NotIn, Prefix. Essential for avoiding noise from legitimate system processes.
matchNamespaces: filters by Linux namespace (Pid, Mnt, Net, Ipc, Cgroup, User). It allows policies specific to processes in containers versus the host.
matchCapabilities: filters by the process’s effective capabilities. Block actions requiring CAP_SYS_ADMIN that run in pods that should not have them.
matchNamespaceChanges: detects namespace changes (typical of container escape).
matchCapabilityChanges: detects capability changes (privilege escalation).
matchActions: the actions executed when all the preceding matchers hit.

Actions: from a simple Post to Sigkill

When a selector matches, an action is executed. Tetragon defines several:

Post: emits an event to userspace (the observability case). It supports rateLimit to avoid flooding the agent if the condition fires a thousand times per second. The syntax accepts 5 for 5 seconds, 5m for 5 minutes, 1h for 1 hour.
Sigkill: sends SIGKILL to the offending process from inside the kernel, before the syscall completes. This is the only thing that guarantees enforcement without a race.
Override: overwrites the syscall’s return value. Useful for making the process believe the syscall failed (Override -EPERM) without killing it. A better experience for apps that can handle errors; worse for apps that assume success.
Signal: sends any arbitrary signal (not just SIGKILL).
NoPost: does not emit an event, useful when combined with another selector that does emit and you only want the action without duplicated telemetry.
FollowFD and UnfollowFD: mark a file descriptor to follow its lifecycle and enrich subsequent events with the original path. Useful for auditing “which process read this file after opening it”.
TrackSock and UntrackSock: the same for sockets.
GetUrl and DnsLookup: make HTTP requests or DNS resolutions from the kernel. Designed for integrations with external systems (security webhooks, IP reputation lookups).
NotifyEnforcer and CleanupEnforcerNotification: communication with Tetragon’s enforcement subsystem for complex actions.

Modes: detection vs enforcement

A policy can be declared in one of two explicit modes:

enforce: enforcement actions (Sigkill, Override, Signal) are active. This is production.
monitoring: enforcement actions are ignored; only Post events are emitted. This is the “let us see what would happen if this were switched on” mode, critical for testing policies without breaking applications.

Control is done with the spec.options[].name: policy-mode field and value: monitoring or enforce. It is the best practice: start in monitoring, collect events for days, tune the selectors until no false positives come out, and only then switch to enforce.

Full example: blocking writes to `/etc/passwd` in the prod namespace

A realistic policy, commented line by line:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-passwd-write
 namespace: prod
spec:
 kprobes:
 - call: "fd_install"
 syscall: false # kernel function, not a syscall
 args:
 - index: 0
 type: int  # file descriptor
 - index: 1
 type: "file" # struct file*
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values:
 - "/etc/passwd"
 matchActions:
 - action: Sigkill  # kills the process
 rateLimit: "1m" # max once per minute
 options:
 - name: policy-mode
 value: enforce  # enforcement mode active

fd_install runs every time a process obtains a new file descriptor; the second argument is the file’s file struct. Tetragon knows how to resolve it to its absolute path. The matchArgs compares that path with /etc/passwd. If it matches, Sigkill kills the process before the descriptor even becomes usable. rateLimit: 1m stops the agent from saturating if a malicious application tries it in a loop.

Common use cases

Now to real use. These are the six cases that show up in any serious Tetragon deployment in 2026.

1. Execution auditing (`execve`)

The most basic use case and, even so, the most valuable. Which binaries are running in each pod? In a container that is supposed to run only nginx, suddenly seeing an sh or a wget is almost always a red flag.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-execve
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm  # struct linux_binprm*
 selectors:
 - matchActions:
 - action: Post  # events only, no enforcement

With no filters: every execve in the cluster generates an event. With K8s metadata, the event includes pod, namespace, container, image, labels. You turn it into a stream of events towards your SIEM and set up rules: “alert if I see sh, bash, nc, curl, wget or python running in any pod of the prod-api namespace”.

An enforcement variant: instead of Post, use matchBinaries with Operator: NotIn and a whitelist, plus Sigkill if the binary is not on the list. A very rigid box, but effective in pods that are “single-binary” (like a Go microservice).

2. Sensitive file access

Detecting (or blocking) reads and writes on critical files: /etc/shadow, /etc/kubernetes/, Secret mounts, /var/run/docker.sock, /proc/*/cmdline.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: sensitive-file-access
spec:
 kprobes:
 - call: "security_file_open" # LSM-ish via kprobe
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/etc/shadow"
 - "/var/run/secrets/"
 - "/var/run/docker.sock"
 matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/kubelet" # legitimate kubelet access
 matchActions:
 - action: Post

The matchBinaries: NotIn is important: kubelet and other legitimate node agents access these paths constantly and would generate noise. We filter those out in the kernel.

In enforcement: swap Post for Override with argError: -1 (EPERM), so that the open fails but the offending process stays alive and produces the error so that tracing tools pick it up.

3. Unauthorised outbound network connections

Detecting outbound connections to destinations outside the corporate range. Useful for spotting data exfiltration or malware command-and-control.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-external-egress
 namespace: prod
spec:
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr" # destination NOT in these CIDRs
 values:
 - "10.0.0.0/8"
 - "192.168.0.0/16"
 - "172.16.0.0/12"
 matchActions:
 - action: Sigkill
 options:
 - name: policy-mode
 value: enforce

This kills any attempt at a TCP connection to an IP that is not in the corporate CIDRs, in the prod namespace. Cilium already does this with NetworkPolicy, but Tetragon has two complementary advantages:

It gives you the process that attempted the connection, not just “pod X tried to connect to Y”.
It also works for exotic protocols where NetworkPolicy is less expressive.

4. Container escape detection

Container escape is the operational nightmare: a process inside a container manages to break the isolation (via a kernel exploit, a badly set capability, a misconfigured hostPath mount) and gain access to the host. Three typical signals:

A namespace change on the process (it leaves the container’s pid namespace).
setns or unshare in non-init processes.
Access to /proc/1/root or /dev/ from a container.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-container-escape
spec:
 kprobes:
 - call: "__x64_sys_setns"
 syscall: true
 args:
 - index: 0
 type: int
 - index: 1
 type: int
 selectors:
 - matchNamespaces:
 - namespace: Pid
 operator: NotIn
 values: ["host_ns"] # only processes NOT in the host pid namespace
 matchActions:
 - action: Sigkill
 - call: "__x64_sys_unshare"
 syscall: true
 args:
 - index: 0
 type: int
 selectors:
 - matchNamespaceChanges:
 - unshare: true
 matchActions:
 - action: Post

__x64_sys_setns targeting the host namespace from a process in a container is practically always malicious (legitimate containers do not need this at runtime).

5. Cryptomining

Mining processes have fairly recognisable profiles:

Processes with names like xmrig, minerd, cgminer, or legitimate processes such as python running CPU-intensive scripts.
Outbound connections to known mining pools (a public list of IPs and domains).
Anomalous use of /dev/cpu_dma_latency to avoid throttling.

A combined policy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-cryptomining
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm
 selectors:
 - matchArgs:
 - index: 4
 operator: "Postfix"
 values:
 - "/xmrig"
 - "/minerd"
 - "/cgminer"
 matchActions:
 - action: Sigkill
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "DPort"
 values: ["3333", "5555", "7777", "14444"] # common pool ports
 matchActions:
 - action: Post  # log only

The double policy: kill binaries with the classic names (belt) and log connections to pool ports (braces), so you also get an alert when someone renames xmrig to nginx-helper or uses exotic ports.

6. Detecting rootkits and suspicious kernel modules

Modern rootkits load kernel modules to patch functions (hide processes, hide network connections, hide files). Detecting them:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: kernel-module-load
spec:
 kprobes:
 - call: "do_init_module"
 syscall: false
 args:
 - index: 0
 type: "string"
 selectors:
 - matchActions:
 - action: Post
 - call: "security_kernel_read_file"
 syscall: false
 args:
 - index: 0
 type: "file"
 - index: 1
 type: int
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values: ["READING_MODULE"]
 matchActions:
 - action: Post

In a “well configured” Kubernetes cluster no new kernel modules get loaded at runtime; any event here is highly suspicious. Combine with enforcement on machines where modules should be fixed: Sigkill for whoever tries to load one.

Bonus: detecting third-party modification of eBPF maps

As a 2025-2026 trend: loading malicious eBPF programs to hide presence. Tetragon can observe the bpf syscall:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-bpf-syscalls
spec:
 kprobes:
 - call: "__x64_sys_bpf"
 syscall: true
 args:
 - index: 0
 type: int  # bpf cmd
 - index: 1
 type: bpf_attr
 selectors:
 - matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/cilium-agent"
 - "/usr/bin/tetragon"
 - "/usr/bin/bpftool"
 matchActions:
 - action: Post

Any process that is not one of the legitimate agents loading eBPF programs: you want to know about it.

Comparison with Falco

Falco is the closest competitor: it is also runtime security for Kubernetes, also originally based on eBPF (and before that on kernel modules), also with declarative policies. Three years ago they were functionally similar. In 2026 the divergence is clear:

Dimension	Tetragon	Falco
Philosophy	Cilium-native, integrated	Standalone, generic
Filtering	In the kernel (eBPF)	Parsing in userspace
Typical overhead	<1% CPU	5-10% CPU
Enforcement	Yes, in-kernel (Sigkill, Override)	Not native (depends on plugins)
Race conditions	No (action atomic with the syscall)	Yes, in enforcement via plugins
False positive rate	Low (K8s context in the kernel)	Higher (parsing afterwards)
Detection latency	5-26 ms	~10 ms (more constant)
Ecosystem maturity	Young, growing	Mature, plenty of material
Community	Cilium / CNCF Incubating	CNCF Graduated
Integrations	Hubble native	Falcosidekick, many
CRDs per policy	TracingPolicy / Namespaced	No CRDs; rules in YAML

When to choose each one:

Tetragon if you already use Cilium, if you need enforcement in the kernel (not detection-only), if overhead matters to you (workloads with many syscalls), and if you value the Hubble integration. Container escape and cryptomining detection is where its advantage over Falco shows up most.
Falco if you want a tool independent of the CNI, if you need the catalogue of ready-made rules and the wide community, if your cluster is not Cilium, or if the integration with SIEMs and ready-made notifiers (Falcosidekick) saves you work.
Both if the organisation is large: Falco for breadth of detection, Tetragon for surgical enforcement on critical workloads. That is what you see most in companies that have spent years with Falco and add Tetragon for specific cases.

Hubble + Tetragon: unified observability

Hubble is Cilium’s traffic observability component: it shows L3-L7 flow logs with zero impact on latency. Tetragon exposes its events over gRPC with the same format and vocabulary as Hubble, which allows you to:

See them in the same UI (Hubble UI shows Tetragon events as one more “layer”).
Correlate network events (Hubble) with process events (Tetragon) on the same timeline.
Export them together to Loki/Tempo/SIEM as a single stream.

The key synergy: Hubble tells you “this pod made a TCP connection to 1.2.3.4:80”. Tetragon tells you “this pod ran curl 1.2.3.4 from a bash binary launched by pid 1234”. Together they give you the full story.

Deployment and operation

Helm

The canonical install with Helm:

helm repo add cilium https://helm.cilium.io
helm install tetragon cilium/tetragon \
 --namespace kube-system \
 --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
 --set tetragon.exportFileMaxSizeMB=50 \
 --set tetragon.exportFileRotationInterval=24h

Tetragon deploys its DaemonSet, its CRDs and a service for Hubble. By default, it exposes the events on the agent pod’s stdout (any cluster log aggregator picks them up).

The `tetra` CLI

Tetragon ships a CLI called tetra for interactive investigation:

# real-time stream of the node's events
tetra getevents -o compact --pods <pod-name>

# structured JSON events to process with jq
tetra getevents -o json --since 5m --namespace prod

# view loaded policies
tetra tracingpolicy list

It is the best tool for debugging policies in monitoring before moving them to enforce.

Exporting to a SIEM

Three usual routes:

stdout + log aggregator: the agent writes JSON to stdout, Fluent Bit/Vector picks it up and sends it to Splunk/Datadog/Elastic. Simple, works with any logging infrastructure.
gRPC streaming: for low-latency integrations. A gRPC consumer of your own or Hubble Relay.
File + rotation: for air-gapped environments or regulatory audits that require persistent logs with controlled rotation.

Performance

Published benchmarks consistently place Tetragon at <1% of the node’s CPU under real workloads, compared with 5-10% for Falco under the same workloads. The reason is the architectural separation: Tetragon filters in the kernel and only carries the events that actually matter to userspace; Falco carries every syscall to userspace and filters there. On clusters with thousands of pods making hundreds of thousands of syscalls per second, the difference shows up on the bill.

Common operational traps

Permanent `monitoring`

The biggest trap is never reaching enforce: starting well with policies in monitoring, collecting events, tuning selectors, and then never switching. The result: you have detection without prevention, exactly what Falco gave you without paying Tetragon’s complexity. If you are going to use Tetragon, plan the road to enforce for the critical policies.

Selectors that are too lax

A policy with a single matchActions: Post and no specific selectors generates events for every syscall of the chosen hook. On a serious node that means tens of thousands per second, which fill logs, saturate exporters and hide the signal in the noise. Always start with strict filters (matchBinaries, matchNamespaces, matchPIDs) and open up once you know what you are looking for.

A kernel that is too old

Tetragon needs modern eBPF features. Kernels < 5.4 do not have the large buffer support (needed for execve with complete argv). Kernels < 5.10 do not have many of the LSM hooks. Kernel 5.15+ is the recommended minimum for production and 6.1+ to have every feature.

Hooks on renamed kernel functions

kprobes are tied to kernel function names that can change between versions. A policy that uses __x64_sys_setns can fail silently on a kernel where the function is called __do_sys_setns. Solutions: use static tracepoints where they are available (more stable), or keep alternative policies with several call entries for compatibility.

`Sigkill` in critical namespaces

Applying Sigkill to processes in kube-system or cilium-system can break the cluster. Enforcement policies must explicitly exclude the platform namespaces with matchNamespaces Operator: NotIn, or limit the scope with TracingPolicyNamespaced to make sure they do not act on systems they should not.

Missing `rateLimit`

A policy without a rateLimit on Post can suffer a catastrophic fan-out if the condition is met millions of times in an instant (typical in attack loops or application bugs). The agent saturates, events are lost, logs overflow. Always put a sensible rateLimit on detection policies, especially on high-frequency hooks such as tcp_connect or execve.

What we have not covered (upcoming articles)

eBPF LSM hooks in depth: how they relate to SELinux/AppArmor and when Tetragon is the right tool versus classic MAC.
Hubble UI with a Tetragon overlay: configuring the UI to show process observability and network observability on the same timeline.
Integration with OPA/Kyverno: how Tetragon complements admission policy engines (Kyverno validates at admission; Tetragon validates at runtime).
Forensics with eBPF: combining Tetragon with tools such as Beyla or OpenTelemetry to trace the full chain of an incident from the initial connection to the final syscall.

References

Official documentation (May 2026):

Tetragon — official site — entry point.
Tetragon docs — Tracing Policy — conceptual reference.
Tetragon docs — Hook points — kprobes, tracepoints, uprobes, LSM, USDT.
Tetragon docs — Selectors — full filter reference.
Tetragon docs — Enforcement Mode.
Tetragon docs — Kubernetes Identity Aware Policies — TracingPolicyNamespaced.
Tetragon GitHub.

Comparisons and analysis:

Comparative Analysis of eBPF-Based Runtime Security Monitoring (SciTePress paper 2025) — a benchmark with independent numbers.
Best eBPF Security Solutions for Kubernetes (ARMO, 2026) — Falco vs Tetragon vs KubeArmor comparison.
Falco vs. Tetragon (Asim Mirza, Medium) — analysis with use cases.
Deep Dive into Tetragon (A Security Engineer) — a walk through the inside of the agent.
Tetragon Series, Part 2: Enforcing Sensitive File Access (Medium).

Ecosystem:

Cilium Hubble — network observability.
Falco — official site — the other big name in the field.
KubeArmor — the third option, with AppArmor + eBPF.

Cross-references:

Part 1 of the series: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack — the eBPF fundamentals we take as read here.
Kubernetes with Cilium BGP: services reachable without Ingress — the starting point of the Cilium ecosystem on this blog.

eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack

Tue, 19 May 2026 04:30:00 +0200

TL;DR

eBPF is a sandboxed virtual machine inside the Linux kernel that runs verified code at well-defined hooks: kprobes, tracepoints, socket events, network drivers. Before eBPF, changing kernel behaviour meant recompiling it or loading an arbitrary module; with eBPF, you load a small program that passes a formal verifier and runs at native speed with memory safety. In networking, this translates into the fact that the packet does not have to travel through the traditional TCP/IP stack: an eBPF program in the NIC driver (XDP) can drop, forward or rewrite the packet before the kernel has done its first alloc; a program on cgroup hooks (sock_ops) can redirect connections to another socket without the packet ever leaving the machine. Cilium is the CNI that has taken this to its logical conclusion: it replaces kube-proxy with pure eBPF (O(1) instead of the O(N) of iptables), routes pod-to-pod without VXLAN where it can, evaluates Network Policies with BPF maps, and since 1.16 it has remade its BGP control plane with a new set of CRDs, CiliumBGPClusterConfig, CiliumBGPPeerConfig, CiliumBGPAdvertisement, CiliumBGPNodeConfigOverride, which replace the monolithic CiliumBGPPeeringPolicy that is already deprecated. This post goes down the three layers (basic eBPF → eBPF networking → Cilium) and ends with the operational CRDs.

The analogy: signed plugins for the kernel

Think of the browser. Twenty years ago, extending a browser meant compiling a native binary and loading it: any extension could crash it, corrupt memory, read your bank cookies. Today, extensions are JavaScript in a sandbox with a manifest that declares permissions, a runtime that enforces the isolation and a store that signs the code. The extension does not touch the browser binary; it lives in a controlled world and can only talk to the browser through defined APIs. Result: massive extensibility with a bounded attack surface.

eBPF is exactly that for the Linux kernel. Loading a classic .ko module means loading native code with full access to kernel memory: one bug and the system is gone. eBPF is a bytecode VM with a static verifier, a controlled allocator, JIT to native hardware after passing the verifier, and a set of kernel “helpers” it can call. The eBPF program can read the kernel memory the verifier allows it to read, and only that. It cannot enter infinite loops (the verifier demands that it terminate). It cannot jump to arbitrary addresses. It cannot dereference pointers without having validated them first. And, most importantly: it is user code, loaded at runtime, executing inside the kernel at native speed.

The consequences are visible from miles away. Before, observing traffic in production meant patching the kernel or loading a risky module. Today, bpftrace -e 'tracepoint:net:net_dev_xmit { @[args->dev->name] = count(); }' gives you a histogram of packets per interface in three lines and zero downtime. Before, replacing iptables with something faster meant rewriting the netfilter subsystem. Today, Cilium loads 60 KB of eBPF bytecode into XDP and unseats iptables with a hash map.

Basic eBPF: what it is and what it is not

The origin and the scope

The name comes from Berkeley Packet Filter, a 1992 idea (McCanne and Jacobson) for filtering packets with a mini-bytecode that tcpdump used internally. In 2014, Alexei Starovoitov renamed it eBPF and extended it enormously: 11 64-bit registers instead of 2 32-bit ones, a 512-byte stack, maps as structures shared with userspace, JIT to native hardware, and a far more sophisticated formal verifier. From being a packet filter, it became a generic kernel extensibility mechanism.

Today eBPF is used for four things:

Networking: XDP, TC, cgroup hooks, socket ops, lightweight tunnels.
Observability: kprobes, uprobes, tracepoints, USDT. The basis of projects like bpftrace, bcc, Pixie, Parca.
Security: BPF LSM (Linux Security Module in eBPF), syscall blocking with seccomp-bpf. Falco, Tetragon, Tracee.
Scheduling: sched_ext (kernel 6.12+), process schedulers written entirely in eBPF. Still at a very early stage.

The VM

An eBPF program is compiled from C (or Rust, or Go with cilium/ebpf) to eBPF bytecode, not to x86/arm64 directly. The kernel loader (via the bpf() syscall) passes that bytecode through the verifier:

It reconstructs the control flow graph.
It performs static analysis of every possible path: every instruction has to be reachable, every memory access has to be within known bounds, every pointer has to have been validated.
It rejects loops without a known upper bound. Recent kernels admit bounded loops (the bpf_loop helper), but the counter is always finite.
It rejects calls to helpers or kfuncs that the hook’s program type does not allow.

If the verifier accepts the program, the JIT translates it to the host’s native code (x86, arm64, etc.) and it stays attached to its hook. From then on it runs every time the hook’s event occurs, with no context switch to userspace, with no syscall cost. Latencies on the order of hundreds of nanoseconds per invocation.

Maps: the bridge to userspace

An isolated eBPF program is not much use. What makes it useful are maps: data structures shared between the kernel program and userspace. There are several types:

BPF_MAP_TYPE_HASH, BPF_MAP_TYPE_LRU_HASH: hash tables with or without LRU eviction.
BPF_MAP_TYPE_ARRAY, BPF_MAP_TYPE_PERCPU_ARRAY: arrays, optionally per-CPU to avoid contention.
BPF_MAP_TYPE_RINGBUF, BPF_MAP_TYPE_PERF_EVENT_ARRAY: channels for streaming events to userspace.
BPF_MAP_TYPE_PROG_ARRAY: arrays of eBPF programs for tail calls (chaining programs without returning to the base kernel).

Userspace reads and writes these maps via bpf() syscalls; the kernel program reads and writes them directly. It is the basis of any eBPF system: the kernel program collects data into a map, the userspace daemon reads it. Cilium does exactly this: the userland agent (Go) manages the policy and translates it into map entries; the eBPF programs living in XDP/TC read the maps and apply the decisions.

CO-RE: compile once, run on any kernel

A classic nightmare of kernel modules: they are tied to the exact kernel version they were compiled against. Distributing a precompiled module for a fleet of machines with different distros was impossible.

eBPF solves this with CO-RE (Compile Once, Run Everywhere): the bytecode includes relocations that the loader resolves on each specific kernel by consulting BTF (BPF Type Format), a representation of the kernel’s struct layouts that the kernel itself publishes. Result: a single eBPF binary works on kernels 5.10, 5.15, 6.1 and 6.8 without recompiling, because the loader adjusts the struct access offsets at runtime.

This is what has allowed productive eBPF distributions to exist. Without CO-RE, every kernel would be a porting project.

eBPF in networking: the hooks that matter

Inside the Linux network subsystem, eBPF has several hooks. The ones relevant to CNIs:

XDP — eXpress Data Path

XDP is the earliest hook: it runs in the NIC driver, before the packet enters the kernel proper. There is no sk_buff (the struct the rest of the kernel uses to represent packets); there is only a pointer to a RAM buffer with the received bytes.

The actions an XDP program can return:

XDP_DROP: discard the packet immediately. The driver drops it and frees the buffer. Cost: nanoseconds. Use case: DDoS mitigation. Cloudflare processed more than 8 million packets/second per CPU with XDP for dropping SYN floods.
XDP_PASS: let the packet continue to the normal kernel. It becomes an sk_buff and enters the traditional stack.
XDP_TX: send it back out on the same interface after possible modifications. Useful for L4 load balancers that rewrite the destination and return it.
XDP_REDIRECT: send the packet to another interface or to a map (to forward to userspace via AF_XDP, or to another NIC, or to a pod’s veth).
XDP_ABORTED: error (increments a counter, drops).

Real use cases:

Cloudflare L3 DDoS protection: XDP rules that drop millions of packets/s.
Facebook Katran: an L4 load balancer that rewrites the destination IP and returns it on the same interface. Handles 10× more connections per server than classic IPVS.
Cilium XDP acceleration: Service load balancing at the lowest layer possible.

TC (Traffic Control) — clsact with BPF

XDP is very fast but limited: the packet does not have an sk_buff yet and many decisions (conntrack, NAT, encapsulation with metadata) are easier when it does. The TC clsact with BPF hook runs after building the sk_buff but before the routing and netfilter decisions. Actions:

TC_ACT_OK: the packet continues through the stack.
TC_ACT_SHOT: drop.
TC_ACT_REDIRECT: redirect to another interface.
TC_ACT_PIPE, TC_ACT_STOLEN: pipeline control for combining with other qdiscs.

Use cases:

Stateful network policy: Cilium evaluates L3-L7 policies in TC with the full sk_buff and conntrack available.
Marking and QoS: traffic marking so the scheduler applies priorities.
Overlay encapsulation: adding VXLAN/Geneve headers when the mode is tunnel.

XDP and TC combine: XDP for the cheap and early stuff (DDoS, simple LB), TC for what needs an skb and state.

Cgroup hooks: sock_ops and CGROUP_SOCK_ADDR

The most radical conceptual step: hooks that are not in the network layer but in the socket layer. Relevant types:

BPF_PROG_TYPE_CGROUP_SOCK_ADDR: invoked when a process in a cgroup calls connect(), bind(), sendto(). The eBPF program can rewrite the destination address before the connection goes out. This is what lets Cilium do Service load balancing without the packet entering the network stack: if the client tries to connect to 10.96.0.1:443 (a ClusterIP), an eBPF program on this hook rewrites the destination to the real IP of the backend pod before the syscall continues.
BPF_PROG_TYPE_SOCK_OPS: invoked on TCP events (creation, established, retransmission). It allows tuning socket parameters at runtime and, most importantly, pairing local sockets via bpf_sk_assign to shortcut without the packet travelling over the network.

This is the “third layer” of the bypass: it is not just faster, it is conceptually different. The packet is not built, not serialised, does not traverse the IP layer or the TCP layer. It is the difference between speeding up a road and discovering that for some journeys you do not need to take the car at all.

The long road: what the traditional TCP/IP stack looks like

To appreciate what eBPF saves, it is worth tracing a packet’s full journey through the Linux stack. Take the case “packet arrives on a NIC, goes to a local process”:

NIC (DMA into the driver's ring buffer)
↓
driver: napi_schedule, poll, allocates sk_buff
↓
[XDP hook] ← if there is an XDP program, it is decided here
↓
netif_receive_skb
↓
__netif_receive_skb_core
↓
[TC ingress clsact + BPF] ← if there is a TC ingress program
↓
packet_type handlers (IP, ARP...)
↓
ip_rcv → ip_rcv_core
↓
[netfilter NF_INET_PRE_ROUTING] ← iptables PREROUTING
↓
routing decision (FIB lookup)
↓
[netfilter NF_INET_LOCAL_IN] or [NF_INET_FORWARD]
↓
tcp_v4_rcv → tcp_v4_do_rcv
↓
tcp_rcv_established
↓
sk_data_ready
↓
process reads with recv()/read()

Every arrow is a function call with a measurable cost. Every netfilter hook walks all the registered iptables/nftables rules. With kube-proxy in iptables mode and 5,000 Services × 10 endpoints each, there are on the order of 150,000 rules evaluated sequentially at NF_INET_PRE_ROUTING. Published benchmarks show latencies of tens of microseconds per packet in large Kubernetes clusters in the netfilter step alone, before the application receives anything.

And that is the normal path. On the way out the same thing happens in reverse: tcp_sendmsg → ip_output → NF_INET_LOCAL_OUT → routing → NF_INET_POSTROUTING → dev_queue_xmit → driver → NIC.

How Cilium skips this stack

Cilium does not eliminate the TCP/IP stack; it is still there for the cases that need it. What it does is shortcuts at the points where it hurts.

Shortcut 1 — XDP for the Service datapath

For a cluster with 5,000 Services, kube-proxy iptables has an O(N) cost in evaluating rules (even with iptables-restore --noflush and tricks, it is still linear in the number of chains the packet traverses).

Cilium replaces it like this:

Every Service and its endpoints live in an eBPF hash map.
When a packet comes in destined for a ClusterIP, Cilium’s XDP program does an O(1) lookup in that map and obtains the backend endpoint.
It rewrites the destination and does XDP_TX (returns it on the same interface towards the backend) or XDP_REDIRECT (sends it to the corresponding local pod’s veth).

This means the cost does not grow with the number of Services. 100 Services or 100,000, constant lookup in the map. Published benchmarks show latency reductions of 30-50% in clusters with many Services compared with kube-proxy iptables, and of an order of magnitude compared with IPVS in some cases.

Shortcut 2 — socket-LB: the packet is never built

Cilium 1.6+ introduced socket-level load balancing, based on cgroup hooks. It works like this:

When a pod calls connect(10.96.0.1:443) (a Service’s ClusterIP), the syscall enters the kernel.
Before the kernel builds anything network-related, an eBPF program on CGROUP_SOCK_ADDR/connect4 intercepts it and rewrites the destination address to the real IP of the backend pod.
The kernel carries on with the connect as if the client had written 10.0.0.42:8080 directly.

Why does it matter? Because when the backend pod is on the same node, this shortcut turns a call that would have involved:

syscall connect → kernel stack → veth → bridge → veth → kernel stack → syscall accept

into:

syscall connect (with the destination rewritten) → direct loopback

The TCP/IP stack is literally avoided. There is no encapsulated packet, no journey through veth pairs, no netfilter. L7 pod-to-pod latencies on the same node drop to local communication levels (~5-15 µs instead of ~30-50 µs for services with kube-proxy iptables and traditional veth).

Shortcut 3 — pod-to-pod direct routing

The traditional overlay mode (Flannel, Calico VXLAN) encapsulates every pod-to-pod packet in VXLAN/Geneve. Every packet carries an extra 50-byte header, requires encap/decap, and consumes MTU.

Cilium supports direct routing: the pod CIDRs are advertised to the underlying fabric (with BGP, which is where the control plane we will look at comes in) and the physical routers route the pod-to-pod packets without encapsulating. The packet leaves a pod with its original IP as source and the destination pod’s IP as dest, the node’s NIC hands it to the network, the network routes it, it arrives at the destination node and is delivered to the pod. Zero encap, full MTU, minimal latency.

Cilium does this via eBPF programs in TC that rewrite the necessary headers and decide whether the packet goes via encap or direct according to the policy configured per node.

Shortcut 4 — Network Policy in TC with maps

Network Policies in classic CNIs are usually translated into iptables rules, another factor that explodes linearly. Cilium evaluates them in eBPF programs that read identity maps: every workload has a numeric identifier computed from its labels, and the policy is a map (src_identity, dst_identity, port, proto) → allow|deny. One hash map lookup per packet.

This also enables Cilium’s L7 policies (HTTP, gRPC, Kafka filtering): the eBPF program recognises the L7 handshake, selectively redirects to the embedded Envoy proxy (which lives as a sidecar of the datapath, not as a pod sidecar) and only on that subset does it pay the cost of the L7 proxy. All the L3/L4 traffic stays on the eBPF fast path.

Cilium: the architecture

Cilium combines two planes:

Agent (Go): lives as a DaemonSet on every node. It is the “slow” part: it translates the intent expressed in CRDs (CiliumNetworkPolicy, CiliumBGPClusterConfig, etc.) into entries in eBPF maps. It talks to the Kubernetes API server to discover endpoints, services, pods. It embeds a GoBGP for the BGP control plane. It embeds an Envoy for L7 policies.
Datapath (eBPF): the programs loaded into XDP, TC, cgroup hooks. They are the “fast” part: they see every packet, read the maps the agent maintains, and decide in nanoseconds.

This separation is what makes Cilium operationally comfortable: the intent is expressed in YAML, the agent materialises it into maps, the maps are read by the datapath. If the agent goes down temporarily, the datapath keeps working with the last loaded configuration. As in any well-built control/data plane system.

BGP Control Plane v2: the CRDs you have to know

Cilium has had BGP support for several years. The first version used a single monolithic CRD, CiliumBGPPeeringPolicy, which mixed node configuration, peers, timers and advertisements into a single object. Since Cilium 1.16 there is BGP Control Plane v2, which breaks that configuration into separate CRDs with clear responsibilities. CiliumBGPPeeringPolicy (the cilium.io/v2alpha1 API) is deprecated and migration warnings appear in the operator logs if you still use it.

The new CRDs (the cilium.io/v2 API):

1. `CiliumBGPClusterConfig`

Defines BGP instances and the peers they connect to, from the cluster’s perspective. Which nodes apply this configuration is selected with a nodeSelector.

apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
 name: cilium-bgp-cluster
spec:
 nodeSelector:
 matchLabels:
 bgp-policy: rack-1  # only nodes with this label
 bgpInstances:
 - name: instance-65000
 localASN: 65000
 peers:
 - name: top-of-rack-1a
 peerASN: 64512
 peerAddress: 10.0.1.1
 peerConfigRef:
 name: tor-shared-config  # → reference to CiliumBGPPeerConfig
 - name: top-of-rack-1b
 peerASN: 64512
 peerAddress: 10.0.1.2
 peerConfigRef:
 name: tor-shared-config

A BGP instance is the abstraction “this node takes part in BGP with this local ASN and these peers”. Several can coexist on the same node (multi-instance for multi-VRF).

2. `CiliumBGPPeerConfig`

Defines the shared parameters of the peering: timers, address families, transport, MD5 password, graceful restart, etc. It is referenced from CiliumBGPClusterConfig via peerConfigRef. This avoids repeating the same configuration for every peer when there are dozens of them.

apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
 name: tor-shared-config
spec:
 timers:
 holdTimeSeconds: 30
 keepAliveTimeSeconds: 10
 connectRetryTimeSeconds: 5
 gracefulRestart:
 enabled: true
 restartTimeSeconds: 120
 families:
 - afi: ipv4
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp  # → binds to CiliumBGPAdvertisement
 - afi: ipv6
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp
 authentication:
 password:
 name: bgp-md5-secret  # Secret with the MD5 password
 key: password

A single CiliumBGPPeerConfig can be referenced by many different peers. You change timers or families in one place.

3. `CiliumBGPAdvertisement`

Declares which prefixes are advertised: the node’s pod CIDRs, the ClusterIPs and ExternalIPs of Services, the IPs assigned by CiliumLoadBalancerIPPool for type=LoadBalancer Services. They are bound to CiliumBGPPeerConfig via labels.

apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
 name: services-and-pods
 labels:
 advertise: bgp  # ← the label the PeerConfig matches
spec:
 advertisements:
 - advertisementType: PodCIDR  # advertises the node's pod CIDR
 attributes:
 communities:
 standard:
 - "65000:100"
 - advertisementType: Service  # advertises ClusterIPs / LoadBalancer IPs
 service:
 addresses:
 - LoadBalancerIP
 - ClusterIP
 - ExternalIP
 selector:
 matchLabels:
 bgp-advertise: "true" # only Services with this label
 attributes:
 communities:
 standard:
 - "65000:200"
 localPreference: 200

The granularity is very fine: you can advertise different types of prefixes with different BGP communities, different local-preference, different path attributes, and filter Services with label selectors. This was literally impossible with CiliumBGPPeeringPolicy v1.

4. `CiliumBGPNodeConfig` (auto-generated)

This CRD is not configured by hand. The Cilium operator generates one per node from the CiliumBGPClusterConfig that applies to that node. It is the materialised per-node state that each node’s agent reads to bring up its peerings. If you want to see what BGP configuration is actually running on a node, kubectl get ciliumbgpnodeconfig <nodename> -o yaml shows you.

5. `CiliumBGPNodeConfigOverride`

Optional. It allows overriding the generated configuration for a specific node when you need something non-standard. Use cases:

Pinning the BGP router-id to a specific IP (useful when the node has several interfaces).
Specifying the peer’s local address when there are several outgoing interfaces.
Changing timers only for one problematic node.

apiVersion: cilium.io/v2
kind: CiliumBGPNodeConfigOverride
metadata:
 name: node-rack1-master01  # the name must match the node's
spec:
 bgpInstances:
 - name: instance-65000
 routerID: 10.0.1.10 # router-id override
 peers:
 - name: top-of-rack-1a
 localAddress: 10.0.1.10 # specific local interface

Relationship diagram

CiliumLoadBalancerIPPool: it complements, it is not BGP

Although it is not strictly a BGP CRD, it is worth mentioning: CiliumLoadBalancerIPPool is the CRD that supplies IPs to type=LoadBalancer Services. It defines a range (10.20.0.0/24, for example) that Cilium assigns automatically to LoadBalancer Services. Combined with a CiliumBGPAdvertisement that advertises LoadBalancerIP, it gives the complete cycle: new Service → IP assigned from the pool → BGP advertisement to the routers → IP routable from the corporate network, with no external balancer.

apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
 name: lb-pool-rack1
spec:
 blocks:
 - start: "10.20.0.10"
 stop: "10.20.0.250"
 serviceSelector:
 matchLabels:
 lb-pool: rack1

Full manifest: pod CIDRs + LoadBalancer Services advertised to a redundant ToR pair

A realistic example of a cluster with two top-of-rack switches as BGP peers, both in the same AS (64512), Cilium in AS 65000:

# 1. CiliumBGPPeerConfig — shared config for both ToRs
---
apiVersion: cilium.io/v2
kind: CiliumBGPPeerConfig
metadata:
 name: tor-peers
spec:
 timers:
 holdTimeSeconds: 30
 keepAliveTimeSeconds: 10
 gracefulRestart:
 enabled: true
 restartTimeSeconds: 120
 families:
 - afi: ipv4
 safi: unicast
 advertisements:
 matchLabels:
 advertise: bgp

# 2. CiliumBGPAdvertisement — what gets advertised
---
apiVersion: cilium.io/v2
kind: CiliumBGPAdvertisement
metadata:
 name: pods-and-lb
 labels:
 advertise: bgp
spec:
 advertisements:
 - advertisementType: PodCIDR
 - advertisementType: Service
 service:
 addresses: [LoadBalancerIP]
 selector:
 matchExpressions:
 - { key: io.kubernetes.service.namespace, operator: NotIn, values: [kube-system] }

# 3. CiliumBGPClusterConfig — which nodes talk to which peers
---
apiVersion: cilium.io/v2
kind: CiliumBGPClusterConfig
metadata:
 name: cluster-bgp
spec:
 nodeSelector:
 matchLabels:
 bgp: enabled
 bgpInstances:
 - name: instance-65000
 localASN: 65000
 peers:
 - name: tor-a
 peerASN: 64512
 peerAddress: 10.0.1.1
 peerConfigRef: { name: tor-peers }
 - name: tor-b
 peerASN: 64512
 peerAddress: 10.0.1.2
 peerConfigRef: { name: tor-peers }

# 4. CiliumLoadBalancerIPPool — range of LB IPs
---
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
 name: lb-corporate
spec:
 blocks:
 - cidr: "10.20.0.0/24"

Four objects. Before, in v1, it was a single CiliumBGPPeeringPolicy that mixed everything together and turned out to be hard to maintain in clusters of more than 5 nodes with heterogeneous configuration. The new separation is longer but clearly factorable: one PeerConfig per peer type, one Advertisement per advertisement policy, one ClusterConfig connecting nodes with peers.

Common operational traps

`routingMode: tunnel` mode with BGP

BGP only makes sense with direct routing (routingMode: native). If you have tunnel mode (VXLAN/Geneve) and configure BGP, you will advertise pod CIDRs but the packets will still go out encapsulated, producing confusing behaviour (sometimes via tunnel, sometimes direct depending on routes). Configure routingMode: native and disable the tunnel.

eBPF host routing vs `kube-proxy replacement`

They are two different things. kubeProxyReplacement: true enables the replacement of kube-proxy (the Services). bpf.hostRouting: true enables the host’s iptables bypass (the node’s routing decisions are made with eBPF instead of the traditional FIB). The second needs kernel 5.10+ with all the bpf features enabled; if you do not have that kernel, it falls back to legacy mode and the performance is only “almost as good”.

Aggressive BGP timers over flapping NICs

With holdTimeSeconds: 9 / keepAliveSeconds: 3, a NIC that blinks for 5 seconds breaks the BGP session and all the advertised routes disappear from the fabric. That node’s pods become unreachable until the session is re-established. For clusters on hardware with suspect NICs, use the conservative values (holdTime: 30, keepAlive: 10) and consider graceful restart explicitly (it is already in the example above).

Advertising ClusterIP to the corporate network

Advertising ClusterIP to external routers is rarely what you want: these are internal Service IPs, not designed to be reached from outside the cluster. For external exposure, use LoadBalancerIP from a CiliumLoadBalancerIPPool. Advertising ClusterIP only makes sense in very specific topologies (multi-cluster mesh with shared service discovery).

Mixing v2alpha1 (`CiliumBGPPeeringPolicy`) and v2 (`CiliumBGPClusterConfig`)

It does not work well. The operator emits warnings in the logs about the use of the deprecated API, and conflicts between what the peering policy defines and what the cluster config defines can produce strange states. Migrate from one to the other in a single pass; do not run both.

MD5 password and MTU

If you configure an MD5 password in CiliumBGPPeerConfig.authentication, the TCP header is larger. On links with a tight MTU (1500 - 50 for the upstream fabric’s VXLAN, for example), the BGP handshake can fragment and die silently. Either use MTU 9000 between nodes and ToR, or make sure the MSS values are negotiated correctly.

What we have not covered

Cilium Cluster Mesh: federation of several Cilium clusters so their Services can see each other. It fits with BGP when you want native routing between clusters; it has its own CRDs.
L7 Policies and the embedded Envoy: HTTP/gRPC/Kafka policy. Another layer of eBPF + proxy that deserves its own post.
Hubble: eBPF-based traffic observability that Cilium exposes. Flow log dashboards with zero impact on latency.
Transparent WireGuard: pod-to-pod encryption without sidecars, controlled by Cilium via eBPF redirect to a kernel WireGuard dataplane.
Gateway API in Cilium: the successor to Ingress, with first-class support since Cilium 1.16+.
eBPF for LLM serving: the natural connection with the previous inference series. There is recent work using eBPF for multi-tenant fairness on GPUs and for token tracking; paper territory, not production yet.

In other words, there is material left for another three posts in today’s series. Let us take them in order.

References

Conceptual and project:

eBPF.io — canonical documentation of the eBPF ecosystem.
The BPF Compiler Collection (bcc) and bpftrace — tools to get started with.
eBPF en 2026: How Extended Berkeley Packet Filter Became the Engine of Linux Observability and Networking — state of the art.

XDP, TC and firewalling:

Getting started with XDP and eBPF (Red Hat docs).
Full Guide to BPF Firewalls: XDP, tc, and eBPF Integration (Medium, 2025).
Cloudflare blog: XDP for DDoS mitigation.
Facebook Katran (GitHub) — L4 LB with XDP, code and paper.

Cilium:

Cilium documentation — always the first port of call.
Kubernetes Without kube-proxy — the official guide to the replacement.
Cilium BGP Control Plane Resources (docs) — reference for the v2 CRDs.
Configuring Cilium BGP Control Plane (OneUptime blog, mar 2026) — walkthrough.
A Guide to BGP Control Plane and Cluster Mesh in Cilium Networking (Sigrid Jin, Medium) — a deeper post with use cases.

Cross-references:

Previous post on this blog: Kubernetes with Cilium BGP: services reachable without Ingress — the first step, with version v1 (which now needs migrating).
Series on LLM inference: KV cache, vLLM on Kubernetes, PagedAttention deep dive, LLM K8s Operators — where the fast network (which we will see in the following posts of this series) determines real performance.

Kubernetes with Cilium BGP: services reachable without Ingress

Sun, 08 Mar 2026 00:00:00 +0000

One of the advantages of using Cilium as the CNI in Kubernetes is its native BGP support. It lets you advertise ClusterIPs and LoadBalancer IPs directly to the LAN router, making services reachable without needing Ingress or NodePort.

The problem

In a standard Kubernetes cluster, pods and services live on internal networks that are not reachable from outside the cluster. Getting to them requires NodePort, Ingress or an external LoadBalancer.

With Cilium BGP, the pod CIDRs and service CIDRs are advertised over BGP to the upstream router, making the whole cluster network routable from the LAN.

Work in progress — full configurations for RKE2 + CRS327 coming soon.

Kubernetes on lo0 — Tech Blog

Hubble: eBPF network observability, state of the art 2026 and the new frontier with AI agents

TL;DR

The analogy: tcpdump that speaks Kubernetes

Architecture: four pieces visible from outside

1. Hubble Server (embedded in every Cilium agent)

2. Hubble Relay (aggregator)

3. Hubble CLI (hubble)

4. Hubble UI

What you see: the Hubble flow log from the inside

Verdict and drop attribution

Prometheus metrics and Grafana dashboards

Deployment: Helm on one screen

State of the art in 2026

Direct attribution of drops to a NetworkPolicy

Tracing with IP options

Filtering by encryption status

Hubble field mask API stabilised

AI-driven anomaly detection (predictive security)

Scaling to 10,000+ pods

Cilium 1.20 in development

The new frontier: eBPF and AI agents

AgentSight: zero-instrumentation for LLM agents

How it fits with Hubble and Tetragon

Emerging use cases

Common Hubble use cases

1. NetworkPolicy debugging

2. Auditing inter-namespace communication

3. Exfiltration detection

4. Real-time service SLOs

5. Performance debugging

6. Post-incident forensics

Hubble and the rest of the observability stack

Comparison with the alternatives

Operational traps

Cardinality in Prometheus

L7 visibility costs CPU

Hubble Relay without HA

Encryption status reporting depends on the Cilium config

UI exposed without auth

Storage that does not scale

What we have not covered

References

Tetragon: Cilium's security cousin that sees every syscall in the kernel

TL;DR

The analogy: auditd on steroids in eBPF

What Tetragon is, architecturally

The two CRDs: TracingPolicy and TracingPolicyNamespaced

Anatomy of a TracingPolicy

Supported hook points

Arguments

Selectors: filtering in the kernel

Actions: from a simple Post to Sigkill

Modes: detection vs enforcement

Full example: blocking writes to /etc/passwd in the prod namespace

Common use cases

1. Execution auditing (execve)

2. Sensitive file access

3. Unauthorised outbound network connections

4. Container escape detection

5. Cryptomining

6. Detecting rootkits and suspicious kernel modules

Bonus: detecting third-party modification of eBPF maps

Comparison with Falco

Hubble + Tetragon: unified observability

Deployment and operation

Helm

The tetra CLI

Exporting to a SIEM

Performance

Common operational traps

Permanent monitoring

Selectors that are too lax

A kernel that is too old

Hooks on renamed kernel functions

Sigkill in critical namespaces

Missing rateLimit

What we have not covered (upcoming articles)

References

eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack

3. Hubble CLI (`hubble`)

Full example: blocking writes to `/etc/passwd` in the prod namespace

1. Execution auditing (`execve`)

The `tetra` CLI

Permanent `monitoring`

`Sigkill` in critical namespaces

Missing `rateLimit`

1. `CiliumBGPClusterConfig`

2. `CiliumBGPPeerConfig`

3. `CiliumBGPAdvertisement`

4. `CiliumBGPNodeConfig` (auto-generated)

5. `CiliumBGPNodeConfigOverride`

`routingMode: tunnel` mode with BGP

eBPF host routing vs `kube-proxy replacement`

Mixing v2alpha1 (`CiliumBGPPeeringPolicy`) and v2 (`CiliumBGPClusterConfig`)