Tetragon on lo0 — Tech Blog

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

Tue, 09 Jun 2026 17:00:00 +0200

Operational companion to The contractor with the master key. That post explains the why and the where: the threat model, the five isolation families, which domain uses each. This one is the how, with commands. If you have not read it, read it first: here I take for granted what the blast radius is, why bwrap runs without root and what Tetragon watches. The procedure comes in two independent tracks, client and cluster, because, as the sibling post argues, the control extrapolates but the primitive gets rewritten.

TL;DR

Two reproducible procedures. Client (workstation): install ai-jail (it wraps bubblewrap), generate the per-project .ai-jail, audit with --dry-run, fix the allowlists with --bootstrap, use --lockdown for anything you do not trust, and leave the agent without git push permission. Cluster (RKE2 with Cilium + Tetragon): put the pod baseline in place (unprivileged securityContext, seccomp: RuntimeDefault, default-deny NetworkPolicy), put the untrusted agent’s pod in a microVM with runtimeClassName: kata, and deploy the Tetragon TracingPolicy files in two phases: observe with action: Post to raise the baseline, then promote to action: Sigkill on tcp_connect (egress) and security_file_open (secret paths). The golden rule of the Tetragon phase: adopt first, block later; never put a Sigkill into production without having seen the events in observation mode first.

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

ai-jail wraps the sandbox; on Linux it needs bubblewrap separately, on macOS it needs no extra dependency.

# ai-jail (macOS and Linux)
brew tap akitaonrails/tap && brew install ai-jail
# or, with cargo:
cargo install ai-jail
# or, with mise:
mise use -g ubi:akitaonrails/ai-jail

# bubblewrap on Linux (pick your distro)
sudo pacman -S bubblewrap # Arch
sudo apt install bubblewrap # Debian / Ubuntu
sudo dnf install bubblewrap # Fedora

Check the binary is there and that bwrap runs without root (it must not ask for sudo):

ai-jail --version
bwrap --ro-bind / / --unshare-all echo "bwrap ok without root"

If bwrap fails asking for privileges, your kernel has unprivileged user namespaces disabled; enable them (sysctl kernel.unprivileged_userns_clone=1 on older Debian/Ubuntu) before continuing.

A1 — The per-project .ai-jail file

On the first launch inside the project, ai-jail creates a .ai-jail (TOML) that is committable to the repo: any colleague who clones inherits the same policy.

cd ~/Projects/mi-app
ai-jail claude # creates .ai-jail and launches Claude Code inside the sandbox

The generated file:

# .ai-jail — sandbox configuration (commit it to the repo)
command = ["claude"]
rw_maps = [] # extra writable directories
ro_maps = [] # extra read-only directories

Before trusting the sandbox, audit it. --dry-run --verbose prints every mount point, every isolation flag and the full bwrap command, without executing anything:

ai-jail --dry-run --verbose claude

Read the output and confirm three things: that $HOME is mounted as a tmpfs (not the real one), that ~/.ssh, ~/.aws and ~/.gnupg do not appear among the mounts, and that the only writable directory is the project one. If you need an extra directory:

ai-jail --rw-map ~/Projects/shared-lib claude # extra, writable
ai-jail --map /opt/datasets claude # extra, read-only

Other agents, same binary:

ai-jail codex
ai-jail opencode
ai-jail bash # bare shell to debug the sandbox
ai-jail -- python script.py # any command

A2 — Permission allowlists with –bootstrap

--bootstrap generates each agent’s permission configuration, with sensible allow/deny/ask, and takes a backup before overwriting:

ai-jail --bootstrap

What it produces, in summary:

Agent	File	Base policy
Claude Code	`~/.claude/settings.json`	allow: `git status/diff/log`, `ls`, `grep`, `cargo`, `npm`, `python`, `docker compose` · ask: `git push`, `rm`, `docker run` · deny: `rm -rf`, `sudo`, `chmod 777`, `git push --force`
Codex	`~/.codex/config.toml`	`approval_policy = "on-request"`
OpenCode	`~/.config/opencode/opencode.json`	permissions for `bash`, `edit`, `write`

The operational key: git push is under ask, not allow, and git push --force under deny. The agent can commit, branch and rebase locally as much as it wants; none of that touches the remote. (If you use the Claude Code /sandbox, also set "allowUnsandboxedCommands": false to close the dangerouslyDisableSandbox escape hatch, which is opt-out out of the box.)

A3 — Lockdown for anything you do not trust

To audit third-party code or run an agent over a project you do not know, --lockdown goes further: project mounted read-only, GPU/Docker/display disabled, --rw-map/--map ignored, $HOME a pure tmpfs with no host dotfiles, network cut with --unshare-net and the environment wiped with --clearenv.

ai-jail --lockdown bash

It is the most restrictive sandbox possible short of a VM. Use it as your mental default for anything that is not your own code on your own machine.

A4 — The safety net: git with no push

It is not a flag, it is a property of the environment that changes the risk arithmetic. If the project is in git with a remote, and the agent does not have push permission, the worst case, that it corrupts every file in the project, is reverted with:

git checkout . # back to the last commit
# and if it touched .git (unlikely): delete the dir and re-clone

The remote was never touched. Sandbox for the filesystem + git for the code + manual push is already a reasonable level for daily use: ai-jail protects your data and the system, git protects the code, and the decision to publish is still yours.

Track B — Cluster (RKE2 with Cilium + Tetragon)

The untrusted agent, or the inference that executes generated code, runs as a pod. The same principle as on the client, different primitives. We assume a generic RKE2 cluster with Cilium as CNI and Tetragon already deployed (the eBPF agent DaemonSet on every node).

B0 — The pod baseline

Before any eBPF, the standard kit. Unprivileged securityContext, read-only root, default seccomp:

apiVersion: v1
kind: Pod
metadata:
 name: ai-agent
 namespace: agentes
 labels:
 app: ai-agent
spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 10001
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: agent
 image: registry.interno/ai-agent:pinned
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop: ["ALL"]
 volumeMounts:
 - { name: work, mountPath: /work }  # the only writable one
 volumes:
 - name: work
 emptyDir: {}

And the default egress cut, the cluster twin of --unshare-net. A default-deny egress NetworkPolicy in the namespace, opening only DNS and the bare essentials:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-egress
 namespace: agentes
spec:
 podSelector: {}
 policyTypes: ["Egress"]
 egress:
 - to:
 - namespaceSelector:
 matchLabels: { kubernetes.io/metadata.name: kube-system }
 ports:
 - { protocol: UDP, port: 53 }
 - { protocol: TCP, port: 53 }

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

For genuinely untrusted code, take it out of the shared kernel. With Kata deployed there is a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
 name: kata
handler: kata

And the pod asks for it with one line, runtimeClassName: kata, running in its own microVM with a dedicated kernel instead of sharing the node’s:

spec:
 runtimeClassName: kata  # ← the pod runs in a microVM, not on the node's kernel
 # ...the rest the same as B0

It is the cluster twin of isolation by construction: a kernel exploit inside the pod does not reach the node.

B2 — Tetragon, observation phase (Post)

Now the layer that separates a platform with runtime visibility from one without. Observe first, never kill from the outset. A TracingPolicyNamespaced, scoped to the namespace and to the agent’s label, that reports (does not kill) three things: process executions, network connections and opens of sensitive paths. action: Post only emits the event.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-observa
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- outbound connections ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchActions:
 - action: Post
 # --- opens of sensitive files ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets"
 - "/work/.git/config"
 matchActions:
 - action: Post

(Process executions need no kprobe: Tetragon emits process_exec/process_exit natively.) Deploy it and watch the events live from the node’s Tetragon pod:

kubectl apply -f agente-observa.yaml
# readable events, filtered by namespace:
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
 tetra getevents -o compact --namespace agentes

Leave this running for a typical working day of the agent. Note which destinations it really connects to (your internal registry, your HF mirror, your vLLM endpoint) and which paths it opens. That is your baseline: the list of what is legitimate. Without this step, a Sigkill kills good work and generates an availability incident, precisely what ENS asks you to avoid.

B3 — Tetragon, enforcement phase (Sigkill)

With the baseline in hand, promote to blocking. Two rules. The first: kill any connection whose destination is not on the allowlist — NotDAddr inverts the match: it fires for everything that is not those networks. The second: kill any attempt to open a secrets path.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-enforce
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- egress: kill everything that is NOT on the allowlist ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr"
 values:
 - "127.0.0.1"
 - "10.0.0.0/8" # internal cluster network
 - "172.16.10.20" # internal registry (example)
 matchActions:
 - action: Sigkill
 # --- reading secrets: kill the process ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets/kubernetes.io/serviceaccount/token"
 - "/work/.ssh"
 matchActions:
 - action: Sigkill

kubectl apply -f agente-enforce.yaml

Now the agent can do whatever it likes inside the pod, but the instant it tries to connect to a disallowed destination or read the service account token, Tetragon kills it in the kernel, before the packet leaves or the read returns bytes. It is the cluster twin of the curl blocklist and the unmounted ~/.ssh, but applied at runtime and over any binary, not just the ones you know about.

Operational warning. Enforcement with Sigkill requires a recent kernel with eBPF support for the action (5.10+ is safe). Deploy agente-enforce in a test namespace first, and keep agente-observa active in parallel: if the block fires, the Post event tells you exactly what triggered it. Adopt first, block later.

The client ↔ cluster equivalence table

The same vector, the two primitives. This is “extrapolating the technology” made explicit:

Threat vector	Client (workstation)	Cluster (RKE2)
`$HOME` / writable root	`$HOME` as ephemeral tmpfs (`bwrap`)	`readOnlyRootFilesystem: true` + `emptyDir`
Arbitrary egress	`curl`/`wget` blocklist · `--unshare-net`	default-deny NetworkPolicy + Tetragon `NotDAddr`→`Sigkill`
Reading secrets	`~/.ssh`/`~/.aws`/`~/.gnupg` not mounted	secrets outside the pod + Tetragon `security_file_open`→`Sigkill`
Kernel escape	Landlock (2nd VFS barrier)	`runtimeClassName: kata` (microVM, own kernel)
No escape hatch	process inside `bwrap`, no way out	no `privileged`, `drop ALL`, `allowPrivilegeEscalation:false`
Damage to the code	git remote with no `push` → `git checkout .`	GitOps + PR review, the agent does not apply to `main`
Visibility	`--dry-run --verbose` (static, pre-run)	Tetragon `tetra getevents` (dynamic, at runtime)

Checklist of gotchas

Do not put in a Sigkill without passing through Post. The observation baseline is not optional: it is what separates “blocking a C2” from “killing your own fine-tuning job”.
The .ai-jail gets committed; secrets do not. The TOML is policy, not credentials. Check you are not putting paths holding sensitive data into rw_maps.
readOnlyRootFilesystem breaks apps that write to /tmp. Mount an emptyDir at /tmp as well as the work one.
A NetworkPolicy with no DNS rule leaves the pod blind. Open port 53 to kube-system or nothing resolves.
Kata is not free. It adds startup latency and not every workload with special devices (GPU passthrough) fits; reserve it for the untrusted, not for everything.
The Claude Code /sandbox does not cover MCP or hooks unless you enable sandbox-runtime. If your agent uses MCP servers, assume they run with full permissions until you do.
NotDAddr with literal IPs ages badly. Document the allowlist and review it when the registry or the inference endpoint changes; consider stable internal CIDRs instead of loose IPs.

References

ai-jail (Fabio Akita), GPL-3.0: https://github.com/akitaonrails/ai-jail
bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io
Tetragon — TracingPolicy: https://tetragon.io/docs/concepts/tracing-policy/
Tetragon — enforcement (Sigkill/Override): https://tetragon.io/docs/concepts/enforcement/
Kata Containers — Kubernetes RuntimeClass: https://katacontainers.io
Kubernetes — Pod Security & seccomp: https://kubernetes.io/docs/tutorials/security/seccomp/
Kubernetes — Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Cilium: https://cilium.io

The contractor with the master key: isolating AI agents from workstation to cluster

Tue, 09 Jun 2026 16:00:00 +0200

First post of a pair on AI agent isolation. This one fixes the what and the where: the complete map of isolation primitives and which domain each belongs to. The sibling runbook fixes the how, with commands: ai-jail and bubblewrap on the client, Tetragon TracingPolicy and RuntimeClass on the cluster. If you are only going to read one, this one gives you the mental model; the other one gives you the files to copy and paste.

TL;DR

An AI agent that runs code needs access to your filesystem and your tools: compiler, linter, grep, make, cargo, npm. That is the minimum to be useful. The problem is that alongside that access travels the ability to read ~/.aws/credentials, exfiltrate your SSH keys or fire an rm -rf outside the project directory. And no malicious model is needed: a compromised dependency in an npm install is enough, because the well-meaning agent and the poisoned post-install script run with the same permissions. The answer is not to trust the LLM’s good intentions; it is to isolate in order to bound the blast radius. This post walks the five isolation families of 2026, from the process sandbox to the full VM, and sorts them into two columns: what applies on the client (the developer’s workstation: bubblewrap, ai-jail, sandbox-exec, Landlock, the native Claude Code and Codex sandboxes) and what applies on the cluster (where the agent or the inference runs on Kubernetes: namespaces+seccomp, gVisor, Firecracker/Kata microVMs and eBPF/Tetragon as the layer for live observation and enforcement). The thesis: the threat model is the same in both places; the tools are not. Policy extrapolates; the primitive gets rewritten.

The analogy: the contractor with the master key

You hire a builder for a renovation. He is competent and acts in good faith. But two things happen that you do not control. The first: he may misread the instruction and knock down the wrong partition wall. The second, worse: his toolbox may have been tampered with before he came through your door — somebody put something inside — and when he opens it in your living room, that something activates.

Nobody sensible gives him the master key to the whole building. You open the room he works in, you leave him the tools he needs, and you keep the study with the safe and the server room locked. If the renovation goes wrong, by error or by sabotage, the damage stays in that room.

An AI agent is that contractor. The sandbox is the key policy: you give it the project room and the tools, not the master key to the system. And here is the twist that justifies two posts: the builder works in two different buildings. One is your flat, the developer’s workstation, with your credentials, your ~/.ssh, the browser’s password vault. The other is the datacenter, the cluster where inference and autonomous agents serve customers, with data from several tenants at once. The key policy is identical in both: least privilege, bound the radius. But the lock on your flat’s door is not the same as the one at the datacenter. In the flat you fit a cylinder lock (bubblewrap). At the datacenter you put a guard watching every door and a separate wing of the building (Tetragon + microVM). Same principle, different hardware. That is extrapolating the technology, not copying it.

The threat model: what a runaway agent can do

Before choosing a lock it helps to enumerate the burglar. The attack surface of an agent that executes arbitrary bash breaks down into five concrete threats. Not all are defended by the same layer and, an uncomfortable fact that the security documentation itself acknowledges, no single layer covers them all.

Threat	What the agent does	Minimum layer that cuts it
Filesystem out of scope	Reads `.env`, `~/.ssh/id_rsa`, system secrets; modifies sources outside the project	Process sandbox (path allowlist)
Arbitrary network egress	Exfiltrates data, receives instructions from a remote C2, calls unauthorised APIs	Network blocking / NetworkPolicy / microVM
Kernel syscall surface	A kernel exploit from inside the container escalates to the host (shared kernel)	gVisor or microVM (dedicated kernel)
Cross-tenant leakage	One customer’s workload reads another’s data on a multi-tenant platform	microVM (de facto standard)
Secret exfiltration	Extracts tokens and environment variables via `/proc` or the environment	`--clearenv` / tmpfs `$HOME` / secrets outside the pod

There is a sixth threat that no sandbox solves: prompt injection. If an attacker manages to slip instructions into the agent’s context — a poisoned comment in the code, a malicious file the agent reads, an adversarial response from a tool — the agent will execute those instructions with whatever permissions the sandbox grants it. Isolation shrinks the impact radius of a successful injection; it does not prevent the injection. That is why the sandbox is a layer, not the solution: on top go input validation, tool-call allowlists and output auditing. The sentence to internalise: isolation does not make the agent trustworthy; it bounds what an untrustworthy agent can break.

Two domains, one policy

The builder works in two buildings. The distribution of tools reads better in two columns:

The client: isolating the agent on the workstation

Here the agent is a coding assistant — Claude Code, Codex, OpenCode, Cursor — that a developer launches on their machine. One process, one user, and right next to it the juiciest assets that exist: ~/.aws/credentials, ~/.ssh, ~/.gnupg, the browser’s password store. The matching tier is the lightest one: the process sandbox.

bubblewrap (Linux) and sandbox-exec (macOS). bubblewrap (bwrap) is the same sandbox Flatpak uses to isolate each desktop app: a ~50 KB binary, ~4,000 lines of C, maintained by the GNOME team and, the key property, it runs without root via CLONE_NEWUSER, creating namespaces with no elevated privileges. It mounts $HOME as an ephemeral tmpfs and only exposes the project directory, writable; the rest of the system becomes invisible. On macOS the equivalent is sandbox-exec with SBPL profiles: a legacy Apple API, officially deprecated and with no public replacement, but working today. Parity between the two is not exact — on macOS the GPU (Metal) and the display (Cocoa) are system services and sandbox-exec does not restrict them — but both protect what matters: access to the sensitive areas of the filesystem.

Landlock as a second barrier. bubblewrap isolates via namespaces and mounts; Landlock, a Linux Security Module available since kernel 5.13, restricts access at the VFS level, independently of namespaces. It does not replace bwrap: it complements it. It closes vectors that mount-based isolation does not cover on its own (escape routes via /proc, symlink tricks inside allowed mounts) and acts as a safety net should the namespace machinery have a bug. It is defence in depth inside the client itself, and it degrades cleanly to a no-op on kernels that do not support it.

Dev containers, when reproducibility is needed. A dev container (devcontainer.json, what Codespaces and Cursor use) is a Docker container with a configuration layer on top. It gives reasonable filesystem isolation and easy reset (destroy and recreate), but it shares the host kernel, the same limitation as any Docker, and it tends to be long-lived, accumulating state. For an agent running your own team’s code on your own machine it is a good repeatability story; it is not, on its own, the isolation layer for untrusted code.

What wraps all of this. A hand-written bash script works, but it does not scale to a team. The tools that package it:

ai-jail (Rust, GPL-3.0): wraps bwrap/sandbox-exec with per-project config in a .ai-jail file (TOML, committable to the repo, so the whole team inherits the same policy), auto-detection of GPU/Docker/display, a --lockdown mode (project read-only, network cut with --unshare-net, --clearenv), --dry-run for auditing, and --bootstrap to generate each agent’s permission allowlists. It is tool-agnostic: the same binary serves Claude, Codex, OpenCode or Crush. It also applies Landlock automatically on 5.13+ kernels as defence in depth.
The Claude Code /sandbox: since October 2025 Claude Code ships its own sandbox that uses, exactly, bubblewrap on Linux and sandbox-exec on macOS. Its Sandboxed Bash isolates shell commands, but not the file tools, the MCP servers or the hooks, which run with the process’s full permissions unless you enable the beta sandbox-runtime package, which wraps the whole process. There is one nuance worth knowing: if a command fails because of a restriction, the agent can retry with dangerouslyDisableSandbox — it is opt-out, not opt-in.
Codex CLI: three modes via --sandbox (read-only, workspace-write, danger-full-access); the recommended default is workspace-write. The philosophy is deliberate: Codex does not provide the isolation, it delegates it to the environment that wraps it. danger-full-access only makes sense inside a microVM.
Cursor: its cloud agents run in isolated VMs; /worktree creates a single-use isolated worktree per task, and /best-of-n launches several attempts in parallel in separate worktrees.

The cluster: isolating the agent in production

The second building is the datacenter. Here the “agent” can be an autonomous agent running with no human in front of it, or the inference service itself executing generated code, or a multi-tenant workload where one customer’s pod must not touch another’s data. The process is no longer one: they are pods in a Kubernetes cluster (RKE2/RKE3 on a typical sovereign platform). The primitives change in nature.

The pod baseline. First of all, the standard kit: Linux namespaces, seccomp (RuntimeDefault) to trim the syscall surface, cgroups for resource limits, an unprivileged securityContext (runAsNonRoot, readOnlyRootFilesystem, dropping all capabilities) and NetworkPolicy to cut egress. It is the cluster equivalent of the process sandbox allowlist. Necessary, but it shares the kernel with the host: insufficient for genuinely untrusted code.

gVisor (runsc). Google’s user-space kernel: it intercepts the workload’s syscalls before they reach the host kernel and serves them inside a Linux kernel reimplemented in Go (the Sentry). The surface exposed to host kernel vulnerabilities drops sharply, while keeping fast startup and a low footprint. It is the middle ground when the risk of a kernel escape is real but the overhead of a microVM is not acceptable.

Firecracker / Kata microVMs. The de facto standard for untrusted code in 2026. Firecracker (AWS’s VMM written in Rust, on top of KVM) gives each sandbox a dedicated Linux kernel: a kernel exploit inside the microVM does not reach the host by construction. It is what sits underneath Vercel Sandbox (GA January 2026) and E2B. On Kubernetes, Kata Containers brings that model to a RuntimeClass: you mark the untrusted agent’s pod with runtimeClassName: kata and it runs in its own microVM instead of sharing the node’s kernel. For multi-tenant with generated code, this is the baseline, not the luxury.

eBPF / Tetragon: the layer we already have. Here is the piece that separates a platform with runtime observability from one that only trusts its configuration. The previous layers are static: they define what the pod can do before it starts. Tetragon, Cilium’s runtime security component, based on eBPF, is dynamic: it observes, in the kernel and at minimal cost, every process execution, every network connection and every file open in every pod, and it can act inline. It does not replace the sandbox; it watches it from inside the kernel. Where bubblewrap on the client blocks curl with a command blocklist, Tetragon on the cluster hooks tcp_connect in the kernel and, if the destination is not allowed, kills the process with Sigkill before the packet leaves. Where the client hides ~/.ssh behind a tmpfs, Tetragon hooks security_file_open and reports, or kills, any attempt to read a sensitive mounted path. It is the guard patrolling the corridors while the microVMs are the walls. And it is exactly the kind of control that materialises the ENS monitoring and traceability measures (op.mon, op.exp) without instrumenting the application: visibility lives in the kernel, not in the agent’s code.

The landscape table

The five families, their relative isolation strength, their startup cost and the domain where they live:

Tier	Primitive	Isolation	Startup	Natural domain
Process sandbox	Seatbelt · bubblewrap	Baseline	~0 ms	Client (Claude Code default)
Dev container	Docker + seccomp	Moderate	seconds	Client / cluster (repeatability)
User-space kernel	gVisor (`runsc`)	Strong	ms	Cluster (medium multi-tenant)
microVM	Firecracker · Kata	Strongest (practical)	<1 s	Cluster (untrusted code)
Full VM	KVM · EC2	Maximum	30 s+	Cluster (external boundary, compliance)
Runtime enforcement	eBPF · Tetragon	Cross-cutting	always on	Cluster (watches+kills on top of any tier)

Tetragon gets its own row on purpose: it is not a rung on the ladder, it is a cross-cutting layer that operates on top of any of the others. It stacks with all of them.

A numerical note on why the “startup” column decides as much as the “isolation” column. A full VM wins on raw isolation but takes tens of seconds to provision; for an agent that needs a fresh environment per request or per session, that cost is prohibitive. A Firecracker microVM starts in under 1 second and a process sandbox in ~0 ms. That is why the dominant pattern in 2026 is not “the most isolated VM”, but a full VM as the external boundary plus a microVM as the per-request execution unit inside, the architecture of Vercel, AWS Lambda and E2B. On the client the arithmetic is the opposite: the developer launches the agent dozens of times a day interactively, and a startup measured in seconds would break the flow; hence the process sandbox, with its microsecond overhead, is the correct default.

Extrapolate, do not copy

The thesis of this pair of posts fits in one sentence: the threat model is invariant across domains; the primitive that implements it is not. The client and the cluster defend exactly the same five vectors — filesystem, network, kernel, multi-tenant, secrets — but with toolboxes that do not overlap. Every control has its twin on the other side:

$HOME as an ephemeral tmpfs (client) ↔ readOnlyRootFilesystem + emptyDir (cluster).
A curl/wget blocklist in bwrap (client) ↔ a TracingPolicy on tcp_connect in Tetragon plus NetworkPolicy (cluster).
--unshare-net in lockdown (client) ↔ default-deny NetworkPolicy (cluster).
No escape hatch, the process lives inside bwrap (client) ↔ no privileged, no hostPath, RuntimeClass kata (cluster).
~/.ssh and ~/.aws never mounted (client) ↔ secrets outside the pod plus Tetragon watching security_file_open (cluster).

The sibling runbook turns each of these equivalences into concrete files. What matters here is the method: when someone shows you an agent sandbox, be it the Claude Code /sandbox on a laptop or a microVM on a PaaS, the useful question is not “which tool does it use?”, but “which of the five vectors does it close, and which does it leave open?”. The tool gets replaced; the threat map stays.

What no sandbox solves

Three limits that Anthropic’s own documentation states, and which are worth keeping in view so as not to sell smoke:

Egress remains a risk in any sandbox that allows outbound connections. If the agent can open a connection, it can exfiltrate. That is why the client lockdown cuts the network and the cluster uses default-deny NetworkPolicy plus Tetragon: you do not trust “filtering well”, you trust “not letting anything out”.
Code modification remains possible in any sandbox with the project directory mounted writable. The remedy is not a sandbox-technical one, it is git: with the remote untouched and no push permission, the worst case is corrupting the local working copy — git checkout . and start again. The damage never reaches the remote.
No sandbox prevents a compromised prompt from reaching the API. Isolation bounds the impact of an injection; it does not prevent it. The complementary defences — input validation, tool-call allowlists, output auditing — are mandatory alongside isolation, not instead of it.

The operational conclusion: isolation shrinks the blast radius; defence in depth is what closes the circle. A process sandbox for trusted code on a known machine is appropriate and practically free. For an agent acting on user prompts, executing generated code or running multi-tenant, the acceptable minimum in 2026 is a microVM, with Tetragon watching from above. Pick the tier that matches your real threat, check which vector it leaves open, and stack complementary controls on top.

References

bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io · https://docs.rs/landlock
ai-jail (Fabio Akita): https://github.com/akitaonrails/ai-jail
gVisor: https://gvisor.dev
Firecracker: https://firecracker-microvm.github.io
Kata Containers: https://katacontainers.io
Tetragon (Cilium): https://tetragon.io
Vercel Sandbox — concepts: https://vercel.com/docs/vercel-sandbox/concepts
E2B: https://github.com/e2b-dev/E2B
Claude Code sandboxing: https://docs.claude.com/en/docs/claude-code
Codex CLI: https://github.com/openai/codex

Tetragon: Cilium's security cousin that sees every syscall in the kernel

Tue, 19 May 2026 05:00:00 +0200

TL;DR

Tetragon is the runtime security and observability engine that the Cilium project published as a companion to the CNI. Its job is not to route packets, Cilium is already there for that, but to observe what happens inside the node’s processes in real time: which binary runs in each pod, which files it opens, which syscalls it invokes, which capabilities it asks for, which network connections it establishes, which kernel modules get loaded. It does this by loading eBPF programs into the kernel’s hook points (kprobes, tracepoints, uprobes, LSM hooks) and filtering the relevant events inside the kernel itself with a declarative language expressed as a CRD (TracingPolicy and TracingPolicyNamespaced). The result is a stream of events enriched with Kubernetes metadata (pod, namespace, labels) that costs less than 1% of CPU and, the thing that sets Tetragon apart from the competition, can block actions inside the kernel, killing the process with SIGKILL or overwriting a syscall’s return value, before they finish executing, with no race conditions. Against Falco (which parses syscalls in userspace, 5-10% overhead, detection-only), Tetragon is “cheaper and with enforcement”; against the bare kernel, it is “a declarative layer your operations colleague can read”. This article is the extensive introduction you need to take it on seriously: architecture, all the hooks and selectors, the operating modes, a guide to use cases (exec auditing, sensitive file access, container escape, cryptomining, rootkit detection, network observability) and the traps you see in production.

This article is part 2 of the eBPF series. Part 1, eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack, covered basic eBPF, the networking hooks (XDP, TC, sock_ops), how Cilium implements the datapath and the BGP Control Plane v2 CRDs. Here we take those same eBPF hooks and use them for something different: observing and, if needed, stopping what the cluster’s processes do.

The analogy: auditd on steroids in eBPF

Anyone who has spent a few years administering Linux has used auditd. It is the classic kernel subsystem for auditing syscalls: you configure a rule with auditctl (for instance, “monitor any open on /etc/shadow”) and the kernel sends events to a userspace daemon that persists them. It works, but it has two limitations that weigh heavily on modern Kubernetes clusters:

No Kubernetes context. auditd reports processes by PID and UID. Knowing which pod, which namespace, which image, which labels, the information that actually matters when responding to an incident, requires correlating afterwards with data from cri-o or containerd. It is operationally miserable.
No granular enforcement. auditd can generate events, but it cannot take the decision to kill the offending process before the syscall finishes. You leave that to a higher layer that reads the events, processes them and kills the process… if it gets there in time. A race by design.

Tetragon is auditd on steroids: the same conceptual ideas, hooks on syscalls and events to userspace, but implemented with modern eBPF, with filtering inside the kernel so you do not pay the cost of waking the daemon for every irrelevant syscall, with Kubernetes metadata injected by an agent that knows the cluster, and with actions executed inside the kernel itself without waiting for userspace to decide. If the rule says “kill any process that opens /etc/shadow from the prod namespace”, the decision is taken in the kernel kprobe and SIGKILL is delivered before the open completes. There is no race; there is no window between detection and action.

What Tetragon is, architecturally

Tetragon is an agent deployed as a DaemonSet (one pod per node) and a set of CRDs that define the policies to apply. The agent has four responsibilities:

Load eBPF programs into the hook points the active TracingPolicies demand.
Maintain a cache of Kubernetes metadata (pods, namespaces, labels) by reading the API server, so it can enrich every event with the right context.
Collect the events the eBPF programs emit (via ring buffers) and serialise them.
Export the events to configurable destinations: stdout JSON (typical in sidecars or log-collection agents), gRPC streaming (to consume them from Hubble or another consumer), a file, or Fluentd/Loki/SIEM.

The eBPF programs are not written by the user. Tetragon generates the bytecode from the TracingPolicies: it reads the declarative policy, decides which hooks to attack, which arguments to read from the kernel, which filters to apply inline and which actions to execute. The user only writes YAML.

This separation of declarative policy → generated eBPF bytecode is what makes Tetragon usable. Writing eBPF programs by hand is a specialist’s job; writing a TracingPolicy is the job of an SRE with a good example in front of them.

The two CRDs: TracingPolicy and TracingPolicyNamespaced

Tetragon exposes exactly two main CRDs:

TracingPolicy (cluster-scoped, cilium.io/v1alpha1): applies to the whole cluster, every node, every pod. Suitable for platform policies (the whole cluster must be audited the same way): for example, “log every execve in every pod” or “kill any process that tries to load a kernel module”.
TracingPolicyNamespaced (namespaced, same group and version): defined inside a namespace and applied only to the pods of that namespace. Suitable for policies with per-tenant autonomy: for example, “in the prod-payments namespace, kill any outbound connect to an IP outside the corporate range”.

Both CRDs have exactly the same internal structure. The difference is one of scope. The distinction was introduced precisely to allow multi-tenancy: the central security team defines cluster-wide TracingPolicy objects and each tenant can add its own with TracingPolicyNamespaced without needing cluster-admin permissions.

Anatomy of a TracingPolicy

A policy is made up of:

Hook points: which kernel events to observe.
Arguments: which data to read when the hook fires.
Selectors: filters evaluated inside the kernel to discard irrelevant events and, optionally, execute actions when they match.

Supported hook points

The official documentation lists five families of hook points:

kprobes: hook a kernel function. Syscalls are a particular case (when syscall: true) because their ABI differs from that of internal functions. Typical examples: sys_open, sys_openat, sys_connect, tcp_connect, do_mount, commit_creds. It is the most versatile hook and the one used 80% of the time.
tracepoints: hook static tracepoints compiled into the kernel. More stable across kernel versions than kprobes (they do not depend on function names that can change). Examples: syscalls/sys_enter_openat, sched/sched_process_exec.
uprobes: hook functions in userspace libraries or binaries. They serve to observe runtime primitives such as libssl functions, libc, the Go runtime, the JVM.
USDT tracepoints (User Statically Defined Tracepoints): static tracepoints defined in userspace binaries (like those MySQL, PostgreSQL and OpenJDK expose). Useful for application observability.
lsmHooks (LSM, Linux Security Module): hooks of the LSM subsystem, where SELinux/AppArmor plug in. They allow security policies very similar to traditional MAC but programmable with eBPF. Example: file_open, inode_unlink, socket_bind.

Arguments

Every hook can read the arguments of the function it is attached to. The supported types cover the primitives (int, uint64, bool, string, char_buf) and higher abstractions (file, path, sock, linux_binprm, capability, bpf_attr, cred). The high-level types are pointers to kernel structures that Tetragon knows how to parse; instead of having to read an offset, you write type: file and Tetragon gives you the full path of the descriptor’s file.

There is an important capability detail: on kernels ≥ 5.4, Tetragon can read up to 327,360 bytes of an argument if the large buffers flag is enabled. That is the difference between being able to audit execve with all of its long argv complete versus truncating them at 256 bytes and losing context.

Selectors: filtering in the kernel

Selectors are what make Tetragon cheap. Without them, every syscall on the node would fire an event that would travel kernel → ring buffer → agent → processed → filtered → discarded. With selectors, the filtering happens inside the eBPF program itself, in the kernel, and only the events that matter reach userspace.

The available selectors include:

matchArgs: filters by an argument’s value. Operators: Equal, NotEqual, Prefix, Postfix, GreaterThan, LessThan, Mask, SPort (source port), DPort (dest port), Family (AF_INET vs AF_INET6), State (socket state).
matchPIDs: filters by PID; useful for targeted observation.
matchBinaries: filters by the binary executing the syscall (absolute path), with Operator: In, NotIn, Prefix. Essential for avoiding noise from legitimate system processes.
matchNamespaces: filters by Linux namespace (Pid, Mnt, Net, Ipc, Cgroup, User). It allows policies specific to processes in containers versus the host.
matchCapabilities: filters by the process’s effective capabilities. Block actions requiring CAP_SYS_ADMIN that run in pods that should not have them.
matchNamespaceChanges: detects namespace changes (typical of container escape).
matchCapabilityChanges: detects capability changes (privilege escalation).
matchActions: the actions executed when all the preceding matchers hit.

Actions: from a simple Post to Sigkill

When a selector matches, an action is executed. Tetragon defines several:

Post: emits an event to userspace (the observability case). It supports rateLimit to avoid flooding the agent if the condition fires a thousand times per second. The syntax accepts 5 for 5 seconds, 5m for 5 minutes, 1h for 1 hour.
Sigkill: sends SIGKILL to the offending process from inside the kernel, before the syscall completes. This is the only thing that guarantees enforcement without a race.
Override: overwrites the syscall’s return value. Useful for making the process believe the syscall failed (Override -EPERM) without killing it. A better experience for apps that can handle errors; worse for apps that assume success.
Signal: sends any arbitrary signal (not just SIGKILL).
NoPost: does not emit an event, useful when combined with another selector that does emit and you only want the action without duplicated telemetry.
FollowFD and UnfollowFD: mark a file descriptor to follow its lifecycle and enrich subsequent events with the original path. Useful for auditing “which process read this file after opening it”.
TrackSock and UntrackSock: the same for sockets.
GetUrl and DnsLookup: make HTTP requests or DNS resolutions from the kernel. Designed for integrations with external systems (security webhooks, IP reputation lookups).
NotifyEnforcer and CleanupEnforcerNotification: communication with Tetragon’s enforcement subsystem for complex actions.

Modes: detection vs enforcement

A policy can be declared in one of two explicit modes:

enforce: enforcement actions (Sigkill, Override, Signal) are active. This is production.
monitoring: enforcement actions are ignored; only Post events are emitted. This is the “let us see what would happen if this were switched on” mode, critical for testing policies without breaking applications.

Control is done with the spec.options[].name: policy-mode field and value: monitoring or enforce. It is the best practice: start in monitoring, collect events for days, tune the selectors until no false positives come out, and only then switch to enforce.

Full example: blocking writes to `/etc/passwd` in the prod namespace

A realistic policy, commented line by line:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-passwd-write
 namespace: prod
spec:
 kprobes:
 - call: "fd_install"
 syscall: false # kernel function, not a syscall
 args:
 - index: 0
 type: int  # file descriptor
 - index: 1
 type: "file" # struct file*
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values:
 - "/etc/passwd"
 matchActions:
 - action: Sigkill  # kills the process
 rateLimit: "1m" # max once per minute
 options:
 - name: policy-mode
 value: enforce  # enforcement mode active

fd_install runs every time a process obtains a new file descriptor; the second argument is the file’s file struct. Tetragon knows how to resolve it to its absolute path. The matchArgs compares that path with /etc/passwd. If it matches, Sigkill kills the process before the descriptor even becomes usable. rateLimit: 1m stops the agent from saturating if a malicious application tries it in a loop.

Common use cases

Now to real use. These are the six cases that show up in any serious Tetragon deployment in 2026.

1. Execution auditing (`execve`)

The most basic use case and, even so, the most valuable. Which binaries are running in each pod? In a container that is supposed to run only nginx, suddenly seeing an sh or a wget is almost always a red flag.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-execve
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm  # struct linux_binprm*
 selectors:
 - matchActions:
 - action: Post  # events only, no enforcement

With no filters: every execve in the cluster generates an event. With K8s metadata, the event includes pod, namespace, container, image, labels. You turn it into a stream of events towards your SIEM and set up rules: “alert if I see sh, bash, nc, curl, wget or python running in any pod of the prod-api namespace”.

An enforcement variant: instead of Post, use matchBinaries with Operator: NotIn and a whitelist, plus Sigkill if the binary is not on the list. A very rigid box, but effective in pods that are “single-binary” (like a Go microservice).

2. Sensitive file access

Detecting (or blocking) reads and writes on critical files: /etc/shadow, /etc/kubernetes/, Secret mounts, /var/run/docker.sock, /proc/*/cmdline.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: sensitive-file-access
spec:
 kprobes:
 - call: "security_file_open" # LSM-ish via kprobe
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/etc/shadow"
 - "/var/run/secrets/"
 - "/var/run/docker.sock"
 matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/kubelet" # legitimate kubelet access
 matchActions:
 - action: Post

The matchBinaries: NotIn is important: kubelet and other legitimate node agents access these paths constantly and would generate noise. We filter those out in the kernel.

In enforcement: swap Post for Override with argError: -1 (EPERM), so that the open fails but the offending process stays alive and produces the error so that tracing tools pick it up.

3. Unauthorised outbound network connections

Detecting outbound connections to destinations outside the corporate range. Useful for spotting data exfiltration or malware command-and-control.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: block-external-egress
 namespace: prod
spec:
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr" # destination NOT in these CIDRs
 values:
 - "10.0.0.0/8"
 - "192.168.0.0/16"
 - "172.16.0.0/12"
 matchActions:
 - action: Sigkill
 options:
 - name: policy-mode
 value: enforce

This kills any attempt at a TCP connection to an IP that is not in the corporate CIDRs, in the prod namespace. Cilium already does this with NetworkPolicy, but Tetragon has two complementary advantages:

It gives you the process that attempted the connection, not just “pod X tried to connect to Y”.
It also works for exotic protocols where NetworkPolicy is less expressive.

4. Container escape detection

Container escape is the operational nightmare: a process inside a container manages to break the isolation (via a kernel exploit, a badly set capability, a misconfigured hostPath mount) and gain access to the host. Three typical signals:

A namespace change on the process (it leaves the container’s pid namespace).
setns or unshare in non-init processes.
Access to /proc/1/root or /dev/ from a container.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-container-escape
spec:
 kprobes:
 - call: "__x64_sys_setns"
 syscall: true
 args:
 - index: 0
 type: int
 - index: 1
 type: int
 selectors:
 - matchNamespaces:
 - namespace: Pid
 operator: NotIn
 values: ["host_ns"] # only processes NOT in the host pid namespace
 matchActions:
 - action: Sigkill
 - call: "__x64_sys_unshare"
 syscall: true
 args:
 - index: 0
 type: int
 selectors:
 - matchNamespaceChanges:
 - unshare: true
 matchActions:
 - action: Post

__x64_sys_setns targeting the host namespace from a process in a container is practically always malicious (legitimate containers do not need this at runtime).

5. Cryptomining

Mining processes have fairly recognisable profiles:

Processes with names like xmrig, minerd, cgminer, or legitimate processes such as python running CPU-intensive scripts.
Outbound connections to known mining pools (a public list of IPs and domains).
Anomalous use of /dev/cpu_dma_latency to avoid throttling.

A combined policy:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: detect-cryptomining
spec:
 tracepoints:
 - subsystem: "sched"
 event: "sched_process_exec"
 args:
 - index: 4
 type: linux_binprm
 selectors:
 - matchArgs:
 - index: 4
 operator: "Postfix"
 values:
 - "/xmrig"
 - "/minerd"
 - "/cgminer"
 matchActions:
 - action: Sigkill
 kprobes:
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "DPort"
 values: ["3333", "5555", "7777", "14444"] # common pool ports
 matchActions:
 - action: Post  # log only

The double policy: kill binaries with the classic names (belt) and log connections to pool ports (braces), so you also get an alert when someone renames xmrig to nginx-helper or uses exotic ports.

6. Detecting rootkits and suspicious kernel modules

Modern rootkits load kernel modules to patch functions (hide processes, hide network connections, hide files). Detecting them:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: kernel-module-load
spec:
 kprobes:
 - call: "do_init_module"
 syscall: false
 args:
 - index: 0
 type: "string"
 selectors:
 - matchActions:
 - action: Post
 - call: "security_kernel_read_file"
 syscall: false
 args:
 - index: 0
 type: "file"
 - index: 1
 type: int
 selectors:
 - matchArgs:
 - index: 1
 operator: "Equal"
 values: ["READING_MODULE"]
 matchActions:
 - action: Post

In a “well configured” Kubernetes cluster no new kernel modules get loaded at runtime; any event here is highly suspicious. Combine with enforcement on machines where modules should be fixed: Sigkill for whoever tries to load one.

Bonus: detecting third-party modification of eBPF maps

As a 2025-2026 trend: loading malicious eBPF programs to hide presence. Tetragon can observe the bpf syscall:

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
 name: audit-bpf-syscalls
spec:
 kprobes:
 - call: "__x64_sys_bpf"
 syscall: true
 args:
 - index: 0
 type: int  # bpf cmd
 - index: 1
 type: bpf_attr
 selectors:
 - matchBinaries:
 - operator: "NotIn"
 values:
 - "/usr/bin/cilium-agent"
 - "/usr/bin/tetragon"
 - "/usr/bin/bpftool"
 matchActions:
 - action: Post

Any process that is not one of the legitimate agents loading eBPF programs: you want to know about it.

Comparison with Falco

Falco is the closest competitor: it is also runtime security for Kubernetes, also originally based on eBPF (and before that on kernel modules), also with declarative policies. Three years ago they were functionally similar. In 2026 the divergence is clear:

Dimension	Tetragon	Falco
Philosophy	Cilium-native, integrated	Standalone, generic
Filtering	In the kernel (eBPF)	Parsing in userspace
Typical overhead	<1% CPU	5-10% CPU
Enforcement	Yes, in-kernel (Sigkill, Override)	Not native (depends on plugins)
Race conditions	No (action atomic with the syscall)	Yes, in enforcement via plugins
False positive rate	Low (K8s context in the kernel)	Higher (parsing afterwards)
Detection latency	5-26 ms	~10 ms (more constant)
Ecosystem maturity	Young, growing	Mature, plenty of material
Community	Cilium / CNCF Incubating	CNCF Graduated
Integrations	Hubble native	Falcosidekick, many
CRDs per policy	TracingPolicy / Namespaced	No CRDs; rules in YAML

When to choose each one:

Tetragon if you already use Cilium, if you need enforcement in the kernel (not detection-only), if overhead matters to you (workloads with many syscalls), and if you value the Hubble integration. Container escape and cryptomining detection is where its advantage over Falco shows up most.
Falco if you want a tool independent of the CNI, if you need the catalogue of ready-made rules and the wide community, if your cluster is not Cilium, or if the integration with SIEMs and ready-made notifiers (Falcosidekick) saves you work.
Both if the organisation is large: Falco for breadth of detection, Tetragon for surgical enforcement on critical workloads. That is what you see most in companies that have spent years with Falco and add Tetragon for specific cases.

Hubble + Tetragon: unified observability

Hubble is Cilium’s traffic observability component: it shows L3-L7 flow logs with zero impact on latency. Tetragon exposes its events over gRPC with the same format and vocabulary as Hubble, which allows you to:

See them in the same UI (Hubble UI shows Tetragon events as one more “layer”).
Correlate network events (Hubble) with process events (Tetragon) on the same timeline.
Export them together to Loki/Tempo/SIEM as a single stream.

The key synergy: Hubble tells you “this pod made a TCP connection to 1.2.3.4:80”. Tetragon tells you “this pod ran curl 1.2.3.4 from a bash binary launched by pid 1234”. Together they give you the full story.

Deployment and operation

Helm

The canonical install with Helm:

helm repo add cilium https://helm.cilium.io
helm install tetragon cilium/tetragon \
 --namespace kube-system \
 --set tetragon.exportFilename=/var/log/tetragon/tetragon.log \
 --set tetragon.exportFileMaxSizeMB=50 \
 --set tetragon.exportFileRotationInterval=24h

Tetragon deploys its DaemonSet, its CRDs and a service for Hubble. By default, it exposes the events on the agent pod’s stdout (any cluster log aggregator picks them up).

The `tetra` CLI

Tetragon ships a CLI called tetra for interactive investigation:

# real-time stream of the node's events
tetra getevents -o compact --pods <pod-name>

# structured JSON events to process with jq
tetra getevents -o json --since 5m --namespace prod

# view loaded policies
tetra tracingpolicy list

It is the best tool for debugging policies in monitoring before moving them to enforce.

Exporting to a SIEM

Three usual routes:

stdout + log aggregator: the agent writes JSON to stdout, Fluent Bit/Vector picks it up and sends it to Splunk/Datadog/Elastic. Simple, works with any logging infrastructure.
gRPC streaming: for low-latency integrations. A gRPC consumer of your own or Hubble Relay.
File + rotation: for air-gapped environments or regulatory audits that require persistent logs with controlled rotation.

Performance

Published benchmarks consistently place Tetragon at <1% of the node’s CPU under real workloads, compared with 5-10% for Falco under the same workloads. The reason is the architectural separation: Tetragon filters in the kernel and only carries the events that actually matter to userspace; Falco carries every syscall to userspace and filters there. On clusters with thousands of pods making hundreds of thousands of syscalls per second, the difference shows up on the bill.

Common operational traps

Permanent `monitoring`

The biggest trap is never reaching enforce: starting well with policies in monitoring, collecting events, tuning selectors, and then never switching. The result: you have detection without prevention, exactly what Falco gave you without paying Tetragon’s complexity. If you are going to use Tetragon, plan the road to enforce for the critical policies.

Selectors that are too lax

A policy with a single matchActions: Post and no specific selectors generates events for every syscall of the chosen hook. On a serious node that means tens of thousands per second, which fill logs, saturate exporters and hide the signal in the noise. Always start with strict filters (matchBinaries, matchNamespaces, matchPIDs) and open up once you know what you are looking for.

A kernel that is too old

Tetragon needs modern eBPF features. Kernels < 5.4 do not have the large buffer support (needed for execve with complete argv). Kernels < 5.10 do not have many of the LSM hooks. Kernel 5.15+ is the recommended minimum for production and 6.1+ to have every feature.

Hooks on renamed kernel functions

kprobes are tied to kernel function names that can change between versions. A policy that uses __x64_sys_setns can fail silently on a kernel where the function is called __do_sys_setns. Solutions: use static tracepoints where they are available (more stable), or keep alternative policies with several call entries for compatibility.

`Sigkill` in critical namespaces

Applying Sigkill to processes in kube-system or cilium-system can break the cluster. Enforcement policies must explicitly exclude the platform namespaces with matchNamespaces Operator: NotIn, or limit the scope with TracingPolicyNamespaced to make sure they do not act on systems they should not.

Missing `rateLimit`

A policy without a rateLimit on Post can suffer a catastrophic fan-out if the condition is met millions of times in an instant (typical in attack loops or application bugs). The agent saturates, events are lost, logs overflow. Always put a sensible rateLimit on detection policies, especially on high-frequency hooks such as tcp_connect or execve.

What we have not covered (upcoming articles)

eBPF LSM hooks in depth: how they relate to SELinux/AppArmor and when Tetragon is the right tool versus classic MAC.
Hubble UI with a Tetragon overlay: configuring the UI to show process observability and network observability on the same timeline.
Integration with OPA/Kyverno: how Tetragon complements admission policy engines (Kyverno validates at admission; Tetragon validates at runtime).
Forensics with eBPF: combining Tetragon with tools such as Beyla or OpenTelemetry to trace the full chain of an incident from the initial connection to the final syscall.

References

Official documentation (May 2026):

Tetragon — official site — entry point.
Tetragon docs — Tracing Policy — conceptual reference.
Tetragon docs — Hook points — kprobes, tracepoints, uprobes, LSM, USDT.
Tetragon docs — Selectors — full filter reference.
Tetragon docs — Enforcement Mode.
Tetragon docs — Kubernetes Identity Aware Policies — TracingPolicyNamespaced.
Tetragon GitHub.

Comparisons and analysis:

Comparative Analysis of eBPF-Based Runtime Security Monitoring (SciTePress paper 2025) — a benchmark with independent numbers.
Best eBPF Security Solutions for Kubernetes (ARMO, 2026) — Falco vs Tetragon vs KubeArmor comparison.
Falco vs. Tetragon (Asim Mirza, Medium) — analysis with use cases.
Deep Dive into Tetragon (A Security Engineer) — a walk through the inside of the agent.
Tetragon Series, Part 2: Enforcing Sensitive File Access (Medium).

Ecosystem:

Cilium Hubble — network observability.
Falco — official site — the other big name in the field.
KubeArmor — the third option, with AppArmor + eBPF.

Cross-references:

Part 1 of the series: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack — the eBPF fundamentals we take as read here.
Kubernetes with Cilium BGP: services reachable without Ingress — the starting point of the Cilium ecosystem on this blog.

Tetragon on lo0 — Tech Blog

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

TL;DR

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

A1 — The per-project .ai-jail file

A2 — Permission allowlists with –bootstrap

A3 — Lockdown for anything you do not trust

A4 — The safety net: git with no push

Track B — Cluster (RKE2 with Cilium + Tetragon)

B0 — The pod baseline

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

B2 — Tetragon, observation phase (Post)

B3 — Tetragon, enforcement phase (Sigkill)

The client ↔ cluster equivalence table

Checklist of gotchas

See also

References

The contractor with the master key: isolating AI agents from workstation to cluster

TL;DR

The analogy: the contractor with the master key

The threat model: what a runaway agent can do

Two domains, one policy

The client: isolating the agent on the workstation

The cluster: isolating the agent in production

The landscape table

Extrapolate, do not copy

What no sandbox solves

See also

References

Tetragon: Cilium's security cousin that sees every syscall in the kernel

TL;DR

The analogy: auditd on steroids in eBPF

What Tetragon is, architecturally

The two CRDs: TracingPolicy and TracingPolicyNamespaced

Anatomy of a TracingPolicy

Supported hook points

Arguments

Selectors: filtering in the kernel

Actions: from a simple Post to Sigkill

Modes: detection vs enforcement

Full example: blocking writes to /etc/passwd in the prod namespace

Common use cases

1. Execution auditing (execve)

2. Sensitive file access

3. Unauthorised outbound network connections

4. Container escape detection

5. Cryptomining

6. Detecting rootkits and suspicious kernel modules

Bonus: detecting third-party modification of eBPF maps

Comparison with Falco

Hubble + Tetragon: unified observability

Deployment and operation

Helm

The tetra CLI

Exporting to a SIEM

Performance

Common operational traps

Permanent monitoring

Selectors that are too lax

A kernel that is too old

Hooks on renamed kernel functions

Sigkill in critical namespaces

Missing rateLimit

What we have not covered (upcoming articles)

References

Full example: blocking writes to `/etc/passwd` in the prod namespace

1. Execution auditing (`execve`)

The `tetra` CLI

Permanent `monitoring`

`Sigkill` in critical namespaces

Missing `rateLimit`