Ai-Agents on lo0 — Tech Blog

The contractor with the master key: isolating AI agents from workstation to cluster

Tue, 09 Jun 2026 16:00:00 +0200

First post of a pair on AI agent isolation. This one fixes the what and the where: the complete map of isolation primitives and which domain each belongs to. The sibling runbook fixes the how, with commands: ai-jail and bubblewrap on the client, Tetragon TracingPolicy and RuntimeClass on the cluster. If you are only going to read one, this one gives you the mental model; the other one gives you the files to copy and paste.

TL;DR

An AI agent that runs code needs access to your filesystem and your tools: compiler, linter, grep, make, cargo, npm. That is the minimum to be useful. The problem is that alongside that access travels the ability to read ~/.aws/credentials, exfiltrate your SSH keys or fire an rm -rf outside the project directory. And no malicious model is needed: a compromised dependency in an npm install is enough, because the well-meaning agent and the poisoned post-install script run with the same permissions. The answer is not to trust the LLM’s good intentions; it is to isolate in order to bound the blast radius. This post walks the five isolation families of 2026, from the process sandbox to the full VM, and sorts them into two columns: what applies on the client (the developer’s workstation: bubblewrap, ai-jail, sandbox-exec, Landlock, the native Claude Code and Codex sandboxes) and what applies on the cluster (where the agent or the inference runs on Kubernetes: namespaces+seccomp, gVisor, Firecracker/Kata microVMs and eBPF/Tetragon as the layer for live observation and enforcement). The thesis: the threat model is the same in both places; the tools are not. Policy extrapolates; the primitive gets rewritten.

The analogy: the contractor with the master key

You hire a builder for a renovation. He is competent and acts in good faith. But two things happen that you do not control. The first: he may misread the instruction and knock down the wrong partition wall. The second, worse: his toolbox may have been tampered with before he came through your door — somebody put something inside — and when he opens it in your living room, that something activates.

Nobody sensible gives him the master key to the whole building. You open the room he works in, you leave him the tools he needs, and you keep the study with the safe and the server room locked. If the renovation goes wrong, by error or by sabotage, the damage stays in that room.

An AI agent is that contractor. The sandbox is the key policy: you give it the project room and the tools, not the master key to the system. And here is the twist that justifies two posts: the builder works in two different buildings. One is your flat, the developer’s workstation, with your credentials, your ~/.ssh, the browser’s password vault. The other is the datacenter, the cluster where inference and autonomous agents serve customers, with data from several tenants at once. The key policy is identical in both: least privilege, bound the radius. But the lock on your flat’s door is not the same as the one at the datacenter. In the flat you fit a cylinder lock (bubblewrap). At the datacenter you put a guard watching every door and a separate wing of the building (Tetragon + microVM). Same principle, different hardware. That is extrapolating the technology, not copying it.

The threat model: what a runaway agent can do

Before choosing a lock it helps to enumerate the burglar. The attack surface of an agent that executes arbitrary bash breaks down into five concrete threats. Not all are defended by the same layer and, an uncomfortable fact that the security documentation itself acknowledges, no single layer covers them all.

Threat	What the agent does	Minimum layer that cuts it
Filesystem out of scope	Reads `.env`, `~/.ssh/id_rsa`, system secrets; modifies sources outside the project	Process sandbox (path allowlist)
Arbitrary network egress	Exfiltrates data, receives instructions from a remote C2, calls unauthorised APIs	Network blocking / NetworkPolicy / microVM
Kernel syscall surface	A kernel exploit from inside the container escalates to the host (shared kernel)	gVisor or microVM (dedicated kernel)
Cross-tenant leakage	One customer’s workload reads another’s data on a multi-tenant platform	microVM (de facto standard)
Secret exfiltration	Extracts tokens and environment variables via `/proc` or the environment	`--clearenv` / tmpfs `$HOME` / secrets outside the pod

There is a sixth threat that no sandbox solves: prompt injection. If an attacker manages to slip instructions into the agent’s context — a poisoned comment in the code, a malicious file the agent reads, an adversarial response from a tool — the agent will execute those instructions with whatever permissions the sandbox grants it. Isolation shrinks the impact radius of a successful injection; it does not prevent the injection. That is why the sandbox is a layer, not the solution: on top go input validation, tool-call allowlists and output auditing. The sentence to internalise: isolation does not make the agent trustworthy; it bounds what an untrustworthy agent can break.

Two domains, one policy

The builder works in two buildings. The distribution of tools reads better in two columns:

The client: isolating the agent on the workstation

Here the agent is a coding assistant — Claude Code, Codex, OpenCode, Cursor — that a developer launches on their machine. One process, one user, and right next to it the juiciest assets that exist: ~/.aws/credentials, ~/.ssh, ~/.gnupg, the browser’s password store. The matching tier is the lightest one: the process sandbox.

bubblewrap (Linux) and sandbox-exec (macOS). bubblewrap (bwrap) is the same sandbox Flatpak uses to isolate each desktop app: a ~50 KB binary, ~4,000 lines of C, maintained by the GNOME team and, the key property, it runs without root via CLONE_NEWUSER, creating namespaces with no elevated privileges. It mounts $HOME as an ephemeral tmpfs and only exposes the project directory, writable; the rest of the system becomes invisible. On macOS the equivalent is sandbox-exec with SBPL profiles: a legacy Apple API, officially deprecated and with no public replacement, but working today. Parity between the two is not exact — on macOS the GPU (Metal) and the display (Cocoa) are system services and sandbox-exec does not restrict them — but both protect what matters: access to the sensitive areas of the filesystem.

Landlock as a second barrier. bubblewrap isolates via namespaces and mounts; Landlock, a Linux Security Module available since kernel 5.13, restricts access at the VFS level, independently of namespaces. It does not replace bwrap: it complements it. It closes vectors that mount-based isolation does not cover on its own (escape routes via /proc, symlink tricks inside allowed mounts) and acts as a safety net should the namespace machinery have a bug. It is defence in depth inside the client itself, and it degrades cleanly to a no-op on kernels that do not support it.

Dev containers, when reproducibility is needed. A dev container (devcontainer.json, what Codespaces and Cursor use) is a Docker container with a configuration layer on top. It gives reasonable filesystem isolation and easy reset (destroy and recreate), but it shares the host kernel, the same limitation as any Docker, and it tends to be long-lived, accumulating state. For an agent running your own team’s code on your own machine it is a good repeatability story; it is not, on its own, the isolation layer for untrusted code.

What wraps all of this. A hand-written bash script works, but it does not scale to a team. The tools that package it:

ai-jail (Rust, GPL-3.0): wraps bwrap/sandbox-exec with per-project config in a .ai-jail file (TOML, committable to the repo, so the whole team inherits the same policy), auto-detection of GPU/Docker/display, a --lockdown mode (project read-only, network cut with --unshare-net, --clearenv), --dry-run for auditing, and --bootstrap to generate each agent’s permission allowlists. It is tool-agnostic: the same binary serves Claude, Codex, OpenCode or Crush. It also applies Landlock automatically on 5.13+ kernels as defence in depth.
The Claude Code /sandbox: since October 2025 Claude Code ships its own sandbox that uses, exactly, bubblewrap on Linux and sandbox-exec on macOS. Its Sandboxed Bash isolates shell commands, but not the file tools, the MCP servers or the hooks, which run with the process’s full permissions unless you enable the beta sandbox-runtime package, which wraps the whole process. There is one nuance worth knowing: if a command fails because of a restriction, the agent can retry with dangerouslyDisableSandbox — it is opt-out, not opt-in.
Codex CLI: three modes via --sandbox (read-only, workspace-write, danger-full-access); the recommended default is workspace-write. The philosophy is deliberate: Codex does not provide the isolation, it delegates it to the environment that wraps it. danger-full-access only makes sense inside a microVM.
Cursor: its cloud agents run in isolated VMs; /worktree creates a single-use isolated worktree per task, and /best-of-n launches several attempts in parallel in separate worktrees.

The cluster: isolating the agent in production

The second building is the datacenter. Here the “agent” can be an autonomous agent running with no human in front of it, or the inference service itself executing generated code, or a multi-tenant workload where one customer’s pod must not touch another’s data. The process is no longer one: they are pods in a Kubernetes cluster (RKE2/RKE3 on a typical sovereign platform). The primitives change in nature.

The pod baseline. First of all, the standard kit: Linux namespaces, seccomp (RuntimeDefault) to trim the syscall surface, cgroups for resource limits, an unprivileged securityContext (runAsNonRoot, readOnlyRootFilesystem, dropping all capabilities) and NetworkPolicy to cut egress. It is the cluster equivalent of the process sandbox allowlist. Necessary, but it shares the kernel with the host: insufficient for genuinely untrusted code.

gVisor (runsc). Google’s user-space kernel: it intercepts the workload’s syscalls before they reach the host kernel and serves them inside a Linux kernel reimplemented in Go (the Sentry). The surface exposed to host kernel vulnerabilities drops sharply, while keeping fast startup and a low footprint. It is the middle ground when the risk of a kernel escape is real but the overhead of a microVM is not acceptable.

Firecracker / Kata microVMs. The de facto standard for untrusted code in 2026. Firecracker (AWS’s VMM written in Rust, on top of KVM) gives each sandbox a dedicated Linux kernel: a kernel exploit inside the microVM does not reach the host by construction. It is what sits underneath Vercel Sandbox (GA January 2026) and E2B. On Kubernetes, Kata Containers brings that model to a RuntimeClass: you mark the untrusted agent’s pod with runtimeClassName: kata and it runs in its own microVM instead of sharing the node’s kernel. For multi-tenant with generated code, this is the baseline, not the luxury.

eBPF / Tetragon: the layer we already have. Here is the piece that separates a platform with runtime observability from one that only trusts its configuration. The previous layers are static: they define what the pod can do before it starts. Tetragon, Cilium’s runtime security component, based on eBPF, is dynamic: it observes, in the kernel and at minimal cost, every process execution, every network connection and every file open in every pod, and it can act inline. It does not replace the sandbox; it watches it from inside the kernel. Where bubblewrap on the client blocks curl with a command blocklist, Tetragon on the cluster hooks tcp_connect in the kernel and, if the destination is not allowed, kills the process with Sigkill before the packet leaves. Where the client hides ~/.ssh behind a tmpfs, Tetragon hooks security_file_open and reports, or kills, any attempt to read a sensitive mounted path. It is the guard patrolling the corridors while the microVMs are the walls. And it is exactly the kind of control that materialises the ENS monitoring and traceability measures (op.mon, op.exp) without instrumenting the application: visibility lives in the kernel, not in the agent’s code.

The landscape table

The five families, their relative isolation strength, their startup cost and the domain where they live:

Tier	Primitive	Isolation	Startup	Natural domain
Process sandbox	Seatbelt · bubblewrap	Baseline	~0 ms	Client (Claude Code default)
Dev container	Docker + seccomp	Moderate	seconds	Client / cluster (repeatability)
User-space kernel	gVisor (`runsc`)	Strong	ms	Cluster (medium multi-tenant)
microVM	Firecracker · Kata	Strongest (practical)	<1 s	Cluster (untrusted code)
Full VM	KVM · EC2	Maximum	30 s+	Cluster (external boundary, compliance)
Runtime enforcement	eBPF · Tetragon	Cross-cutting	always on	Cluster (watches+kills on top of any tier)

Tetragon gets its own row on purpose: it is not a rung on the ladder, it is a cross-cutting layer that operates on top of any of the others. It stacks with all of them.

A numerical note on why the “startup” column decides as much as the “isolation” column. A full VM wins on raw isolation but takes tens of seconds to provision; for an agent that needs a fresh environment per request or per session, that cost is prohibitive. A Firecracker microVM starts in under 1 second and a process sandbox in ~0 ms. That is why the dominant pattern in 2026 is not “the most isolated VM”, but a full VM as the external boundary plus a microVM as the per-request execution unit inside, the architecture of Vercel, AWS Lambda and E2B. On the client the arithmetic is the opposite: the developer launches the agent dozens of times a day interactively, and a startup measured in seconds would break the flow; hence the process sandbox, with its microsecond overhead, is the correct default.

Extrapolate, do not copy

The thesis of this pair of posts fits in one sentence: the threat model is invariant across domains; the primitive that implements it is not. The client and the cluster defend exactly the same five vectors — filesystem, network, kernel, multi-tenant, secrets — but with toolboxes that do not overlap. Every control has its twin on the other side:

$HOME as an ephemeral tmpfs (client) ↔ readOnlyRootFilesystem + emptyDir (cluster).
A curl/wget blocklist in bwrap (client) ↔ a TracingPolicy on tcp_connect in Tetragon plus NetworkPolicy (cluster).
--unshare-net in lockdown (client) ↔ default-deny NetworkPolicy (cluster).
No escape hatch, the process lives inside bwrap (client) ↔ no privileged, no hostPath, RuntimeClass kata (cluster).
~/.ssh and ~/.aws never mounted (client) ↔ secrets outside the pod plus Tetragon watching security_file_open (cluster).

The sibling runbook turns each of these equivalences into concrete files. What matters here is the method: when someone shows you an agent sandbox, be it the Claude Code /sandbox on a laptop or a microVM on a PaaS, the useful question is not “which tool does it use?”, but “which of the five vectors does it close, and which does it leave open?”. The tool gets replaced; the threat map stays.

What no sandbox solves

Three limits that Anthropic’s own documentation states, and which are worth keeping in view so as not to sell smoke:

Egress remains a risk in any sandbox that allows outbound connections. If the agent can open a connection, it can exfiltrate. That is why the client lockdown cuts the network and the cluster uses default-deny NetworkPolicy plus Tetragon: you do not trust “filtering well”, you trust “not letting anything out”.
Code modification remains possible in any sandbox with the project directory mounted writable. The remedy is not a sandbox-technical one, it is git: with the remote untouched and no push permission, the worst case is corrupting the local working copy — git checkout . and start again. The damage never reaches the remote.
No sandbox prevents a compromised prompt from reaching the API. Isolation bounds the impact of an injection; it does not prevent it. The complementary defences — input validation, tool-call allowlists, output auditing — are mandatory alongside isolation, not instead of it.

The operational conclusion: isolation shrinks the blast radius; defence in depth is what closes the circle. A process sandbox for trusted code on a known machine is appropriate and practically free. For an agent acting on user prompts, executing generated code or running multi-tenant, the acceptable minimum in 2026 is a microVM, with Tetragon watching from above. Pick the tier that matches your real threat, check which vector it leaves open, and stack complementary controls on top.

References

bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io · https://docs.rs/landlock
ai-jail (Fabio Akita): https://github.com/akitaonrails/ai-jail
gVisor: https://gvisor.dev
Firecracker: https://firecracker-microvm.github.io
Kata Containers: https://katacontainers.io
Tetragon (Cilium): https://tetragon.io
Vercel Sandbox — concepts: https://vercel.com/docs/vercel-sandbox/concepts
E2B: https://github.com/e2b-dev/E2B
Claude Code sandboxing: https://docs.claude.com/en/docs/claude-code
Codex CLI: https://github.com/openai/codex

MLOps specific to LLMs in 2026: the landscape of three modalities, six stages and ten tools that make them work

Thu, 21 May 2026 05:30:00 +0200

TL;DR

This is the blog’s fourth series and it is called MLOps specific to LLMs. It takes the operational craft of traditional MLOps (reproducible pipelines, model registries, dataset versioning, eval gates, controlled deployments) and redraws it for a world where the model is probabilistic, the outputs are subjective, the dependencies include external vendors that update weights without warning, and the “application” is not a model but an orchestration of models, embeddings, retrievers, guardrails and routers. Gartner predicts that more than 50% of enterprise GenAI deployments will fail before the end of 2026, and the main cause is not the model: it is that deterministic software assumptions were applied to probabilistic systems. This post opens the series with the frame: the seven structural differences between LLMOps and classic MLOps; the six-stage pipeline (data → tune → eval → deploy → observe → retrain); the three modalities of preparing a model (continuous fine-tuning, RAG over datalakes, agent training) with their decision matrix, where 60% of 2025-2026 deployments use hybrid because each modality solves a different problem: “fine-tune for behaviour, RAG for volatile knowledge”; and the 2026 tool landscape, which already forms reasonably stable layers: MLflow 3.10 (March 2026) as a GenAI-aware registry, W&B Weave and ZenML for tracing and pipelines, Kubeflow + KServe vLLM 0.8.1+ for serving, BentoML for flexibility, DVC + lakeFS (joined since November 2025) for data, Langfuse for prompts and observability. The three following posts will go down into the detail of the most critical pieces.

This is the opening of series 4: MLOps for LLMs. It continues the tradition of the previous series: LLM inference (the first), eBPF (the second) and post-tracing (the third). Here we enter the discipline that ties all the pieces together: how an LLM system is operated in production over months, not just deployed once.

The analogy: the SRE’s craft redrawn

Anyone who has worked for years as an SRE or a platform engineer recognises the classic pillars: reproducibility (same code + same data + same config = same result), observability (what happens can be measured), safe rollback (if something goes wrong, I go back within minutes), gradual rollout (the new thing goes to 1% before it goes to 100%). These pillars are not negotiable. The question is whether they hold up when the central component is an LLM.

The answer is: same pillars, radically different mechanics. Reproducibility: versioning code and data is no longer enough; you have to version prompts, retrieval configurations, snapshots of the vendor’s model (which change without warning). Observability: error and latency metrics are no longer enough; you have to measure subjective quality via LLM-as-judge and embedding drift. Rollback: dropping the binary’s version is no longer enough; you have to keep the old model cached because loading a new one takes minutes. Gradual rollout: a % of traffic is no longer enough; you have to decide what % of what kind of queries by segment.

It is the same craft, practised with partly new tools and reflexes. MLOps specific to LLMs, or “LLMOps”, as the field has christened itself, is the discipline that codifies those reflexes.

The seven structural differences between LLMOps and traditional MLOps

Before going down to the pipeline, let us fix the differences that make this territory new rather than a mere continuation. Each one has concrete practical consequences.

1. Non-deterministic outputs

Traditional MLOps: the model receives structured input and returns a bounded, reproducible prediction. Same input → same output. Unit tests work.

LLMOps: same input → different output every time (through sampling, through temperature, through the order of tools invoked, through the retrieval context that changed). The idea of a “unit test” breaks.

Operational consequence: tests over properties (was the tone maintained?, does it mention the source?, does it respect the JSON schema?), not over equality. Statistical evals over a distribution, not over samples.

2. Behaviour metrics, not statistical accuracy

Traditional MLOps: F1, accuracy, AUC, RMSE. Metrics with a clear number.

LLMOps: subjective rubric scores (G-Eval, faithfulness, helpfulness, toxicity), judge LLMs, human feedback. The “number” depends on who judges.

Operational consequence: platforms have to treat evals as versioned artifacts, not just “model v3 scored 0.87”, but “model v3 evaluated with the judge claude-3-5-sonnet-20251022 over the dataset gold-rag-v7 with the prompt judge-v2 scored 0.87”. Versioning the judge is as important as versioning the model being evaluated.

3. The model is an external dependency, not an internal asset

Traditional MLOps: you train the model, it lives in your registry, it does not change until you change it.

LLMOps: the base model belongs to Anthropic, OpenAI, Google, Meta. They change it on you without warning. The claude-3-5-sonnet version that answered well yesterday answers somewhat differently today.

Operational consequence: drift detection becomes far more critical (previous post). Pinning to specific snapshots (claude-3-5-sonnet-20251022) when the vendor allows it. For high-commitment apps, self-hosting the base model to guarantee reproducibility.

4. The application is an orchestration, not a model

Traditional MLOps: an app calls a model and consumes its output.

LLMOps 2026: an app connects foundation model + LoRA adapters + retrievers + vector stores + guardrails + routers + tool servers (MCP) + evaluators, all composing the final behaviour. Any component can degrade the result.

Operational consequence: cross-component debugging requires distributed tracing with OTel (covered in previous posts). The registry does not only store “the model” but the composition: which prompt version + which adapter + which vector store + which retriever config.

5. Cost per inference, not per training

Traditional MLOps: the high cost is training; serving is cheap. You optimise training.

LLMOps: the high cost is serving (every token costs, every call to the vendor is paid for, the GPUs doing the serving are switched on 24/7). You optimise inference.

Operational consequence: cost accountability per tenant, per agent, per tool. Metrics such as gen_ai.usage.input_tokens aggregated at customer and product level. Model decisions based on cost per query, not only on quality.

6. GPU-heavy infra with specific primitives

Traditional MLOps: CPU + some GPU for training. Standard Kubernetes.

LLMOps: Hopper/Blackwell SXM GPUs, NVLink/NVSwitch, tensor parallel, paged attention, KV cache. Infra that only fits into Kubernetes with primitives such as LeaderWorkerSet, GPU Operator, KEDA with LLM metrics (covered in vLLM on Kubernetes).

Operational consequence: the orchestration stack includes specialised operators (OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, see LLM K8s Operators) that traditional MLOps does not contemplate.

7. RLHF and human feedback as a first-class citizen

Traditional MLOps: human feedback means labelling data before training.

LLMOps: human feedback lives inside the model in production, whether through RLHF of the foundation models (Anthropic, OpenAI), through RLAIF, through DPO, or through explicit user feedback that is fed back into fine-tuning.

Operational consequence: bidirectional production → training pipelines. Datasets grow with real incidents. Model decisions are taken with continuous feedback, not in a training project every N months.

Why Gartner predicts 50%+ failures

Gartner published that more than 50% of enterprise GenAI deployments will fail before 2026. The causes are not technical ones about the model but about the system:

Hallucinated outputs from poor grounding: badly designed RAG, poor retrieval, insufficient context.
Data architectures that are not ready: companies have data in silos, with no stable schemas, with no controlled freshness. Connecting an LLM to this data without a serious pipeline produces erratic answers.
Lack of structured workflows for prompt-driven systems: teams that treat prompts as code in hardcoded strings, with no versioning, no tests, no gates.

The conclusion the field draws: LLMOps is not optional. Companies that deploy GenAI without operational discipline fall into one of the three failure modes. Those that apply it (MLflow/W&B for tracking, DVC/lakeFS for data, Langfuse for prompts and evals, KServe or vLLM Production Stack for serving, drift detection in production) are the ones that keep the system working six months after the first release.

The six-stage LLMOps pipeline

On to the pipeline. The six stages that any serious LLM system goes through, in order:

[1. Data] → [2. Tune] → [3. Eval] → [4. Deploy] → [5. Observe] → [6. Retrain]
│
└─→ back to 1

Each stage is an operational domain of its own, with its own tools and traps:

Stage 1 — Data

What happens: ingestion, cleaning, curation, versioning, indexing of the corpus. It is where real projects suffer most, because companies have data in heterogeneous silos.

Typical sub-tasks: extraction from source (CDC over Kafka, batch from data lakes, scraping), cleaning (PII removal, dedup, format), curation (labelling for fine-tuning, golden datasets for eval), versioning (DVC + lakeFS), indexing (embeddings + vector store for RAG).

Traps: schema drift at the source, undetected PII, poor dedup that injects redundancy into training, a vector store that is not updated.

Stage 2 — Tune

What happens: preparing the model for your use case. Three modalities (we go deeper into them shortly): fine-tuning, RAG, agent training.

Typical sub-tasks: base model selection, adapter preparation (LoRA, QLoRA), training loop with continuous eval, hyperparameter sweep (Optuna, W&B Sweeps), saving the checkpoint.

Traps: catastrophic forgetting if the fine-tuning is very aggressive, overfitting to the golden dataset, no independent validation set.

Stage 3 — Eval

What happens: validating that the model + adapters + RAG configuration is acceptable before promoting. Covered in Evals.

Typical sub-tasks: running an eval framework (DeepEval, Promptfoo, Ragas) against the golden dataset, judge LLM evaluations, human review over a sample, gates with thresholds.

Traps: a golden dataset that ages, an uncalibrated judge, evals that pass in CI but fail in production because of distribution shift.

Stage 4 — Deploy

What happens: going from “the model evaluated well” to “the model serves real traffic”. Covered in LLM K8s Operators.

Typical sub-tasks: serving with vLLM/SGLang/TRT-LLM, runtime configuration, gradual rollout (canary, shadow, blue-green), routing between models (LiteLLM, OpenRouter, LangChain routers).

Traps: a naive rolling update that cuts sessions, autoscaling by CPU% that does not respond to LLM metrics (covered), a new model that performs worse in production than in eval.

Stage 5 — Observe

What happens: seeing what is happening in real time. Covered in the whole post-tracing series and the eBPF series.

Typical sub-tasks: tracing (Langfuse, LangSmith, Phoenix, OpenLLMetry), metrics (TTFT, TPOT, queue depth, cost per query), active guardrails (NeMo, Llama Guard), drift detection (Evidently, NannyML, WhyLabs).

Traps: cardinality explosion in metrics, batch evals with no tail-sampling over real traces, drift that is ignored until an incident makes it real.

Stage 6 — Retrain

What happens: closing the loop. Production feedback (incidents, worst-scoring cases, detected drift) generates new data to go back to stage 1.

Typical sub-tasks: extraction of problematic logs, human labelling of the sample, incorporation into the golden dataset, re-fine-tuning if applicable, decision on a new release.

Traps: an “open” loop where production never informs the dataset, human feedback that gets lost, lack of a clear retrain cadence.

The three modalities of “preparing the model”

Stage 2 (Tune) is where there is most confusion. In 2026 three modalities coexist, each solving a different problem:

Fine-tuning

What it does: modifying the model’s weights (or those of a LoRA/QLoRA adapter on top) so that it learns specific behaviour patterns: tone, output structure, idiomatic decisions of the domain.

When: when your main failure is behavioural inconsistency between calls. The model sometimes answers formally, sometimes not; sometimes it structures the JSON, sometimes not; sometimes it follows the company’s conventions, sometimes it invents them. Fine-tuning stabilises it.

When NOT: when what you need is up-to-date knowledge. Fine-tuning fixes knowledge in frozen weights; the day after the fine-tuning, the model knows nothing new.

RAG (Retrieval-Augmented Generation)

What it does: leaving the model intact and, on each call, retrieving fresh context from a knowledge base (vector store + lexical search, typically) and passing it to the model so that it answers on that basis.

When: when the knowledge you need is dynamic or very large. Documentation that changes, a product catalogue that is updated, an internal knowledge base that grows.

When NOT: when the problem is behavioural (RAG does not teach the model how to behave, it only gives it information). Or when retrieval is so noisy that the context arriving is worse than nothing.

Agent training

What it does: going beyond conventional fine-tuning with Reinforcement Learning techniques. RFT (OpenAI’s Reinforcement Fine-Tuning), classic RLHF, RLAIF (with AI feedback), DPO (Direct Preference Optimization) over datasets of (good, bad) pairs.

When: when the model needs to learn complex multistep trajectories: when to choose each tool, how to decompose a task, when to ask the user for confirmation. It is what is turning Claude, Gemini and GPT into agents capable of long tasks.

When NOT: when your case is simple chat or RAG. It is overkill, expensive and complicated for problems that the previous modalities solve.

Decision matrix

Observed problem	Modality
Inconsistent answers in tone/structure	Fine-tuning
The model makes things up (hallucinates)	RAG
Out-of-date knowledge (>1 year)	RAG
The model chooses tools badly	Agent training (RLAIF/RFT)
Mixed behaviour + knowledge	Hybrid (fine-tune + RAG)
Multi-step trajectory fails	Agent training
A specific language/regional style	Fine-tuning

The 2026 verdict: hybrid is the default

Multiple reports agree that in 2025-2026, around 60% of production projects use hybrid: fine-tuning for behaviour + RAG for knowledge. The key insight:

Fine-tune for behaviour (brand voice, decision protocol, output structure); use RAG for volatile knowledge you need the model to cite. Do not force a single tool to do both jobs.

A practical observation: the biggest quality improvements of 2025-2026 come from better reranking in RAG (cross-encoders), not from better embeddings. Rerankers add 15-35% of quality with little complexity.

On cost: combined fine-tuning + RAG tends to be 30-50% cheaper than pure RAG with frontier models at high volume, because the fine-tuned model can be smaller and cheaper while keeping equivalent quality.

The 2026 tool landscape

On to the concrete pieces, grouped by function. The field has matured enough for each piece to have 2-3 reasonable options and a couple of leaders.

Experiment tracking and model registry

MLflow remains the de facto standard, now with LLM-specific traction. MLflow 3 was published in June 2025; version 3.10.1 (March 2026) added:

A GenAI Overview dashboard with ready-made metrics for LLM apps.
Multi-workspace support for large teams.
Cost tracking in traces (gen_ai.usage.* aggregated per experiment).
MemAlign: a new specific eval algorithm.
Integrated native OpenTelemetry tracing.
First-class support for LangChain, LlamaIndex, AutoGen as frameworks.

MLflow treats prompts and agents as first-class citizens alongside classic models. It is the biggest change relative to MLflow 2.x.

Weights & Biases (W&B) with its LLM-specific product Weave offers tracing + eval + debug with a very polished UI. More commercial, less self-host friendly, but excellent UX.

ZenML is the piece that most cleanly integrates “classic MLOps + emerging LLMOps” into a single framework. Its automatic artifact versioning captures prompt templates, retrieval chunks and agent conversation histories with no extra work. Open-source. The most complete unification option that exists.

Dataset versioning

DVC remains the OSS standard. It extends Git to large files and pipelines. Important news from November 2025: lakeFS acquired DVC, consolidating the two OSS data versioning projects under one organisation. The combined roadmap is oriented towards LLM training and RAG datalakes specifically.

Typical pattern: Git for code + DVC for data/models + MLflow or W&B for experiment tracking + registry. Few teams use just one; the combination is what covers the cycle. Detailed in the dedicated post on data versioning for LLMOps — the four artifacts to version separately, schema contracts, end-to-end lineage, and a golden eval set with a strict holdout.

Prompt versioning and observability

Covered in depth in the AgentSight post where we go deeper into Langfuse as the OSS reference. Summary here:

Langfuse: MIT, self-host, prompt management with v1/v2/v3 versioning + labels + cache + linkage with traces.
LangSmith: if your stack is LangChain.
Arize Phoenix: ELv2, OTel-native.

Pipeline orchestration

For the LLMOps pipeline steps, the dominant options:

Kubeflow Pipelines: the K8s-native standard. KServe (the serving part of Kubeflow) has its vLLM runtime upgraded to v0.8.1+ with support for reasoning models, tool calling, embeddings, reranking, Llama 4 and Qwen 3.
ZenML: already mentioned; also a pipeline orchestrator.
Metaflow (Netflix-originated): Python-first pipelines, less LLM-specific but workable.
Argo Workflows: a pure K8s OSS alternative.
Flyte: Kubernetes-native, OSS.

Serving

Covered in depth in vLLM on Kubernetes and LLM K8s Operators. Summary:

vLLM Production Stack: a curated Helm chart.
KServe vLLM runtime: K8s-native, vLLM 0.8.1+ with full agentic support.
BentoML: flexible serving, popular in startups for its simplicity.
NVIDIA Dynamo: the successor to Triton.
llm-d: CNCF Sandbox.
OME: an LMSYS operator with native SGLang.

Evals and guardrails

Covered in Evals and Guardrails. Ultra-short summary:

Evals CI: DeepEval, Promptfoo, Ragas.
Evals platform: Langfuse, LangSmith, Phoenix, Braintrust.
Guardrails: NeMo Guardrails, Llama Guard 4, Llama Prompt Guard 2, LLM Guard, Lakera.

Drift detection and observability

Covered in the closing eBPF post. Summary:

Drift: Evidently AI, NannyML, WhyLabs.
Tracing: Langfuse, OpenLLMetry, Phoenix.
eBPF: AgentSight, Hubble, Tetragon, ProfInfer.

The typical 2026 stack table

Stage	Dominant piece	Alternatives
Data ingestion + versioning	DVC + lakeFS (unified Nov 2025)	Pachyderm, Quilt
Vector store / RAG index	Milvus, Qdrant, pgvector, Weaviate	LanceDB, Pinecone, Chroma
Experiment tracking	MLflow 3.10	W&B Weave, Neptune
Pipeline orchestration	Kubeflow + Argo	ZenML, Metaflow, Flyte
Model registry	MLflow registry	W&B Models, KServe ModelMesh
Prompt versioning	Langfuse	LangSmith, MLflow Prompts
Serving	vLLM Production Stack	KServe, BentoML, Dynamo, llm-d, OME
Evals CI	DeepEval, Ragas	Promptfoo, OpenAI Evals
Evals platform	Langfuse, Phoenix	LangSmith, Braintrust
Guardrails	NeMo + Llama Guard	LLM Guard, Lakera
Tracing	OpenLLMetry + Langfuse	Phoenix, LangSmith
Drift detection	Evidently AI	NannyML, WhyLabs
eBPF observability	AgentSight + Tetragon + Hubble	(new territory, few alternatives)

13 pieces. No org uses them all; any serious org uses at least six. This is the current LLMOps stack.

The operational reality: nobody uses a single tool

Multiple comparisons agree on one thing: the teams that win combine. Recurring patterns:

ZenML to orchestrate + MLflow for tracking + KServe for serving: the most popular OSS stack in companies coming from classic MLOps.
Kubeflow + W&B + BentoML: for teams with a research focus.
Langfuse + DeepEval + Phoenix + LiteLLM: for pure-LLM teams with no classic MLOps background.
MLflow + DVC + Argo + KServe: an idiomatic cloud-native stack without additional LLM-specifics (with its limitations).

The choice depends on the team’s background, on the licensing model they can afford, on the level of self-hosting they need, and on which frictions blocked them most in previous projects. There is no “one right answer”; there is a stable meta-pattern of layers worth covering.

Common operational traps

Treating the prompt as text in code

Hardcoding prompts in strings in the repo. Changing them requires a PR + redeploy. Result: teams that do not iterate on prompts because every change costs hours of pipeline. Solution: externalised prompt management (Langfuse, MLflow Prompts) with versioning, labels, hot-reload.

Skipping dataset versioning

“DVC is complicated, we will add it later”. Result: two months later, nobody knows which dataset trained which model. Impossible to reproduce incidents. Solution: DVC + lakeFS from day 1, even if only with a small subset.

Mixing layers in the same pipeline

Teams that put ingestion, fine-tuning, eval and deploy into a single giant pipeline. When something fails, the whole pipeline fails. Solution: independent pipelines per stage, with versioned artifacts as the interfaces between them.

Tracking with no structure

Logging everything to stdout and “we will look at it in CloudWatch later”. Result: impossible to correlate, compare, debug. Solution: OTel from day 1 with gen_ai.* semantic conventions.

Evals that block nothing

You have evals, you run them, you look at them, but they do not prevent the deploy if they drop. Eventually it drops gradually and nobody notices. Solution: eval gates in CI/CD that block the merge if critical metrics regress by more than X%.

No retrain cadence

You launch v1 and never go back to the model. Six months later, drift has degraded it but the team is on other projects. Solution: a formal retrain cadence (monthly, quarterly) tied to the production incident queue.

Invisible vendor lock-in

You start with the OpenAI API + LangSmith + Pinecone. When you want to self-host, you discover that migrating is a 3-month project. Solution: abstraction layers (LiteLLM, OpenLLMetry) and vendor-neutrality from the start.

What is coming in the following posts of the series

Post 2 — RAG over datalakes with Kafka: end-to-end technical architecture — the most hands-on one. Kafka as source-of-truth, Flink CDC, embedding pipelines, continuous indexing in Milvus/Qdrant, a complete example with real numbers and manifests.
Post 3 — The six-stage LLMOps pipeline: global architecture — the master map of the complete system with a reusable “you are here” SVG for the following posts. A deep dive into each of the six stages (Data, Tune, Eval, Deploy, Observe, Retrain).
Post 4 — PostgreSQL + Qdrant in the ingestion stage — synchronisation patterns (dual-write, outbox + CDC, event-driven), a complete microservices architecture, a Qdrant cluster manifest.
Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / alignment runtime, continuous fine-tuning in depth, edge LLMs.
Synthesis post — Anatomy of an LLM request in production, May 2026 — the complete journey of a real request through the six stages and the two cross-cutting components, with cross-links to each dedicated post. It works as a mental map of the blog and as an integrator’s guide.
Parallel OSS vs hyperscalers catalogue — The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure — for each stage of the pipeline, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, identification of real gaps and the multi-tenant chatbot ported to an AWS stack component by component. A neutral editorial stance.
OSS catalogue entry by entry — The OSS catalogue for LLMOps in six stages: entry by entry — the zoom in on the open source side: ~150 words of description per core tool (vLLM, Langfuse, DVC, Qdrant, Airflow, NeMo Guardrails, Presidio…), licence and governance, a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.

References

LLMOps vs MLOps:

Tools:

MLflow — registry + tracking + serving.
Weights & Biases Weave — LLM tracing.
ZenML — pipeline orchestration MLOps + LLMOps.
Kubeflow — K8s-native MLOps.
KServe — model serving K8s.
BentoML — flexible serving.
Metaflow — Netflix’s pipelines.
DVC — dataset versioning.
lakeFS — enterprise data versioning, acquired DVC in Nov 2025.

2026 comparisons:

RAG vs Fine-Tuning:

Cross-references (the three previous series):

LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.

MCP from the inside and its deep observability: the LSP of AI agents and how to see everything with OpenTelemetry

Wed, 20 May 2026 06:00:00 +0200

TL;DR

Model Context Protocol (MCP) is the standard Anthropic published at the end of 2024 and which by 2026 has become the dominant protocol for connecting AI agents to external tools and data. Its value, the reason the whole industry adopted it in under 18 months, is that it solves a combinatorial problem: before MCP, integrating M AI apps with N tools required M×N ad-hoc integrations; with MCP, M + N. It is the same move the Language Server Protocol made in 2016 for code editors. The architecture is three well-defined roles, Host (the AI app), Client (the connection, one per server) and Server (the piece that exposes capabilities); the primitives are six, three on the server side (Tools, Resources, Prompts) and three on the client side (Sampling, Roots, Elicitation); the protocol is JSON-RPC over two transports, stdio for local processes and Streamable HTTP for remote. The operational challenge appears when there are 10-20 MCP servers running simultaneously, each with several tools, connected to an agent that chains multistep calls: observing what happens, where things fail, how much each tool costs, which tenant invokes what becomes critical. The ecosystem’s answer in 2026: the new OpenTelemetry GenAI semantic conventions for MCP (already stable), trace context propagation via params._meta (because JSON-RPC does not bring it natively), FastMCP with built-in OTel instrumentation, MCP Gateways as a centralised layer (Traefik Hub, MintMCP, OpenObserve), and MCP Inspector for interactive debugging. This article walks the architecture from the outside in, puts each concept in its exact place, and goes down to the detail of observability: traces, RED metrics, real use cases and pitfalls.

This is the third post in the post-tracing series. Previous posts: Evals and Guardrails. Here we go down to the protocol that connects agents to tools, and how to see it in production.

The master analogy (in three versions)

MCP is a communication protocol. Like any protocol, it is best understood with the right analogy. I am going to give you three, because each one lights up a different facet and the combination leaves you understanding it better than any technical definition.

Version 1 — The USB-C of AI apps (the official one)

This is the analogy Anthropic adopted when presenting it. Before USB-C, every electronic device had its own connector. Your phone had microUSB or Lightning, your laptop a proprietary power port, your headphones a 3.5mm jack, your external disk USB-A at one end and mini-USB at the other. The result: three boxes full of specific cables that got lost, none of them good for two things, and buying a new device meant buying new accessories.

USB-C changed that. A single physical connector that many protocols cross: data (USB 3, USB 4, Thunderbolt), video (DisplayPort), power (Power Delivery), audio. You connect anything to anything and it works; the protocols negotiate above.

MCP plays the same role for AI apps. Before MCP, every application that wanted to integrate tools with an LLM, Claude Desktop, Cursor, Continue, in-house custom agents, invented its own way of doing it. Every tool vendor had to write N different integrations, one per app. The result: massive fragmentation, a lot of duplicated code, integrations that broke when an app changed its internal API.

With MCP, the connector is one: any app that speaks MCP can use any MCP tool. Just as your USB-C talks to printers, monitors and disks without the printer “knowing” whether the cable is connected to a Mac or to a Linux box.

Version 2 — The LSP of code editors (the most technically precise)

This is my favourite because the analogy is structurally identical, not just metaphorical.

Until 2016, if you wanted your code editor to support a new language, Rust, Go, TypeScript, somebody had to write a specific plugin for your particular editor. VSCode had its Rust plugin, IntelliJ a different one, Vim another, Emacs another. Every decent feature (go-to-definition, autocompletion, refactoring) was an implementation duplicated N times. M editors × N languages = M·N integrations.

In 2016 Microsoft proposed the Language Server Protocol (LSP): each language implements a single “language server” (a process that understands that language); each editor implements a single LSP client; when you work with Rust code in VSCode, VSCode launches rust-analyzer as a subprocess and talks LSP to it over stdio. Any LSP editor + any LSP server = it works. M + N.

MCP is literally this pattern, transferred from “editor + language server” to “AI app + tool provider”. And it shares even the technical detail: both pass JSON-RPC over stdio (among other transports). When Anthropic designed MCP, they looked at LSP. Anyone coming from the world of editors and IDEs will find MCP familiar.

Version 3 — The operating system driver (the operational one)

Finally, an analogy that helps you understand what a specific MCP server does.

An operating system does not know directly how to talk to your particular HP LaserJet printer. What it knows is a generic interface: “print document”, “query status”, “cancel job”. The printer driver is the piece that translates that generic interface into the proprietary commands of your specific printer.

An MCP server does exactly the same:

Your AI agent knows a generic interface: invoke a tool with a defined schema, read a resource by URI, ask for a prompt template by name.
The MCP server is the driver: it translates those generic operations into the concrete APIs of the underlying system, your PostgreSQL database, your filesystem, your GitHub API, your Stripe.

This leaves the AI agent free from knowing how it authenticates with GitHub, what exact SQL PostgreSQL uses, what endpoints Stripe has. It speaks MCP; the server takes care of the details.

With the three analogies combined: MCP is the layer between the LLM and the world, a standard USB-C implemented like LSP in JSON-RPC, with each server acting as a driver for one concrete underlying system.

What concrete problem MCP solves

Before going down to the architecture, it is worth pinning down the specific problem MCP solves, because without that many design decisions look arbitrary.

The problem is the quadratic cost of integrations.

Imagine you have M applications that use LLMs (Claude Desktop, Cursor, Continue, ChatGPT Desktop, your own custom agent, and so on) and N external tools those LLMs could use (filesystem, GitHub, Slack, PostgreSQL, Jira, Notion, and so on). Without a standard:

Each (application, tool) pair requires a specific integration.
Every time the application changes its internal API, N integrations have to be updated.
Every time the tool changes its API, M have to be updated.
For your new tool to be adopted, you have to write M integrations.
For your new application to support the ecosystem, you have to write N.

The real result in 2023-2024: massive fragmentation. OpenAI’s function calling was not compatible with Anthropic’s tool use; every framework (LangChain, LlamaIndex, dspy) had its own wrapper; Claude Desktop plugins did not work in Cursor; and so on.

MCP breaks the quadratic. Each application implements the protocol once; each tool implements the protocol once; any pair works. M + N.

It is exactly what happened with USB-C, with LSP, with SQL (before there were proprietary APIs per database), with POSIX (before there were proprietary APIs per operating system). The pattern repeats because it always solves the same kind of problem.

The architecture: three roles, clearly placed

We are going to pin down where each thing lives, because mixing up the roles is the number one source of confusion in MCP.

Three roles. Let us pin down what each one does and where it physically lives.

Host: the AI application

The Host is the application the user opens. Claude Desktop, Cursor, Continue, ChatGPT Desktop, a custom agent your team builds, a VSCode extension. What the user perceives as “the product”.

The Host is responsible for:

Deciding which MCP servers to connect (configured by the user in a file or via the UI).
Launching or connecting to each MCP server.
Creating one MCP Client per server (it is 1:1, they are not shared).
Embedding the LLM (or calling it via API) that takes the decisions about which tools to use.
Mediating the user’s authorisation for sensitive actions (showing the human “the agent wants to run tool X, do you allow it?”).

Important: the LLM lives inside the Host, not in the servers. The servers are dumb; they execute operations when asked. The reasoning (“should I call this tool now?”) lives in the host’s LLM.

Client: the connection, one per server

An MCP Client is a specific connection between the Host and a Server. If your Host has 5 MCP servers configured, it has 5 clients, not one shared. Each client:

Maintains its socket or stdio pipe with the server.
Negotiates capabilities in the initial handshake (which protocol version, which primitives both support).
Serialises JSON-RPC requests to the server and deserialises responses.
Is the point where the Host invokes operations on the server.

The 1:1 client-server separation matters because it lets each server have its own session state, its specific permissions and its independent authenticated context. There is no multiplexing in the client.

Server: the piece that exposes capabilities

The MCP Server is the piece that implements the tool-provider side of the protocol. It receives JSON-RPC from the client, processes it, executes the action against the underlying system and returns a response.

There are two flavours physically:

Local server: it starts as a subprocess of the Host and communicates over stdio. Its lifecycle is the Host’s (when you close Claude Desktop, the local servers die). Typical model: your Host launches node filesystem-mcp-server.js as a child.
Remote server: it runs as an independent service, reachable over HTTP. Multi-tenant, authenticated, scalable. Typical model: a company publishes https://mcp.acme.com/v1 and many hosts connect to it.

This difference has enormous consequences for observability (we will come back to it shortly).

Summary of where each thing lives

Component	Lives in	How many there are	Speaks what with whom
Host	User’s machine	1 (the open app)	UI with the user; launches clients
LLM	Embedded in the Host (or cloud API)	1 (the main one)	Reasons; asks for tools
Client	Host	1 per server	JSON-RPC with its server
Local server	Subprocess of the Host	1 per local integration	stdio with its client
Remote server	External service	1 per service	HTTP/SSE with its clients
Underlying system	External	Depends	API/DB/FS, not MCP

If you get confused in a discussion, come back to this table. The number one source of errors in MCP is saying “the server” when you mean “the host”.

The two layers of the protocol

MCP separates the data layer and the transport layer. This separation is what lets the protocol work over local stdio and over remote HTTP without changing anything in the primitives.

Data Layer: JSON-RPC with MCP extensions

The data layer defines the vocabulary of the messages. It is JSON-RPC 2.0. Each message is a JSON with jsonrpc: "2.0", a method (eg tools/call, resources/read), params, and an id to correlate request with response.

On top of JSON-RPC, MCP adds:

Lifecycle: the initial handshake (initialize, initialized) that negotiates capabilities.
The primitives (next section): tools/*, resources/*, prompts/*, sampling/* and so on.
Notifications: messages without a response (eg notifications/cancelled to abort a tool in progress).
Meta-information: the params._meta field carries cross-cutting metadata by convention (trace context, request IDs).

Transport Layer: how the messages move

The transport layer defines how the JSON-RPC messages travel. Two official transports:

stdio: the client launches the server as a subprocess and they communicate over its stdin/stdout/stderr with JSON-RPC. One message per line, separated by a newline. No network, no TLS handshake, no auth (trust is inherited from the operating system itself: if you launch the subprocess, you trust it). Minimal latency (~100 μs round-trip), maximum bandwidth (memcpy, not a socket).

Use case: local servers that live on the same machine as the host. Most of the MCP servers you see in public directories are stdio.

Streamable HTTP: the client sends a POST to an HTTP endpoint on the server; the server responds with JSON, optionally opening a Server-Sent Events stream to send asynchronous notifications or long responses. Auth by bearer token, API key or custom headers.

Introduced in the November 2025 spec, it replaces the pure SSE transport of earlier versions, which had bidirectionality limitations. Use case: remote servers serving many simultaneous clients, with authentication and multi-tenancy.

Important: the primitives are the same on both transports. A tools/call is identical over stdio and over HTTP. The transport is accidental, not fundamental.

The six primitives: placed in the architecture

Here is the meat. There are six primitives in MCP. They are often confused because several look like they do similar things. The key classification: three live on the server side (the server exposes, the client consumes) and three on the client side (the client exposes, the server consumes).

Server-side: what the server gives the host

Tools are actions the server exposes. Each tool has a schema (typed parameters, description) and an implementation. When the host’s LLM decides to invoke a tool, the client sends tools/call to the server, which executes it and returns a result.

Example: the github-mcp server exposes create_issue(repo, title, body). The host’s LLM decides “I am going to create an issue”, calls this tool, github-mcp talks to the GitHub API, returns the issue ID to the LLM.
Architectural place: the server exposes them, the LLM consumes them.

Resources are contextual data the server exposes, addressable by URI. They are not actions; they are content reads. A resource has a URI (file:///path/to/doc.md, postgres://table/users), metadata and an endpoint to read content.

Example: the filesystem-mcp server exposes the files in the authorised directories as resources. The LLM asks for resources/read with the URI file:///docs/api.md and gets the text.
Architectural place: the server exposes them, the host reads them (and optionally passes them to the LLM as context).

Key difference between Tools and Resources: Tools are verbs (they execute, they modify state, they have side effects); Resources are nouns (they exist, they are read, they are idempotent). If you have something that is “search for text in files” → probably a Tool (an action). If it is “this specific file” → a Resource. The distinction matters for auditing and permissions: tools require more control.

Prompts are parameterised prompt templates the server exposes. The user or the host can invoke them to inject a conversational pattern into the model.

Example: a code-review-mcp server exposes a prompt review_diff(diff_text, style="strict") that returns a complete, well-written prompt asking the LLM to review code.
Architectural place: the server exposes them, the user or the host invokes them, the LLM receives them as input.

Prompts are the least used of the three primitives; many servers do not even implement them. But they let a team publish good prompts as a reusable library, separate from the agent.

Client-side: what the host gives the server

This is where MCP differs from protocols like HTTP REST: the server can also ask things of the host, it is not a one-way street. Three primitives travel in that direction.

Sampling: the server asks the host to run a generation with its LLM. That is, the server borrows the host’s LLM in order to reason.

Example: the search-mcp server receives a query from the agent, searches its corpus, finds 50 results and needs to summarise them before returning. Instead of having its own LLM, it sends a sampling/createMessage to the client; the host passes this to its LLM, runs the generation with the user’s permissions, and returns the summary to the server.
Architectural place: the server asks for it, the host (with its LLM and the user’s authorisation) fulfils it.
Why it matters: the user controls which model is used, what cost is paid, what permissions apply. The server does not need its own OpenAI API key.

Roots: the host tells the server where to look. Roots are URIs (directories, repositories, namespaces) the host authorises the server to explore.

Example: your Claude Desktop starts filesystem-mcp with roots [file:///Users/me/projects]. The server knows it must only operate inside that folder, not in /etc/passwd.
Architectural place: the host declares them in the handshake, the server respects them.

Elicitation: the server asks the host for additional information from the human user via a structured UI.

Example: the stripe-mcp server is about to process a refund of €5000. Before executing, it sends an elicitation/createMessage to the client; the host shows the user “Confirm this €5000 refund” with a button; when the user confirms, it returns OK to the server, which then proceeds.
Architectural place: the server asks, the host shows the user, the user decides, the answer goes back to the server.
It is the key primitive for human-in-the-loop on sensitive actions.

Visualising the flow of the six primitives

 HOST SERVER
│ │
Server-side ─────┼─────────────────────────────────────┤
│ │
tools/list ──────┼─────── asks what tools exist ──────▶│
│◀────── returns the list ────────────│
│ │
tools/call ──────┼────────── run this tool ───────────▶│
│◀────── result ──────────────────────│
│ │
resources/read ──┼────────── read this URI ───────────▶│
│◀────── content ─────────────────────│
│ │
prompts/get ─────┼─────── give me this prompt ────────▶│
│◀────── compiled prompt ─────────────│
│ │
Client-side ─────┼─────────────────────────────────────┤
│ │
sampling ────────│◀────── I need a generation ─────────│
│── use my LLM ───┐ │
│── returns ──────▼──────────────────▶│
│ │
roots ───────────┼───── declared in the handshake ────▶│
│ │
elicitation ─────│◀────── ask the user X ──────────────│
│── shows UI ────┐ │
│── confirms ────▼───────────────────▶│

JSON-RPC in action: a concrete example

To make the theory concrete, here is a real MCP conversation between a client and the filesystem-mcp server:

// 1. Initial handshake (client → server)
{
"jsonrpc": "2.0", "id": 1, "method": "initialize",
"params": {
"protocolVersion": "2026-03-01",
"capabilities": {
"sampling": {}, // this client supports sampling
"roots": { "listChanged": true }
},
"clientInfo": { "name": "ClaudeDesktop", "version": "1.2.0" }
}
}
// 2. The server responds with its capabilities
{
"jsonrpc": "2.0", "id": 1, "result": {
"protocolVersion": "2026-03-01",
"capabilities": {
"tools": { "listChanged": true },
"resources": { "subscribe": true, "listChanged": true },
"prompts": {}
},
"serverInfo": { "name": "filesystem-mcp", "version": "0.5.2" }
}
}
// 3. The client asks for the list of tools
{
"jsonrpc": "2.0", "id": 2, "method": "tools/list"
}
// 4. The server returns its tools with a schema
{
"jsonrpc": "2.0", "id": 2, "result": {
"tools": [
{
"name": "read_file",
"description": "Read a file from the filesystem",
"inputSchema": {
"type": "object",
"properties": { "path": { "type": "string" } },
"required": ["path"]
}
},
{ "name": "write_file", "description": "...", "inputSchema": {} },
{ "name": "list_directory", "description": "...", "inputSchema": {} }
]
}
}
// 5. The LLM decides to call read_file; the client sends tools/call
{
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
"params": {
"name": "read_file",
"arguments": { "path": "/Users/me/projects/notes.md" },
"_meta": { // ← the extension where trace context will go
"traceparent": "00-abc123...-def456-01"
}
}
}
// 6. The server returns the file content
{
"jsonrpc": "2.0", "id": 3, "result": {
"content": [
{ "type": "text", "text": "# My notes\n\n..." }
]
}
}

The important thing to note: params._meta. That is the bag where MCP conventionally passes cross-cutting metadata, including trace context. We will come back to it shortly.

The observability problem: why traditional tracing is not enough

So much for theory. Let us go down to the operational problem: in a 2026 production cluster, a typical agent has 5-15 MCP servers connected simultaneously, each with 5-20 tools, and each conversation with the agent can generate dozens of chained tool calls. Without observability, debugging incidents is impossible.

Why generic tracing (Hubble, OTel without MCP conventions) is not enough:

Stdio is not visible on the network. Local servers talk over OS pipes. Your Hubble or your Datadog APM see nothing; there are no packets to capture. AgentSight (seen in the previous post in the eBPF series) with stdiocap captures it but gives you the raw JSON-RPC, without semantic context (which tool it is, which resource, which prompt).

Generic HTTP does not understand MCP either. If you trace the HTTP to a remote MCP server without MCP conventions, you see a POST to /v1 with an opaque JSON-RPC body. You lose “which tool was invoked”, “what arguments”, “was it elicitation or sampling”. RED metrics per endpoint are no use to you; you need RED per tool.

JSON-RPC does not propagate trace context natively. Unlike HTTP (the W3C traceparent header) or gRPC (metadata), JSON-RPC has no standard field for trace context. If you do not propagate it, every call to the server starts a new trace disconnected from the agent’s trace.

Multistep multi-server is very hard to follow. A single user conversation can translate into: 1) a call to github-mcp get_pr; 2) a call to filesystem-mcp read_file for several files; 3) a call to the main LLM with all the context; 4) a call to postgres-mcp query; 5) a call to slack-mcp send_message. Without propagated trace context, those are five disconnected traces. With propagation, it is one tree.

The solution: OpenTelemetry semantic conventions for MCP, already stable in 2026.

OpenTelemetry semantic conventions for MCP

The GenAI MCP semantic conventions are the set of standardised attributes for MCP-related spans and metrics. They were published as part of the GenAI subgroup of the OpenTelemetry SIG and are the first part of the GenAI semantic conventions to reach stable.

Why MCP-specific semantic conventions

Before they existed, teams instrumented MCP with the generic RPC semantic conventions (the ones you would use for gRPC or XML-RPC). It half worked. The MCP-specific conventions add:

Attributes to identify which primitive was executed (mcp.method.name = "tools/call").
Attributes to identify which specific tool/resource/prompt was touched (mcp.tool.name, mcp.resource.uri, mcp.prompt.name).
Attributes for the bidirectional flow (sampling/elicitation requests from the server to the client).
Attributes for the handshake (mcp.protocol.version, mcp.client.name, mcp.server.name).
Standardised RED metrics per tool (mcp.tool.call.duration, mcp.tool.call.errors).

The canonical attributes

The attributes any MCP-aware instrumentation should emit:

Attribute	Meaning	Example
`mcp.method.name`	JSON-RPC method	`"tools/call"`
`mcp.tool.name`	Name of the tool	`"read_file"`
`mcp.resource.uri`	URI of the resource	`"file:///docs/api.md"`
`mcp.prompt.name`	Name of the prompt	`"code_review"`
`mcp.session.id`	MCP session ID	`"sess-abc123"`
`mcp.protocol.version`	Protocol version	`"2026-03-01"`
`mcp.client.name`	Client identity	`"ClaudeDesktop/1.2.0"`
`mcp.server.name`	Server identity	`"filesystem-mcp/0.5.2"`
`mcp.transport`	Transport used	`"stdio"` or `"http"`
`mcp.error.code`	JSON-RPC error code	`-32602` (Invalid params)
`gen_ai.usage.input_tokens`	Tokens consumed (if sampling)	`1240`
`gen_ai.usage.output_tokens`	Tokens generated (if sampling)	`512`

The last two come from the generic GenAI semantic conventions and apply when the MCP call involves sampling (the server using the client’s LLM).

RED metrics per tool

Beyond spans, the semantic conventions define three core metrics:

mcp.tool.call.duration (histogram): the latency of each invocation.
mcp.tool.call.count (counter): the total number of invocations.
mcp.tool.call.errors (counter): errors per tool.

Labelled with mcp.tool.name, mcp.server.name, mcp.client.name. Pivotable in Grafana to answer “which tool is the slowest”, “which tool fails most”, “which client loads which server hardest”.

Trace context propagation: the `params._meta` trick

JSON-RPC has no headers like HTTP, so MCP cannot use the W3C traceparent header directly. The solution the ecosystem has converged on: propagate trace context in params._meta.

When the MCP client sends a tools/call, its OTel instrumentation does:

import json
from opentelemetry.propagate import inject

carrier = {}
inject(carrier) # fills it with traceparent/tracestate from the active span

params = {
 "name": "read_file",
 "arguments": {"path": "/notes.md"},
 "_meta": carrier, # ← propagates trace context
}

When the server receives it, it does the symmetrical thing:

from opentelemetry.propagate import extract

ctx = extract(request.params.get("_meta", {}))
with tracer.start_as_current_span("tools/call", context=ctx):
 # this span is a child of the client's
 return execute_tool(request.params)

The result: the server’s span is a child of the client’s span in the trace tree. When you look at the trace in Tempo or Phoenix, you see the whole chain: user → host → client → server → execution → response → client → host → response to the user.

This requires both ends to instrument consistently. If the server does not extract the context, you see disconnected spans but at least you have traceability on the client side.

Instrumentation patterns

There are three routes to instrumenting MCP, in increasing order of effort:

1. FastMCP with built-in OpenTelemetry

FastMCP is one of the most used Python frameworks for building MCP servers. It ships built-in OpenTelemetry instrumentation: every tool, resource template and prompt operation generates spans automatically with the correct MCP conventions.

from fastmcp import FastMCP
from opentelemetry.sdk.trace.export import OTLPSpanExporter

mcp = FastMCP("my-server", otel_endpoint="https://otel-collector:4318")

@mcp.tool()
def search_docs(query: str) -> str:
 """Search the corpus for matching documents."""
 # this automatically generates a span with
 # mcp.tool.name=search_docs, mcp.method.name=tools/call, etc.
 return run_search(query)

Zero instrumentation code. Spans with the correct conventions. It is the recommended pattern if you are starting an MCP server in Python from scratch.

2. Manual OpenTelemetry SDK

For servers that already exist or are in other languages (TypeScript, Go), the option is to instrument manually with the standard OTel SDK and emit the conventional MCP attributes:

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

async def handle_tools_call(req: JSONRPCRequest):
 ctx = extract_trace_context(req)
 with tracer.start_as_current_span("mcp.tools.call", context=ctx) as span:
 span.set_attribute("mcp.method.name", "tools/call")
 span.set_attribute("mcp.tool.name", req.params["name"])
 span.set_attribute("mcp.server.name", "filesystem-mcp")
 try:
 result = await execute_tool(req.params)
 return result
 except Exception as e:
 span.set_attribute("mcp.error.code", -32603)
 span.record_exception(e)
 raise

More boilerplate but it works with any existing server.

3. MCP Inspector for interactive debugging

MCP Inspector (official) is a tool for interactive protocol-level debugging. It launches a local proxy (port 6277) between your client and the server, and opens a web UI (port 6274) where you see every JSON-RPC message back and forth in real time.

It is not production observability, it is development and debugging. But it is irreplaceable during the bring-up of a new server: you see exactly which requests arrive, which responses are returned, which errors occur. It saves hours of ad-hoc logging.

MCP Gateways: the centralised piece for enterprise

When your organisation has many agents connecting to many MCP servers, managing the connection matrix becomes operationally serious. The natural question, “can there be a proxy in front of all the MCP servers that centralises auth, rate limiting, logging and observability?”, already has an answer: MCP Gateways.

An MCP Gateway is a proxy that:

Accepts MCP connections from the hosts/agents.
Routes them to the corresponding backend MCP servers.
Applies centralised authentication and authorisation (which agent can call which tool).
Applies rate limiting per agent, per tool, per tenant.
Observes: it emits OTel metrics for every operation passing through.
Propagates the identity of the agent to the backend server (with several models: token forwarding, token exchange, impersonation).

The options that have established themselves in 2026:

Traefik Hub MCP Gateway — from the Traefik team. Declarative configuration, native integration with Traefik’s Kubernetes/Helm ecosystem.
MintMCP — a gateway focused on observability and multi-tenancy. SaaS and self-host.
OpenObserve MCP Gateway — integrated with the OpenObserve observability platform.

For small deployments (one team, few agents) a Gateway can be overkill. For enterprise (dozens of agents, dozens of servers, regulated compliance), it is practically mandatory.

Real use cases for MCP observability

Let us land this with five cases where properly instrumented MCP observability delivers immediate value:

1. Audit per tool, per tenant, per agent

Question: “who ran the delete_repo tool last month?”. Without MCP observability, impossible. With OTel conventions + identity propagation: a query in your trace backend filtering by mcp.tool.name="delete_repo", grouping by mcp.client.name or by the user_id propagated in _meta. Compliance happy.

2. Cost per tool and per tenant

Question: “how much does each tool cost?”. If the tools invoke external APIs (Stripe, OpenAI sampling) or consume significant resources (a GPU for an inference tool), knowing their aggregated cost matters. With mcp.tool.call.duration + gen_ai.usage.* aggregated by tool and tenant, you build cost accountability dashboards without instrumenting anything extra.

3. Debugging multistep chains that fail

Question: “the agent failed to complete this task, where did it go wrong?”. The propagated trace connects: the user’s span → the LLM’s span with its CoT → the spans of each tool invoked → the final LLM span. If the chain broke at the third tool, in Tempo you see the red span with the specific error message. Reproducing the failure is trivial.

4. Tool latency and degradation

Question: “which tool is degrading?”. RED metrics per tool in Grafana show p95/p99 latency over time. When a tool starts climbing from 200ms to 800ms (because the underlying service is collapsing), you see it before the users complain.

5. Detecting loops and agentic anomalies

Question: “is any agent stuck in a loop?”. If an agent calls tools/call read_file 80 times in 30 seconds for the same path, something is clearly wrong. An alert on mcp.tool.call.count grouped by (session_id, tool_name) detects this. Combined with loop detection at the reasoning level, it closes the circle.

Operational pitfalls

Lack of identity propagation

Your Gateway authenticates the agent, but passes requests to the backend without propagating identity. The result: the backend’s logs say “service-account” for everything, and it is impossible to audit who invoked what. Choose a propagation strategy early: token forwarding (simple, exposes tokens to the backend), token exchange (more secure), or impersonation with cross-logging.

Stdio servers that do not show up in your APM

This is the field’s number 1 pitfall. Your Cursor agent uses filesystem-mcp over stdio; you see nothing in Datadog because there is no network traffic. Solution: instrument the stdio server with an OTel SDK that exports over OTLP to your collector (via gRPC or HTTP; the OTel collector can receive even though the server talks stdio to its client). Or use AgentSight stdiocap to capture the raw JSON-RPC and process it offline.

Multiple protocol versions in production

Different clients use different MCP versions simultaneously. Your metrics dashboard mixes apples and oranges. ALWAYS label with mcp.protocol.version and filter/group by it.

`_meta` lost when passing through a proxy

Your Gateway accepts the client’s request, rewrites it for the backend, and forgets to copy params._meta. The result: the trace is broken at the Gateway, two disconnected traces. Make sure your Gateway preserves or re-injects trace context at each hop.

Trace volume with chatty servers

Some MCP servers emit many small operations (filesystem listings, partial reads). Without sampling, they fill your backend with useless traces. Apply tail-based sampling that keeps complete sessions or keeps only traces with errors or high latency.

Cardinality in metrics

mcp.tool.call.duration with mcp.session.id as a label blows up cardinality. Do not include unique per-session IDs in labels; keep cardinality under control with labels that take few discrete values (tool name, server name, client name, error code).

Confusing client and server spans

When you look at the tree, distinguish: the client sees the total latency from its perspective (network included); the server sees only its own work. If you look only at the server span to debug the latency perceived by the user, you miss the RTT. Use both.

What we have not covered

Experimental WebSocket MCP transport: an alternative to Streamable HTTP, not yet standard.
MCP servers in cloud-native deployments with sidecars: an emerging pattern of deploying MCP servers as pod sidecars.
MCP federation: composing several servers as one (similar to GraphQL federation).
eBPF + MCP: how AgentSight’s stdiocap and Cilium’s hooks complement native instrumentation.
MCP testing and contract tests: how to validate that your server complies with the spec.

References

Specification and concepts:

Model Context Protocol — sitio oficial — entrada canónica.
MCP architecture overview.
Transports — MCP docs.
MCP Inspector (GitHub) — debugging interactivo.

OpenTelemetry GenAI MCP:

Semantic conventions for Model Context Protocol — OpenTelemetry — referencia normativa.
Adding OpenTelemetry Trace Support to MCP (Discussion #269) — historia de la propuesta.
How to Instrument MCP Servers with OpenTelemetry (OneUptime).
How to trace MCP server tool calls with OpenTelemetry and Elastic APM.
MCP Observability with OpenTelemetry (SigNoz).
Distributed tracing for agentic workflows (Red Hat Developer).
OpenTelemetry for AI Agents in MCP Workflows (MintMCP).

Frameworks and gateways:

FastMCP OpenTelemetry — instrumentación built-in.
Traefik Hub MCP Gateway — gateway de Traefik.
MintMCP — gateway con foco en observabilidad.
OpenObserve MCP Gateway guide.
What is an MCP Gateway (DEV Community).
OpenTelemetry MCP Server (Traceloop) — el patrón inverso: usar MCP para que agentes consulten traces OTel.

Cross-references:

Post anterior: Guardrails y safety.
AgentSight y el nuevo tracing de LLMs — donde se introdujo stdiocap para capturar stdio de servidores MCP locales.
Evals: la capa después del tracing.

AgentSight and the new LLM tracing: zero-instrumentation with eBPF against Langfuse, LangSmith, Phoenix and company

Tue, 19 May 2026 18:00:00 +0200

TL;DR

Observing an LLM agent in production in 2026 splits into two approaches with opposite philosophies. The instrumented one, dominant until 2025, lives in tools such as Langfuse, LangSmith, Arize Phoenix, Helicone, OpenLLMetry/Traceloop or Pydantic Logfire: you install an SDK, decorate your calls, emit spans with the OpenTelemetry GenAI convention (gen_ai.request.model, gen_ai.usage.input_tokens, and so on) and export them to a backend. Enormous depth when you control the code; zero visibility when the agent is an opaque binary you run without instrumenting. The zero-instrumentation one, which AgentSight popularised in the second half of 2025, turns the perspective 180º: it puts eBPF hooks on the uprobes of the SSL/TLS libraries and captures the plaintext of every HTTPS request before encryption, without touching the app’s code, with less than 3% overhead and the guarantee of being tamper-proof (the agent cannot falsify what is seen in the kernel). Combined with BPF capture of stdio for local MCP servers, AgentSight gives you complete observability of any agent, including closed binaries such as Claude Code, Gemini CLI or Cursor, on a Kubernetes cluster. The two families are not enemies: the 2026 reference stack combines both (instrumented for your own apps with LangChain, eBPF for opaque binaries and tamper-proof compliance) over OpenTelemetry GenAI semantic conventions as the common vocabulary the ecosystem is stabilising this year.

This is the fourth and last post of the eBPF series. Part 1: eBPF from zero to Cilium. Part 2: Tetragon: runtime security. Part 3: Hubble: network observability. Here we close the circle with the semantic dimension: what an AI agent does, not only what network it opens or what syscalls it emits.

The analogy: traditional APM vs network sniffer

Anyone who has operated enterprise applications knows the two tribes of monitoring. The APM tribe (New Relic, AppDynamics, Datadog APM): you install an agent or an SDK in each application, mark spans, collect traces with enormous depth inside each process, down to lines of code, SQL queries, Java methods. The wire-level tribe (network sniffers, SolarWinds NPM-style tools, NetFlow): it does not touch the application; it observes the cable, sees protocols, latencies, retransmissions, identifies problems the app does not know it has.

Each one sees different things and both are useful. Anyone who has lived through a serious incident where APM said “all green” while users suffered knows that wire-level would have caught the problem (a saturated middlebox, a badly configured MTU, a TCP timeout). Anyone who has tried to debug a memory leak with sniffers knows it was impossible without APM.

LLM agent observability in 2026 is exactly at this point. The APM-style has been in place for a couple of years: Langfuse, LangSmith, Phoenix, OpenLLMetry. Enormous depth, requires instrumenting the app. The wire-level with eBPF has just arrived: AgentSight is the first project to take it to production. Less depth inside the agent, but it sees any agent without touching anything and it is tamper-proof. Both are useful. The industry is in full coexistence.

Why observing LLM agents is different

Before getting into tools, it is worth pausing on what makes LLM agents specific as subjects of observability:

Non-determinism. The same input can produce different outputs. Reproducing an incident requires capturing exactly the conversation, the model, the parameters and, ideally, the seed. An aggregate “p95 latency” metric falls short; what you need is replay of the individual trace.

A chain of external invocations. A typical agent calls LLM → tools (tool calling) → MCP servers → other APIs → back to the LLM. A chat session can generate dozens of chained calls that have to be correlated by trace_id to understand the decision.

Cost linear in tokens. Every call is paid for in tokens. Without tracing input/output tokens per request, you cannot assign cost to a tenant or a team, nor detect loops that eat your budget in an hour.

Semantic risk. Prompt injection (a user input containing instructions to manipulate the model), jailbreaks, secret leakage through tool calls. It is a class of problem that does not appear in traditional applications, and observability must see it.

Opaque binaries. In 2026, many teams deploy third-party agents such as Claude Code, the Cursor agent, Aider, Gemini CLI or Codex CLI as internal tools. They are not your own applications; they are closed binaries that call the vendor’s API. Instrumenting them is impossible. Observing them requires something else.

Multi-agent and orchestration. More and more architectures have agents that invoke other agents (planner → executor → critic). Observability must understand the topology, not just the individual span.

With these five points in mind, the tools we are about to see differ mainly in which parts of the problem they cover well and which parts they leave blind.

The instrumented approach: how it works

The model is direct and familiar:

Your code calls the LLM or the tools using an official library: openai, anthropic, langchain, llama_index, dspy.
You install a tracer SDK (Langfuse, LangSmith, OpenLLMetry, Logfire) that wraps or monkey-patches those libraries.
Each call emits an OpenTelemetry span with standardised attributes: model used, input/output tokens, latency, parameters, messages, tool invoked, result.
The spans are exported over OTLP to a backend that shows them as a tree of traces.

# Typical example with OpenLLMetry + any SDK
from traceloop.sdk import Traceloop
from openai import OpenAI

Traceloop.init(app_name="my-agent", api_endpoint="https://otel-collector:4318")

client = OpenAI()
# this call automatically emits a span with
# gen_ai.request.model, gen_ai.usage.input_tokens, etc.
resp = client.chat.completions.create(
 model="gpt-4.1",
 messages=[{"role": "user", "content": "..."}]
)

What you see afterwards: a dashboard with each conversation as a trace, each call as a span, the complete prompts and completions (if you opt in), the computed cost, latencies per span, errors flagged.

OpenTelemetry GenAI semantic conventions: the common vocabulary

The fragmentation of the field is being mitigated with OpenTelemetry GenAI Semantic Conventions. It is the CNCF’s effort to get all tools to emit spans with the same attribute names:

gen_ai.system — the provider (openai, anthropic, vertex_ai, and so on).
gen_ai.request.model — the model requested (gpt-4.1, claude-3-5-sonnet).
gen_ai.response.model — the model actually used (sometimes it differs, e.g. fallbacks).
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — counters.
gen_ai.request.temperature, gen_ai.request.top_p, and so on — parameters.
gen_ai.response.finish_reasons — why it ended (stop, length, content_filter).
gen_ai.operation.name — the type of operation (chat, embedding, completion).

In early 2026, the client spans moved from experimental to stable. The rest (server spans, multi-agent events) is still in development. The operational meaning: if your SDK emits these attributes, any backend that understands OTel GenAI can consume them. Moving from Langfuse to Phoenix to Helicone does not imply re-instrumenting, only changing the exporter.

The SIG is actively developing conventions for multi-agent systems: agent teams, tasks, actions, memory, artifact tracking. This is what is missing for complex agent architectures to have a common vocabulary. In 2026 it is experimental; stabilisation is expected at the end of the year or the start of 2027.

Instrumented tools: the 2026 landscape

Tool	Licence	Self-host	Focus	Where it shines
Langfuse	MIT	Yes	LLM observability + evals + prompt mgmt	Best OSS balance, complete suite
LangSmith	Commercial	No	Native LangChain/LangGraph	If you use LangChain, zero-config integration
Arize Phoenix	ELv2 (OSS)	Yes	OTel-native, strong RAG	Vector DBs, retrieval, embeddings
Helicone	Commercial + OSS lite	Yes (lite)	Simple proxy	Setup in minutes, OpenAI-only
OpenLLMetry / Traceloop	Apache 2.0	Yes	OTel SDK for LLMs	Vendor-neutral, exports to any OTel backend
Pydantic Logfire	Commercial	No	App + LLM unified	If you use Pydantic AI, native integration
Weights & Biases Weave	Commercial	Limited	Experimentation + production	If you already use W&B for training
Laminar / Braintrust	Commercial	No / Yes	Evals + tracing	More recent, focused on evaluation

Deep dive: Langfuse

It is worth pausing on Langfuse because in 2026 it is the default choice among the open-source options and the one most teams have adopted this year. It is a YC W23 project, MIT licence, and keeps a sustained release cadence with serious architectural changes between versions.

Four declared pillars: observability (tracing), evaluations, prompt management, playground/datasets. Each one separately has complete commercial products behind it; Langfuse integrates them into a single platform with a single backend.

SDK v4: OTEL-native, not a replacement

The big recent operational change is the SDK v4, a thin layer over the official OpenTelemetry client. The choice is deliberate: instead of maintaining a client of its own that would fall behind the OTel primitives, Langfuse uses the standard SDK and enriches the spans with LLM-specific attributes and helpers. The consequence: any code already instrumented with vanilla OpenTelemetry (@opentelemetry/sdk-node, opentelemetry-sdk in Python) can export to Langfuse without major changes, and the other way round, if tomorrow you want to migrate from Langfuse to another OTel backend, the spans are portable.

In Python the idiomatic decorator is @observe:

from langfuse import observe, get_client

langfuse = get_client()

@observe()
def buscar_documentos(query: str):
 # any internal call is traced as well
 return vector_store.similarity_search(query)

@observe(as_type="generation")
def llamar_llm(prompt: str):
 # marked as "generation" so it appears with LLM metadata
 return openai_client.chat.completions.create(...)

@observe()
def pipeline_rag(pregunta: str):
 docs = buscar_documentos(pregunta)
 return llamar_llm(build_prompt(pregunta, docs))

The call tree is captured automatically: the trace shows pipeline_rag as the root span, with buscar_documentos and llamar_llm as children, nested. Without writing a single with tracer.start_as_current_span(...) by hand.

In TypeScript the equivalent is modular: you install @langfuse/tracing, @langfuse/otel and @opentelemetry/sdk-node, and you can use TS decorators, context managers or manual spans, the three models interoperate. The consequence: third-party libraries that emit OTel spans (openai, @anthropic-ai/sdk, Vercel AI SDK instrumentations) show up in Langfuse with no extra work.

Self-host architecture: designed for serious production

The Langfuse backend architecture has two explicit decisions that set its self-host deployment apart:

Persistence in S3/Blob Storage first. When a tracing event arrives, it is persisted in object storage before touching the database. Only when the later processing confirms OK is it inserted into Postgres/Clickhouse. If the DB goes down temporarily, the events are not lost; they stay in S3 waiting to be reprocessed. For production where losing the traces of an incident is equivalent to losing evidence, this is load-bearing.
Long migrations as background jobs. Schema upgrades that on other platforms mean a downtime window run in the background on Langfuse while the application keeps serving. Upgrade downtime drops drastically.

The officially supported deployment modes:

Docker Compose: for development and POCs. One command, everything up.
VM: a single node, containers, no orchestration. For small environments.
Kubernetes with Helm: the recommended mode for production. Official chart maintained. Supports external Postgres, external Clickhouse, external S3, HPA.

The external dependencies in a typical production setup: Postgres (metadata, prompts, configuration), Clickhouse (tracing events, high-cardinality queries), S3 or a compatible blob store (pending events), Redis (the queue between components). Yes, that is several pieces; it is what holds up the durability and the scale.

Prompt management as a first-class citizen

What sets Langfuse apart from platforms focused only on tracing is that the prompts live in Langfuse, not in the application repo or in spreadsheets. Each prompt has:

Name and version (v1, v2, v3…). Changing the prompt does not require redeploying the app: the app asks the SDK for the prompt, and the SDK caches it and refreshes when there is a new version.
Typed variables: {{user_input}}, {{context}}. Rendering with validation.
Tags and labels: by environment (production, staging), by team, by experiment.
Client and server cache: the SDK caches locally with a configurable TTL, avoiding a roundtrip to Langfuse on every call.
Linkage with traces: each trace records which exact version of which prompt was used. Investigating “this answer came out wrong” leads to prompt version Y, not to “some version of the prompt at some point”.

from langfuse import get_client

langfuse = get_client()

prompt = langfuse.get_prompt("rag-system-prompt", version=3)
# or by label: langfuse.get_prompt("rag-system-prompt", label="production")

compiled = prompt.compile(context=docs_text, user_input=question)
# 'compiled' is the final string, ready to send to the LLM

For teams iterating on prompts daily, this is what avoids the chaos of “which version of the prompt is actually running in production right now”.

Evaluations: four combinable evaluation models

Langfuse covers the four patterns of answer evaluation:

LLM-as-a-judge: you configure a model (typically GPT-4 or Claude) with a rubric and it evaluates each answer. Result: a numeric score (0-1) and a justification. Applicable to automatic tracing (all answers) or batch (a dataset selection).
User feedback: the app lets the user mark an answer as good/bad. The feedback is associated with the trace and the prompt version, which lets you see which versions have a worse rate.
Manual labeling: a UI where human labelers score answers. Useful for golden datasets and for evaluating the judge.
Custom evaluators through the API/SDK: your own evals (a unit test, a business metric) report a score through the API. They integrate with CI.

Combined, they give regression testing of the prompt: you change from v3 to v4, evaluate the golden dataset with LLM-as-judge, compare; if v4 gets worse in any of the segments, the merge fails.

Integrations

Langfuse does not compete with OpenLLMetry, LangChain or LiteLLM: it integrates them. The ones that are tested and documented:

OpenTelemetry: any OTel instrumentation emits to Langfuse over OTLP.
LangChain and LangGraph: a native callback that captures the whole chain.
LlamaIndex: native callback.
OpenAI SDK (Python and TS): a wrapper that adds tracing automatically.
LiteLLM: integration as a callback, which covers 100+ providers through LiteLLM.
OpenLLMetry / Traceloop: they emit to Langfuse like any OTel backend.
MLflow: through an OTel exporter from MLflow to Langfuse.
Vercel AI SDK: native instrumentation.

The strategy is clear: Langfuse is a backend, not an SDK. Your team chooses how it instruments; Langfuse accepts any path. The operational consequence: moving from Langfuse to another OTel backend tomorrow is viable.

When Langfuse is not the answer

So as not to present it as a silver bullet:

If you only use LangChain and have no resources to self-host: LangSmith will give you a smoother integration (it is the same team).
If your only need is a proxy with cost tracking and no evals: Helicone is simpler.
If you want an integrated commercial vendor solution: Datadog LLM Observability, New Relic AI Monitoring or Dynatrace AI are Enterprise alternatives with 24/7 support.
If your load is pure batch massive inference with no agents: you probably do not need semantic tracing; Prometheus + Grafana with OTel metrics is enough.

For everything else, your own apps with serious tracing, multi-tenant with quotas, teams iterating on prompts daily, RAG with continuous evaluation, Langfuse is the safe bet.

Quick choice summary:

LangChain → LangSmith (zero effort, automatic instrumentation).
Your own multi-framework applications with OSS → Langfuse (MIT, self-host, complete).
RAG with vector stores → Arize Phoenix (better retrieval visibility).
Simple proxy, low budget → Helicone.
Strict vendor neutrality → OpenLLMetry/Traceloop.
Pydantic AI → Logfire (same team).

Strengths and weaknesses of the instrumented model

Strengths:

Enormous depth: nested spans with all the context (chain steps, retrieval, embeddings, tool calls).
Semantic vocabulary: the SDK knows the domain (LLM, vector store, agent).
Maturity: three years of evolution, a rich ecosystem, ready-made dashboards.
Built-in evals: the top platforms combine tracing with evaluation (judge LLM, datasets, regression).

Weaknesses:

Requires control of the code: if you cannot instrument, it does not work.
Trust in the app: if the app reports badly or has a bug, so does the trace. It is not tamper-proof.
Coupling to the SDK: version changes in a library can break the instrumentation.
Uneven coverage: Python SDKs are mature; Go, Rust and JS are younger.

The zero-instrumentation approach: AgentSight

AgentSight is the project from the eunomia-bpf group that champions the opposite approach. Its paper on arxiv (2508.02736), presented at the Workshop on Practical Adoption Challenges of ML for Systems, formalises the proposal. The premise is direct:

Instead of instrumenting the agent, observe it at the system boundary.

And “system boundary” means the kernel boundary: the last point before a piece of data leaves the process towards the network or the filesystem. There, with eBPF, things are seen as they are, without the application being able to cooperate in hiding them.

Architecture: three planes

AgentSight assembles three layers:

Plane 1 — SSL/TLS uprobes. eBPF can attach programs to functions in userspace libraries (uprobes). The target functions are the encryption ones: SSL_write, SSL_read from OpenSSL/BoringSSL, and their equivalents in Rustls. AgentSight puts hooks on them that capture the arguments: the plaintext buffer the app passes to be encrypted, just before TLS processes it. On reception it does the symmetric thing: a hook after SSL_read with the freshly decrypted plaintext. Result: AgentSight sees the complete content of any HTTPS request the app makes without needing a man-in-the-middle, certificates or traffic decryption. The payload is plaintext because it was captured before being encrypted.

This works because uprobes are cheap (around 100 ns per invocation) and because apps use common TLS libraries. The few apps that implement their own TLS (rare in production) escape this hook; for those you need a different kprobe or manual instrumentation.

Plane 2 — Kernel events. In parallel, AgentSight observes relevant syscalls through tracepoints: execve (which processes start), connect/accept (network), read/write with file descriptors (filesystem and stdio), unlink, clone. Any action by the agent that has an effect outside the process passes through here. This covers, among other things, shell commands executed by the agent: if a Claude Code agent decides to run rm -rf to “clean up the project”, the execve is seen even if the LLM API does not report it.

Plane 3 — Correlation engine. The two previous planes produce streams of asynchronous events. AgentSight has a userspace component that correlates them causally cross-process: an outgoing HTTP request with bash -c rm -rf can be correlated with the previous LLM response that suggested it, through PIDs, timings and heuristics. The paper mentions the optional use of a secondary LLM (Anthropic Claude, for instance) that analyses the sequence of events and produces semantic alerts: “the agent responded with a tool call that was not in the whitelist”, “the reasoning chain has gone 47 iterations without converging”.

`stdiocap`: capturing the stdio of local MCP servers

One specific piece that deserves its own mention is stdiocap, a separate BPF tool included in the repo. The Model Context Protocol (MCP), popularised by Anthropic in 2024 and mainstream in 2025-2026, has two transport modes: HTTP/SSE (network) and stdio (between the client and the server it starts as a subprocess). Local MCP servers, the ones that run on the same machine and are started by the client as children over pipes, communicate through stdin/stdout/stderr with JSON-RPC.

stdiocap hooks read/write/dup on the stdin/stdout/stderr file descriptors of a target process and records all the JSON-RPC traffic between the MCP client and server. It is the same idea as the SSL capture but for stdio: you observe the conversation without either the client or the server knowing. Typical use case: seeing which tools of the filesystem-mcp MCP server a Claude Code agent has invoked in the last hour, what arguments it passed, what errors it received. Impossible with classic instrumentation (MCP servers are usually third-party binaries).

Guarantees: tamper-proof, kernel safety, <3% overhead

Three properties make AgentSight interesting for production:

Tamper-proof: the observation happens in the kernel (uprobes, syscalls). A malicious or compromised application cannot falsify what is seen. Compare with instrumentation: if the agent decides not to emit the span for its action, it does not appear in Langfuse. Here it has no choice.
Kernel safety: eBPF formally verifies that programs terminate and respect bounds checks. It cannot crash the kernel. The same as in the rest of the eBPF series.
<3% CPU overhead measured on real agent workloads (per the paper). The number compares favourably with SDK instrumentation, which typically adds 5-10% in intensive applications.

What it detects out of the box

The paper and the documentation highlight three classes of detection:

Real-time prompt injection: the correlation engine can apply rules or a detection model over the plaintext captured by the SSL uprobes. If the prompt contains suspicious patterns, “ignore all previous instructions”, a system prompt embedded in a user input, instructions to exfiltrate data, it raises an alert.
Reasoning loops that burn resources: agents that enter infinite loops calling tools without progressing. Detectable because the causal chain does not converge to a “final answer” and the tokens pile up. The correlation engine flags them.
Bottlenecks in multi-agent setups: when several agents coordinate, AgentSight sees the communication matrix between all of them and can detect agents that block while waiting, deadlocks, excessive fan-out.

The clash and the coexistence

The two families look like competitors, but in reality they see different things and complement each other in production.

What only the instrumented one sees

Internal agent variables that never reach the wire: the intermediate state of a LangChain chain, the values before passing them to a tool, how a prompt is built from a template with internal vars.
Deep semantic spans: retrieval > embed > vector_search > rerank > format_context > prompt_template > llm. AgentSight sees only the final call to the LLM; the path taken to build it is invisible.
Evaluations: scoring of answers, judge LLMs, quality regression. This lives only in instrumented platforms.

What only eBPF sees

Opaque binaries: Claude Code, Cursor, Gemini CLI, third-party agents. You do not have the code; you cannot instrument them. Only eBPF sees them.
System-level actions: the agent decides to run git push --force or kubectl delete. The action is seen in the execve. The agent’s instrumentation may not report it (especially if it was a command the agent generated as output without going through an explicit “tool”).
Tamper-proof audit: for regulatory compliance (HIPAA, SOC2, NIS2), having observation the app cannot dodge has formal value. eBPF provides it.
Local MCP servers over stdio: invisible to classic instrumentation unless each server emits its own spans (rare).

What both see, complementarily

Prompts and completions: the instrumented approach emits them with rich metadata; eBPF captures them off the wire. A perfect cross-check for spotting discrepancies.
Calls to external APIs: APM flags it; eBPF confirms it at kernel level.
Latency: APM per span; eBPF measures RTT at TCP level and network connectivity.

Decision matrix

Case	Instrumented	eBPF (AgentSight)
Your own app with LangChain	Yes, first	Optional
Your own multi-framework app	Yes	Optional
Third-party binary (Claude Code, Cursor)	Does not work	Yes, the only path
Tamper-proof regulatory compliance	Insufficient	Yes, required
Zero-trust multi-tenant	Insufficient	Yes, required
Local MCP servers (stdio)	Difficult	Yes, with stdiocap
Answer quality evaluation	Yes, required	No (out of scope)
Depth of the internal chain	Yes, required	No (a black box for AgentSight)
Reasoning loop detection	Possible with plumbing	Yes, built in
Real-time prompt injection	Possible (post-processed)	Yes, in stream

The natural conclusion: for your own apps, instrumented; for opaque binaries or compliance, eBPF; for everything that matters, both.

2026 reference architecture

Four recipes that cover the bulk of real cases:

Setup A — Your own application with LangChain or similar

Needs: depth, evals, a team comfortable with SDKs.

Langfuse self-host or LangSmith cloud as the backend.
OpenLLMetry SDK or LangSmith SDK instrumenting the code.
OpenTelemetry Collector between the app and the backend for routing flexibility (to Langfuse + Tempo + Loki, for instance).
Hubble for the network layer in the cluster (inter-pod latency, drop attribution).

Setup B — Productionising an opaque binary (Claude Code, Gemini CLI)

Needs: observe without touching, audit, control cost.

AgentSight deployed as a DaemonSet over the cluster (or standalone on the node).
Grafana with dashboards fed by AgentSight’s metrics.
An OTLP exporter from AgentSight to an OTel backend (Tempo, Jaeger). The spans will use the GenAI semantic conventions once they are fully standardised.
Tetragon optionally for policy over what the agent is allowed to run (Sigkill if it tries rm -rf or similar).

Setup C — Zero-trust multi-tenant platform

Needs: agents from different clients running on the same cluster, mandatory audit, nobody trusts anybody.

AgentSight as the tamper-proof audit layer. Compliance requires it.
Langfuse multi-tenant for the clients that do instrument.
Tetragon with TracingPolicyNamespaced per tenant (different policies per namespace).
Hubble with persistent flow logs for forensics.
Cilium NetworkPolicy to isolate tenants from each other on the network.

Setup D — A local MCP server on a workstation

Needs: seeing what an agent does with a stdio MCP server.

AgentSight stdiocap pointed at the PID of the client or the server.
Complete JSON-RPC capture to a file or to an OTLP endpoint.
Visualisation: Grafana, or simply jq over the log.

A real use case: if you are integrating an MCP server of your own and want to see what tool calls a Claude Code or Cursor agent makes to your server, stdiocap is the cleanest way. You need to modify neither client nor server.

Operational traps

Sensitive data in prompts (instrumented)

By default, Langfuse, LangSmith and similar tools capture the complete content of prompts and completions. If your app processes PII, secrets or medical data, that goes to your observability backend. Configuring redaction or content opt-out before going to production is mandatory. OTel GenAI has specific flags (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false) to avoid it.

Sensitive data in prompts (AgentSight)

The same problem, worse: AgentSight captures literally what goes to the wire, in plaintext. If the agent talked to api.openai.com with a prompt containing sensitive data, AgentSight has that plaintext. You have to encrypt or redact before storing.

Pinned certificates or non-standard TLS

Some high-security apps do certificate pinning or use unconventional TLS implementations (Go’s crypto/tls, custom BoringSSL). In those cases, the uprobes on libssl do not cover them. AgentSight detects when it cannot observe and reports the gap; you still have to add specific hooks for the alternative SDK.

Token volume and storage

An application with medium traffic can generate millions of tokens a day. If you store them all in Langfuse or Phoenix with long retention, the database grows fast. Strategies: aggressive sampling, short retention for normal sessions and long retention only for errors/anomalies, content redaction and keeping only metadata.

Tracing with sampling and consistency

To cut cost, many installations sample: only 1 in every N traces is persisted. Be careful with inconsistent sampling: a trace can carry several spans across multiple services, and if the sampling decision is taken per span, you end up with incomplete traces. OTel has head sampling (in the SDK, at the start), which is consistent, and tail sampling (in the collector, at the end), which allows finer rules. For LLM, tail sampling is ideal: sample everything, discard only the “normal” traces and keep the ones with errors, high latency or high cost.

Multi-agent and trace propagation

When agent A calls agent B, you have to propagate the trace context (W3C Trace Context headers) so it shows up as a single tree. If you do not, you see two disconnected traces. Modern platforms do it automatically with inject/extract, but if your transport between agents is custom (over Redis pub/sub, over a DB), you have to propagate by hand.

The cost of uprobes on critical libraries

Hooking libssl adds around 100 ns per invocation. On extreme TLS traffic loads (tens of thousands of connections/s per core), that adds up. AgentSight keeps it under 3% on typical agent workloads (which are chatty but not networking-intensive). If your use were sniffing all the HTTPS on the node, it could hurt more.

What we have not covered (next series)

Evals: the next layer after tracing. Phoenix, Langfuse, LangSmith and company offer answer evaluation (judge LLM, datasets, regression). It is a world of its own.
Guardrails and safety: NeMo Guardrails, Llama Guard, Llama Prompt Guard, evaluators specific to prompt injection and jailbreaks.
Deep MCP server observability: how the OpenTelemetry GenAI conventions are being extended to MCP servers for trace-aware tools.
eBPF + on-device inference: when the LLM runs locally through vLLM or llama.cpp, the uprobes can see the output token queue BEFORE it goes to the client. New territory.
Statistical analysis of agent flows: detecting drift, outliers, patterns that indicate degradation.

Closing the eBPF series

This series of four articles has travelled eBPF from first principles to the 2026 frontier:

eBPF from zero to Cilium — what eBPF is, networking hooks, how Cilium skips the TCP/IP stack, BGP Control Plane v2.
Tetragon: runtime security — observability and enforcement of processes in the kernel.
Hubble: network observability — L3-L7 flow logs and the frontier with AI agents.
This one — AgentSight, LLM tracing, instrumented vs zero-instrumentation.

If you have got this far you have the map to sit down with a platform, security or AI team in 2026 and recognise what each piece does, what problem it solves and where to start. That whole stack, Cilium for CNI and BGP, Tetragon for runtime security, Hubble for network observability, AgentSight for AI agents, sharing eBPF as a common substrate, Cloud Native governance and OpenTelemetry vocabulary. It is the clean architecture the industry asked for a decade ago and that finally exists.

References

AgentSight:

AgentSight GitHub (eunomia-bpf) — the project.
AgentSight: System-Level Observability for AI Agents Using eBPF (arxiv 2508.02736) — the formal paper.
AgentSight ACM workshop publication.
AgentSight blog post (eunomia.dev) — accessible description.

OpenTelemetry GenAI semantic conventions:

Instrumented platforms:

Langfuse — MIT, self-host + cloud.
LangSmith — LangChain team.
Arize Phoenix — OSS, OTel-native.
Helicone — simple proxy.
OpenLLMetry (Traceloop) — Apache 2.0, OTel SDK.
Pydantic Logfire — AI observability.

2026 comparisons:

Cross-references from the series:

eBPF from zero to Cilium.
Tetragon: runtime security.
Hubble: network observability.
LLM inference series: KV cache, vLLM on Kubernetes, PagedAttention, LLM Operators on K8s.

Hubble: eBPF network observability, state of the art 2026 and the new frontier with AI agents

Tue, 19 May 2026 06:00:00 +0200

TL;DR

Hubble is Cilium’s native network observability, built on the same eBPF programs Cilium uses for enforcement. It does not duplicate the datapath or instrument the kernel its own way: it listens to the hooks Cilium already has and produces structured flow logs with Kubernetes context included, covering pod, namespace, labels, service, policy verdict and L7 payload where applicable. It is what happens when someone decides that tcpdump with grep does not scale to 10,000 pods and builds a distributed system of their own (a Hubble server per node, plus Hubble Relay as the aggregator, plus a CLI and a UI) with practically zero overhead, because the capture was already happening. In 2026 it is at version 1.19.3 (April 2026), with Cilium 1.19 marking the project’s tenth anniversary; IP options tracing has arrived, along with filtering by encryption status, the drop event tagged with the exact NetworkPolicy that caused it (direct attribution), a stabilised field mask API, and the first wave of ML anomaly detection applied to flows for predictive security in IoT/5G clusters. And, most interesting of all for 2026: a new frontier appears where the same eBPF observes AI agents such as Claude Code, Gemini CLI and MCP agents, intercepting SSL/TLS and stdio without instrumenting the code, which turns the Cilium + Hubble + Tetragon + AgentSight stack into a complete set of tools for understanding what an agentic system does inside a cluster.

This article is part 3 of the eBPF series. Part 1: eBPF from zero to Cilium: how the kernel learned to skip its own TCP/IP stack. Part 2: Tetragon: Cilium’s security cousin that sees every syscall in the kernel. Here we complete the observability quadrant: network with Hubble, process with Tetragon, AI agent with AgentSight.

The analogy: tcpdump that speaks Kubernetes

If you have administered networks over the last twenty years, tcpdump and Wireshark have been your daily bread. They capture packets on an interface, parse them, and let you filter with tcp.port == 443 and host 10.0.0.5. They work, they have been working since the 90s, and they are the first thing you open when something smells wrong.

Now stick tcpdump on a Kubernetes cluster with 10,000 pods. The problems show up in order:

One tcpdump session per node. You wanted to “see the traffic between the frontend and the API”; you need SSH to every node, tcpdump on every NIC, synchronised timestamps, manual aggregation.
There is no K8s context. You see a packet from 10.244.5.7 to 10.244.8.42. Which pod was that? Which namespace? Which label? You have to correlate with kubectl get pod -A -o wide every single time.
No L7 understanding. You see a POST to HTTPS, and you cannot know the method and path because it is encrypted on the wire. If there is mTLS between pods, worse.
High cost: full packet capture with a copy to userspace slows down the datapath. Under dense traffic, you notice it.

Hubble is tcpdump redesigned for all of that. It reuses the eBPF programs that are already processing every packet (Cilium puts them there for enforcement) and, while they take their allow/deny decision, they emit a flow event with all the context: source and destination pod identity, namespace, labels, protocol, verdict, and, if Cilium has done L7 parsing via Envoy, the HTTP method, path, status code, DNS query, Kafka topic. That event travels through a ringbuffer to userspace, it is received by the Hubble server that lives inside the node’s Cilium agent, and it is exposed over gRPC. A separate service, Hubble Relay, aggregates the streams from all nodes and gives you a single cluster-wide API. On top of that: a CLI (hubble) and a web UI with a real-time service graph.

Zero additional copies. Zero duplicated parsing. And the result is understandable by anyone who knows what a Pod is.

Architecture: four pieces visible from outside

Hubble is made up of four logical components, all optional depending on what you want to do:

1. Hubble Server (embedded in every Cilium agent)

It lives inside the Cilium agent process (it is not a separate binary). Each node exposes a gRPC endpoint locally on the Unix socket /var/run/cilium/hubble.sock. The server listens to the events the eBPF programs emit to the ringbuffer, enriches them with Kubernetes metadata (which the agent already has in memory), and makes them available to consumers.

Enabling it: --set hubble.enabled=true in Cilium’s Helm chart. By default, the server is only accessible locally; if you want to consume it from another node, it has to be exposed (which is what Hubble Relay does).

2. Hubble Relay (aggregator)

It is a separate Deployment (typically 1 replica, scalable) that connects to every Hubble server in the cluster and aggregates their streams into a single API. When your CLI or UI asks for “the last 1000 flows in the cluster”, the Relay collects them in parallel from all nodes and returns the union.

Enabling it: --set hubble.relay.enabled=true. Without the Relay you only see the traffic of the node you are connected to, which is useful for local debugging but not for a cluster-wide view.

3. Hubble CLI (`hubble`)

A Go binary that speaks gRPC with the Relay (or with a local Hubble server). It supports two main modes:

hubble observe: a real-time stream of flows, with very expressive filters (by namespace, pod, port, verdict, protocol, label).
hubble status: the state of the Hubble cluster (how many nodes connected, lag, flow rate).

And the equivalent of tcpdump’s pcap dump: hubble observe --output jsonpb > flows.json to process afterwards with jq or other tools.

4. Hubble UI

A web frontend that connects to Hubble Relay and shows:

A service graph in real time (which Pod talks to which Service, which protocols it uses, which verdict).
A filterable list of flows.
L7 details where there are any (HTTP method/path/status, DNS query/response).

Enabling it: --set hubble.ui.enabled=true. Useful for presentations to non-CLI teams; it does not replace the CLI for serious debugging.

What you see: the Hubble flow log from the inside

A Hubble flow in JSON format looks roughly like this (simplified):

{
 "time": "2026-05-19T03:12:45.182Z",
 "verdict": "FORWARDED",
 "source": {
 "ID": 5482,
 "identity": 24871,
 "namespace": "prod-api",
 "labels": ["app=checkout", "team=payments"],
 "pod_name": "checkout-7c9f-x8j2",
 "workloads": [{"name": "checkout", "kind": "Deployment"}]
 },
 "destination": {
 "ID": 12041,
 "identity": 18356,
 "namespace": "prod-db",
 "labels": ["app=postgres", "tier=primary"],
 "pod_name": "postgres-0"
 },
 "Type": "L3_L4",
 "l4": {
 "TCP": {
 "source_port": 41982,
 "destination_port": 5432,
 "flags": {"SYN": true}
 }
 },
 "node_name": "rke2-worker-03",
 "Summary": "TCP Flags: SYN"
}

When L7 parsing is active (via embedded Envoy or Hubble’s lightweight parser), the same flow adds:

"l7": {
 "type": "REQUEST",
 "http": {
 "code": 200,
 "method": "GET",
 "url": "/api/v1/cart/items",
 "protocol": "HTTP/1.1",
 "headers": [{"key": "user-agent", "value": "checkout/1.4.2"}]
 }
}

The protocols natively supported for L7 parsing:

HTTP/1.1 and HTTP/2 (including gRPC over HTTP/2).
DNS (queries and responses, with domains, types, response codes).
Kafka (topics, API keys).
TLS handshake (SNI, not the encrypted payload by default).
MySQL, Cassandra (with optional modules).

For HTTP and gRPC, Cilium can enable the embedded Envoy proxy for the flows you want to inspect (not all of them; it is selective via CiliumNetworkPolicy with L7 rules). Without Envoy there is lightweight parsing, but less detailed.

Verdict and drop attribution

Every flow has a verdict: FORWARDED, DROPPED, ERROR, AUDIT, REDIRECTED, TRACED, TRANSLATED. For the DROPPED case, Hubble includes a structured reason (drop_reason) and, since Cilium 1.19, the exact NetworkPolicy that caused it.

That last point changes day-to-day operations. Before, when a pod could not talk to another one, the debug flow was:

See the dropped flow in Hubble.
Look at every CiliumNetworkPolicy in the namespace.
Reason manually about which one, with which labels, is blocking it.

With Cilium 1.19’s attribution, the policy_match_info field tells you directly “it was dropped by the frontend-egress policy, rule 3”. You go from “Sherlock Holmes for 20 minutes” to “kubectl get -o yaml of that specific policy”.

Prometheus metrics and Grafana dashboards

Hubble also exposes aggregated metrics in Prometheus format, separate from the gRPC flow stream. Enabling it: --set hubble.metrics.enabled=true (Helm) plus a list of the set you want to export.

The usual metric groups:

flow: total flows by verdict, source/dest, protocol.
http: requests by method, response code, latency (histogram).
dns: queries, response codes, top-N domains.
tcp: handshakes, retransmissions, congestion window.
drop: drops by reason, with NetworkPolicy attribution.
port-distribution: histogram of active ports.
policy: hits by policy and verdict.

These metrics have rich K8s labels (source_workload, destination_workload, namespace, and so on) that make them pivotable in Grafana. There are prebuilt dashboards on Grafana Labs covering the common cases; importing one and having immediate visibility costs five minutes.

Cost: metrics with many K8s labels can blow up cardinality in Prometheus. For large clusters (>1,000 pods), it is worth reviewing which set you export and using drop rules in Prometheus to keep a lid on it.

Deployment: Helm on one screen

The canonical Cilium install with full Hubble:

# values.yaml
hubble:
 enabled: true
 metrics:
 enabled:
 - dns:query;ignoreAAAA
 - drop
 - tcp
 - flow
 - port-distribution
 - icmp
 - httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
 serviceMonitor:
 enabled: true # auto-discovered by kube-prometheus-stack
 relay:
 enabled: true
 rollOutPods: true
 ui:
 enabled: true
 rollOutPods: true
 ingress:
 enabled: true
 className: cilium
 hosts:
 - hubble.example.local

And the install itself:

helm upgrade --install cilium cilium/cilium -n kube-system -f values.yaml

After installing, validate with cilium status (the Cilium CLI) that the Hubble section shows OK, and try your first flow with:

cilium hubble observe --namespace prod-api --pod checkout-7c9f-x8j2

State of the art in 2026

Cilium 1.19 was released in February 2026, marking the project’s tenth anniversary. Hubble reached version 1.19.3 on 22 April 2026. The relevant news:

Direct attribution of drops to a NetworkPolicy

Already covered above; it is probably the most valuable operational change in the release. Any dropped flow carries the name, namespace and specific rule of the policy responsible. It is also available via Prometheus metrics, which allows alerts of the form “policy X is dropping >N requests/second”.

Tracing with IP options

Hubble can now trace individual packets with IP options enabled. It is a mechanism similar to traceroute, but at L3: you put a mark on the packet and Cilium reports it every time the packet crosses a node or an eBPF decision. Useful for debugging multi-cluster paths, fabric mesh, or NetworkPolicies applied at different layers.

Filtering by encryption status

A new CLI flag: hubble observe --encryption-status=encrypted (or unencrypted). Useful for validating deployments with WireGuard or IPsec enabled pod-to-pod: you confirm that the traffic that should be encrypted is, and you spot regressions quickly.

Hubble field mask API stabilised

The field_mask lets you ask for only the parts of the flow you care about, hugely reducing bandwidth and processing when all you need is, say, source/dest and verdict. It used to be experimental; now it is stable and it is default-on in the CLI.

AI-driven anomaly detection (predictive security)

This is the most talked-about addition of 2026. Cilium 1.19 adds hooks so that an external consumer, typically an ML system, can process flows in streaming and detect statistical anomalies: pods that suddenly talk to new destinations, latency spikes on an API, odd DNS sequences. The detection part happens outside the Cilium agent (nobody wants heavy ML in the datapath), but Cilium exposes the flows with the pre-computed features the model needs. The published use cases focus on IoT and 5G, where traffic is high in volume and low in variety, ideal conditions for anomaly detection.

Scaling to 10,000+ pods

Cilium 1.19 has done serious work on scalability: Hubble Relay can now aggregate streams from hundreds of nodes without saturating; the default field_mask reduces inter-node bandwidth; and flows can be sampled under high load if your use is statistical analysis rather than forensic debugging.

Cilium 1.20 in development

Cilium 1.20 is in the development branch. The most relevant bits for Hubble:

Unification of preferIpv6: the hubble.preferIpv6 flag was deprecated in favour of the global preferIpv6 that applies to every Cilium component.
tetragon-python SDK: although it belongs to Tetragon, not Hubble, it sets a trend, namely eBPF policies written in Python instead of YAML. Hubble will probably follow a similar path.

The new frontier: eBPF and AI agents

So far, the classic Hubble content. But there is a 2026 twist worth covering, because it closes the loop with the other series on this blog.

When a Kubernetes cluster starts running AI agents, such as Claude Code, Gemini CLI, or LangGraph-based agents calling APIs and MCP servers, the observability problem changes shape. It is no longer enough to know “which pod talked to which pod” (that is Hubble) or “which process ran what” (that is Tetragon). You need to know:

Which external APIs the agent is calling and with which prompts.
Which MCP tools it is invoking, with which arguments.
How many tokens it consumes, which model it picks, how much it costs.
Whether the agent drifts from expected behaviour (out-of-policy queries, jailbreak attempts, secret leakage).

The traditional solutions, instrumenting the agent’s code with OpenTelemetry or parsing structured logs, do not work well when the agent is a third-party binary (Anthropic’s Claude Code, Google’s Gemini CLI) or when the MCP servers live in other languages with stdio as their transport.

AgentSight: zero-instrumentation for LLM agents

AgentSight (a project from the eunomia-bpf group, the same ecosystem behind several high-profile eBPF runtimes) attacks this problem with the same philosophy as Hubble: do not instrument; listen. It puts eBPF hooks at two critical points:

uprobes on SSL/TLS libraries (libssl, boringssl, rustls). It captures the plaintext before encryption on send and after decryption on recv. For an HTTP call to https://api.anthropic.com/v1/messages, AgentSight sees the full JSON of the prompt and the response without decrypting anything in transit, simply because it has reached the syscall level before the TLS layer does its work.
stdiocap BPF: it captures read, write and dup on a process’s stdin/stdout/stderr file descriptors. This is what makes it possible to observe MCP servers that speak stdio with their client, the usual pattern for local MCP servers. You capture the JSON-RPC going back and forth without either the client or the server knowing.

Reported overhead: <3% CPU, comparable to Hubble in its own regime.

How it fits with Hubble and Tetragon

The three complement each other cleanly:

Hubble tells you: “the agent’s pod opened a TCP connection to api.anthropic.com:443 with verdict ALLOW”.
Tetragon tells you: “the claude-code process with PID 1843 did a connect() to that IP” (plus the binary, the arguments, the pod namespace).
AgentSight tells you: “the HTTPS content of that connection was a prompt messages=[{role:'user', content:'analyze this repo and modify the firewall config'}] and the response included a tool call to read_file with argument /etc/passwd”.

It is the difference between flow, process and semantics. For a security team that wants to watch AI agents in production, all three are necessary. For anyone who wants to understand cost, all three are useful (Hubble for network latency, Tetragon for resource usage, AgentSight for tokens and chosen model).

Emerging use cases

The patterns consolidating in 2026:

Agent audit trail: recording every LLM call and every tool call for compliance, especially in regulated sectors.
Jailbreak and prompt injection detection: applying rules over the prompts captured by AgentSight (similar to Tetragon’s TracingPolicy, but over semantic content).
Cost accountability: seeing which team/agent consumes which tokens, without instrumenting.
Replay and debugging: reproducing an agent’s reasoning in production without asking it to run again (which is non-deterministic).

It is a young field, AgentSight is months old rather than years, but the “eBPF as zero-instrumentation observability” pattern is very clearly spreading beyond network and process. The coming year will see consolidation and, probably, native integration with Hubble.

Common Hubble use cases

Back to Hubble proper, the cases in which any organisation deploys it:

1. NetworkPolicy debugging

The classic use: “this pod cannot reach this Service”. Without Hubble, it meant SSH, tcpdump, comparing rules. With Hubble:

hubble observe --from-pod prod-api/checkout --to-pod prod-db/postgres --verdict DROPPED

If there are drops, you see the policy responsible (Cilium 1.19+). If there are no drops, the problem is not policy: it is DNS, routing or the target service.

2. Auditing inter-namespace communication

For compliance: validating that isolated namespaces are not communicating against what was declared.

hubble observe --from-namespace prod-payments --to-namespace 'NOT prod-db' --output json

3. Exfiltration detection

Outbound traffic to suspicious public destinations. Hubble detects them by IP/SNI, not by payload (which is encrypted):

hubble observe --to-fqdn 'NOT *.example.com' --to-fqdn 'NOT *.internal' --protocol tcp

Combined with Prometheus metrics and Grafana alerts, this gives you an exfiltration radar at zero cost.

4. Real-time service SLOs

The hubble:http:response_time_seconds metrics with source_workload, destination_workload, method and status_code labels allow SLO dashboards without needing to instrument the apps. The SRE sees the p95 latency of checkout → catalog directly.

5. Performance debugging

hubble:tcp:retransmissions_total and hubble:tcp:flags_total{flag="RST"} are early signals of network problems. A rise correlated with a latency regression points you at something in the infrastructure (NIC, switch, MTU) before you go down to investigate the app.

6. Post-incident forensics

Configuring Hubble to export flows to persistent storage (via OTLP to Tempo/Loki, or hubble observe --output jsonpb to S3) gives you forensic capability: if at T+30 days you detect that something was wrong at T, you can reconstruct the traffic.

Hubble and the rest of the observability stack

Hubble does not replace Prometheus, Loki, Tempo or Jaeger; it complements them:

Prometheus: receives Hubble’s aggregated metrics. Hubble exports a native Prometheus endpoint.
Loki: receives the structured flow logs if you export them as logs. Hubble has no native Loki exporter, but a Fluent Bit with an OTLP plugin or a custom one bridges it easily.
Tempo / Jaeger: the Cilium Operator has an OTLP exporter of flows in trace format (each HTTP/gRPC flow is a span). It integrates with Tempo or any other OTLP tracing backend.
Grafana: there are already public Hubble dashboards. Combined with Prometheus, Loki and Tempo, they give you a unified panel: metrics, logs, traces, all correlated by K8s labels.

The full-stack pile seen in production in 2026 (described in Building a Production eBPF Observability & Security Stack for Kubernetes in 2026):

Data: Cilium + Hubble (network), Tetragon (process), AgentSight (AI agent).
Pipeline: OTLP Collector as the single router.
Storage: Prometheus (metrics), Loki (logs), Tempo (traces).
UI: Grafana with domain-specific dashboards.
Alerting: AlertManager with rules over the Hubble + Tetragon metrics.

Comparison with the alternatives

System	Layer	Focus	Model
Hubble	L3-L7 network	K8s cluster with Cilium	eBPF, pull metrics, push flows gRPC
GKE Dataplane v2 obs	L3-L7 network	GKE managed	eBPF (Cilium-based, managed)
Tigera Calico Whisker	L3-L7 network	Cluster with Calico	eBPF + pcap, own UI
Tetragon	Process/syscall	K8s cluster	eBPF, push events gRPC
Falco	Process/syscall	K8s cluster	eBPF in userspace or kernel module
AgentSight	LLM agent	Agentic systems	eBPF (SSL uprobes + stdio)
Beyla (Grafana)	Application	App L7 + tracing	eBPF (uprobes on libs)
Pixie	App + system	Broad cluster visibility	eBPF + PXL script
Parca	CPU/mem profiling	Performance	eBPF profile sampling

If your CNI is Cilium, Hubble is the natural entry point and it does not compete with the rest: it complements them. For Calico clusters, Whisker is the equivalent. For profiling, Parca. For AI agents, AgentSight. The era of “one tool for everything” is passing: the modern stack combines several specialised pieces, all eBPF-based, exposed via OTLP.

Operational traps

Cardinality in Prometheus

Hubble metrics with all the K8s labels can blow up Prometheus. Measure cardinality before exporting everything. The most prolific metrics are flow and httpV2; start with drop and port-distribution and add the rest incrementally.

L7 visibility costs CPU

Enabling L7 parsing via embedded Envoy adds load to the agent (not to the base datapath, but yes to the node’s envoy proxy). For heavy HTTP traffic, measure. For flows where you only need L4, leave Envoy disabled.

Hubble Relay without HA

A single Relay replica is a single point of failure for the CLI and the UI (not for the local agent, which keeps working). For production, deploy with replicas: 2+ and topologySpreadConstraints so both do not go down together.

Encryption status reporting depends on the Cilium config

The new --encryption-status filter only gives real data if Cilium has encryption enabled (WireGuard or IPsec). Without that, everything is unencrypted and the filter contributes nothing.

UI exposed without auth

Hubble UI has no native auth. If you expose it through Ingress, there has to be authentication in front of it: OIDC via oauth2-proxy, mTLS, IP allowlist. It is not optional.

Storage that does not scale

If you keep flows for days for forensics, the volume is serious. For a cluster with 100 active pods, easily 1-10 GB/day of flow logs. Plan the lifecycle (compaction, retention, cold storage) before enabling it.

What we have not covered

Mesh / multi-cluster Hubble: aggregating flows from several Cilium clusters into a single Relay. Use case: a cross-cluster view, debugging a distributed service mesh.
hubble export: local persistence on the agent’s disk for forensics with low retention.
Anomaly detection with your own models: how to connect the gRPC stream to a custom ML consumer.
AgentSight in depth: the project deserves its own article. Next instalment.
eBPF for LLM serving profiling: how to measure vLLM’s TTFT, TPOT and throughput without instrumenting, using uprobes on libcudart.

References

Hubble and Cilium:

Hubble GitHub — main repo.
Hubble — Network Observability (Cilium docs) — official reference.
Cilium 1.19 release notes (InfoQ, feb 2026) — tenth anniversary and 1.19 news.
Cilium releases — all releases.
Hubble L7 HTTP Metrics — Grafana dashboard 19423 — ready to import.
End‑to‑end L7 Visibility with Cilium Hubble (cloud-cod.com, mar 2026).
Cilium Hubble Observability Platform Internal Analysis (Young-ju).
CiliumNetworkPolicy Python Hubble: L7 Visibility 2026 — one of the threads on the Python SDK.

State of the art 2026 and the full stack:

eBPF + AI agents:

AgentSight (GitHub eunomia-bpf) — the referenced project.
Harnessing eBPF for High‑Performance LLM Workloads (Klizo Solutions).

Cross-references:

Part 1: eBPF from zero to Cilium.
Part 2: Tetragon: Cilium’s security cousin.
LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators — where the traffic Hubble observes carries the prompts AgentSight inspects.

Ai-Agents on lo0 — Tech Blog

The contractor with the master key: isolating AI agents from workstation to cluster

TL;DR

The analogy: the contractor with the master key

The threat model: what a runaway agent can do

Two domains, one policy

The client: isolating the agent on the workstation

The cluster: isolating the agent in production

The landscape table

Extrapolate, do not copy

What no sandbox solves

See also

References

MLOps specific to LLMs in 2026: the landscape of three modalities, six stages and ten tools that make them work

TL;DR

The analogy: the SRE’s craft redrawn

The seven structural differences between LLMOps and traditional MLOps

1. Non-deterministic outputs

2. Behaviour metrics, not statistical accuracy

3. The model is an external dependency, not an internal asset

4. The application is an orchestration, not a model

5. Cost per inference, not per training

6. GPU-heavy infra with specific primitives

7. RLHF and human feedback as a first-class citizen

Why Gartner predicts 50%+ failures

The six-stage LLMOps pipeline

Stage 1 — Data

Stage 2 — Tune

Stage 3 — Eval

Stage 4 — Deploy

Stage 5 — Observe

Stage 6 — Retrain

The three modalities of “preparing the model”

Fine-tuning

RAG (Retrieval-Augmented Generation)

Agent training

Decision matrix

The 2026 verdict: hybrid is the default

The 2026 tool landscape

Experiment tracking and model registry

Dataset versioning

Prompt versioning and observability

Pipeline orchestration

Serving

Evals and guardrails

Drift detection and observability

The typical 2026 stack table

The operational reality: nobody uses a single tool

Common operational traps

Treating the prompt as text in code

Skipping dataset versioning

Mixing layers in the same pipeline

Tracking with no structure

Evals that block nothing

No retrain cadence

Invisible vendor lock-in

What is coming in the following posts of the series

References

MCP from the inside and its deep observability: the LSP of AI agents and how to see everything with OpenTelemetry

TL;DR

The master analogy (in three versions)

Version 1 — The USB-C of AI apps (the official one)

Version 2 — The LSP of code editors (the most technically precise)

Version 3 — The operating system driver (the operational one)

What concrete problem MCP solves

The architecture: three roles, clearly placed

Host: the AI application

Client: the connection, one per server

Server: the piece that exposes capabilities

Summary of where each thing lives

The two layers of the protocol

Data Layer: JSON-RPC with MCP extensions

Transport Layer: how the messages move

The six primitives: placed in the architecture

Server-side: what the server gives the host

Client-side: what the host gives the server

Visualising the flow of the six primitives

JSON-RPC in action: a concrete example

The observability problem: why traditional tracing is not enough

OpenTelemetry semantic conventions for MCP

Trace context propagation: the `params._meta` trick

`_meta` lost when passing through a proxy

`stdiocap`: capturing the stdio of local MCP servers