AgentSight and the new LLM tracing: zero-instrumentation with eBPF against Langfuse, LangSmith, Phoenix and company

Tue, 19 May 2026 18:00:00 +0200

TL;DR

Observing an LLM agent in production in 2026 splits into two approaches with opposite philosophies. The instrumented one, dominant until 2025, lives in tools such as Langfuse, LangSmith, Arize Phoenix, Helicone, OpenLLMetry/Traceloop or Pydantic Logfire: you install an SDK, decorate your calls, emit spans with the OpenTelemetry GenAI convention (gen_ai.request.model, gen_ai.usage.input_tokens, and so on) and export them to a backend. Enormous depth when you control the code; zero visibility when the agent is an opaque binary you run without instrumenting. The zero-instrumentation one, which AgentSight popularised in the second half of 2025, turns the perspective 180º: it puts eBPF hooks on the uprobes of the SSL/TLS libraries and captures the plaintext of every HTTPS request before encryption, without touching the app’s code, with less than 3% overhead and the guarantee of being tamper-proof (the agent cannot falsify what is seen in the kernel). Combined with BPF capture of stdio for local MCP servers, AgentSight gives you complete observability of any agent, including closed binaries such as Claude Code, Gemini CLI or Cursor, on a Kubernetes cluster. The two families are not enemies: the 2026 reference stack combines both (instrumented for your own apps with LangChain, eBPF for opaque binaries and tamper-proof compliance) over OpenTelemetry GenAI semantic conventions as the common vocabulary the ecosystem is stabilising this year.

This is the fourth and last post of the eBPF series. Part 1: eBPF from zero to Cilium. Part 2: Tetragon: runtime security. Part 3: Hubble: network observability. Here we close the circle with the semantic dimension: what an AI agent does, not only what network it opens or what syscalls it emits.

The analogy: traditional APM vs network sniffer

Anyone who has operated enterprise applications knows the two tribes of monitoring. The APM tribe (New Relic, AppDynamics, Datadog APM): you install an agent or an SDK in each application, mark spans, collect traces with enormous depth inside each process, down to lines of code, SQL queries, Java methods. The wire-level tribe (network sniffers, SolarWinds NPM-style tools, NetFlow): it does not touch the application; it observes the cable, sees protocols, latencies, retransmissions, identifies problems the app does not know it has.

Each one sees different things and both are useful. Anyone who has lived through a serious incident where APM said “all green” while users suffered knows that wire-level would have caught the problem (a saturated middlebox, a badly configured MTU, a TCP timeout). Anyone who has tried to debug a memory leak with sniffers knows it was impossible without APM.

LLM agent observability in 2026 is exactly at this point. The APM-style has been in place for a couple of years: Langfuse, LangSmith, Phoenix, OpenLLMetry. Enormous depth, requires instrumenting the app. The wire-level with eBPF has just arrived: AgentSight is the first project to take it to production. Less depth inside the agent, but it sees any agent without touching anything and it is tamper-proof. Both are useful. The industry is in full coexistence.

Why observing LLM agents is different

Before getting into tools, it is worth pausing on what makes LLM agents specific as subjects of observability:

Non-determinism. The same input can produce different outputs. Reproducing an incident requires capturing exactly the conversation, the model, the parameters and, ideally, the seed. An aggregate “p95 latency” metric falls short; what you need is replay of the individual trace.

A chain of external invocations. A typical agent calls LLM → tools (tool calling) → MCP servers → other APIs → back to the LLM. A chat session can generate dozens of chained calls that have to be correlated by trace_id to understand the decision.

Cost linear in tokens. Every call is paid for in tokens. Without tracing input/output tokens per request, you cannot assign cost to a tenant or a team, nor detect loops that eat your budget in an hour.

Semantic risk. Prompt injection (a user input containing instructions to manipulate the model), jailbreaks, secret leakage through tool calls. It is a class of problem that does not appear in traditional applications, and observability must see it.

Opaque binaries. In 2026, many teams deploy third-party agents such as Claude Code, the Cursor agent, Aider, Gemini CLI or Codex CLI as internal tools. They are not your own applications; they are closed binaries that call the vendor’s API. Instrumenting them is impossible. Observing them requires something else.

Multi-agent and orchestration. More and more architectures have agents that invoke other agents (planner → executor → critic). Observability must understand the topology, not just the individual span.

With these five points in mind, the tools we are about to see differ mainly in which parts of the problem they cover well and which parts they leave blind.

The instrumented approach: how it works

The model is direct and familiar:

Your code calls the LLM or the tools using an official library: openai, anthropic, langchain, llama_index, dspy.
You install a tracer SDK (Langfuse, LangSmith, OpenLLMetry, Logfire) that wraps or monkey-patches those libraries.
Each call emits an OpenTelemetry span with standardised attributes: model used, input/output tokens, latency, parameters, messages, tool invoked, result.
The spans are exported over OTLP to a backend that shows them as a tree of traces.

# Typical example with OpenLLMetry + any SDK
from traceloop.sdk import Traceloop
from openai import OpenAI

Traceloop.init(app_name="my-agent", api_endpoint="https://otel-collector:4318")

client = OpenAI()
# this call automatically emits a span with
# gen_ai.request.model, gen_ai.usage.input_tokens, etc.
resp = client.chat.completions.create(
 model="gpt-4.1",
 messages=[{"role": "user", "content": "..."}]
)

What you see afterwards: a dashboard with each conversation as a trace, each call as a span, the complete prompts and completions (if you opt in), the computed cost, latencies per span, errors flagged.

OpenTelemetry GenAI semantic conventions: the common vocabulary

The fragmentation of the field is being mitigated with OpenTelemetry GenAI Semantic Conventions. It is the CNCF’s effort to get all tools to emit spans with the same attribute names:

gen_ai.system — the provider (openai, anthropic, vertex_ai, and so on).
gen_ai.request.model — the model requested (gpt-4.1, claude-3-5-sonnet).
gen_ai.response.model — the model actually used (sometimes it differs, e.g. fallbacks).
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — counters.
gen_ai.request.temperature, gen_ai.request.top_p, and so on — parameters.
gen_ai.response.finish_reasons — why it ended (stop, length, content_filter).
gen_ai.operation.name — the type of operation (chat, embedding, completion).

In early 2026, the client spans moved from experimental to stable. The rest (server spans, multi-agent events) is still in development. The operational meaning: if your SDK emits these attributes, any backend that understands OTel GenAI can consume them. Moving from Langfuse to Phoenix to Helicone does not imply re-instrumenting, only changing the exporter.

The SIG is actively developing conventions for multi-agent systems: agent teams, tasks, actions, memory, artifact tracking. This is what is missing for complex agent architectures to have a common vocabulary. In 2026 it is experimental; stabilisation is expected at the end of the year or the start of 2027.

Instrumented tools: the 2026 landscape

Tool	Licence	Self-host	Focus	Where it shines
Langfuse	MIT	Yes	LLM observability + evals + prompt mgmt	Best OSS balance, complete suite
LangSmith	Commercial	No	Native LangChain/LangGraph	If you use LangChain, zero-config integration
Arize Phoenix	ELv2 (OSS)	Yes	OTel-native, strong RAG	Vector DBs, retrieval, embeddings
Helicone	Commercial + OSS lite	Yes (lite)	Simple proxy	Setup in minutes, OpenAI-only
OpenLLMetry / Traceloop	Apache 2.0	Yes	OTel SDK for LLMs	Vendor-neutral, exports to any OTel backend
Pydantic Logfire	Commercial	No	App + LLM unified	If you use Pydantic AI, native integration
Weights & Biases Weave	Commercial	Limited	Experimentation + production	If you already use W&B for training
Laminar / Braintrust	Commercial	No / Yes	Evals + tracing	More recent, focused on evaluation

Deep dive: Langfuse

It is worth pausing on Langfuse because in 2026 it is the default choice among the open-source options and the one most teams have adopted this year. It is a YC W23 project, MIT licence, and keeps a sustained release cadence with serious architectural changes between versions.

Four declared pillars: observability (tracing), evaluations, prompt management, playground/datasets. Each one separately has complete commercial products behind it; Langfuse integrates them into a single platform with a single backend.

SDK v4: OTEL-native, not a replacement

The big recent operational change is the SDK v4, a thin layer over the official OpenTelemetry client. The choice is deliberate: instead of maintaining a client of its own that would fall behind the OTel primitives, Langfuse uses the standard SDK and enriches the spans with LLM-specific attributes and helpers. The consequence: any code already instrumented with vanilla OpenTelemetry (@opentelemetry/sdk-node, opentelemetry-sdk in Python) can export to Langfuse without major changes, and the other way round, if tomorrow you want to migrate from Langfuse to another OTel backend, the spans are portable.

In Python the idiomatic decorator is @observe:

from langfuse import observe, get_client

langfuse = get_client()

@observe()
def buscar_documentos(query: str):
 # any internal call is traced as well
 return vector_store.similarity_search(query)

@observe(as_type="generation")
def llamar_llm(prompt: str):
 # marked as "generation" so it appears with LLM metadata
 return openai_client.chat.completions.create(...)

@observe()
def pipeline_rag(pregunta: str):
 docs = buscar_documentos(pregunta)
 return llamar_llm(build_prompt(pregunta, docs))

The call tree is captured automatically: the trace shows pipeline_rag as the root span, with buscar_documentos and llamar_llm as children, nested. Without writing a single with tracer.start_as_current_span(...) by hand.

In TypeScript the equivalent is modular: you install @langfuse/tracing, @langfuse/otel and @opentelemetry/sdk-node, and you can use TS decorators, context managers or manual spans, the three models interoperate. The consequence: third-party libraries that emit OTel spans (openai, @anthropic-ai/sdk, Vercel AI SDK instrumentations) show up in Langfuse with no extra work.

Self-host architecture: designed for serious production

The Langfuse backend architecture has two explicit decisions that set its self-host deployment apart:

Persistence in S3/Blob Storage first. When a tracing event arrives, it is persisted in object storage before touching the database. Only when the later processing confirms OK is it inserted into Postgres/Clickhouse. If the DB goes down temporarily, the events are not lost; they stay in S3 waiting to be reprocessed. For production where losing the traces of an incident is equivalent to losing evidence, this is load-bearing.
Long migrations as background jobs. Schema upgrades that on other platforms mean a downtime window run in the background on Langfuse while the application keeps serving. Upgrade downtime drops drastically.

The officially supported deployment modes:

Docker Compose: for development and POCs. One command, everything up.
VM: a single node, containers, no orchestration. For small environments.
Kubernetes with Helm: the recommended mode for production. Official chart maintained. Supports external Postgres, external Clickhouse, external S3, HPA.

The external dependencies in a typical production setup: Postgres (metadata, prompts, configuration), Clickhouse (tracing events, high-cardinality queries), S3 or a compatible blob store (pending events), Redis (the queue between components). Yes, that is several pieces; it is what holds up the durability and the scale.

Prompt management as a first-class citizen

What sets Langfuse apart from platforms focused only on tracing is that the prompts live in Langfuse, not in the application repo or in spreadsheets. Each prompt has:

Name and version (v1, v2, v3…). Changing the prompt does not require redeploying the app: the app asks the SDK for the prompt, and the SDK caches it and refreshes when there is a new version.
Typed variables: {{user_input}}, {{context}}. Rendering with validation.
Tags and labels: by environment (production, staging), by team, by experiment.
Client and server cache: the SDK caches locally with a configurable TTL, avoiding a roundtrip to Langfuse on every call.
Linkage with traces: each trace records which exact version of which prompt was used. Investigating “this answer came out wrong” leads to prompt version Y, not to “some version of the prompt at some point”.

from langfuse import get_client

langfuse = get_client()

prompt = langfuse.get_prompt("rag-system-prompt", version=3)
# or by label: langfuse.get_prompt("rag-system-prompt", label="production")

compiled = prompt.compile(context=docs_text, user_input=question)
# 'compiled' is the final string, ready to send to the LLM

For teams iterating on prompts daily, this is what avoids the chaos of “which version of the prompt is actually running in production right now”.

Evaluations: four combinable evaluation models

Langfuse covers the four patterns of answer evaluation:

LLM-as-a-judge: you configure a model (typically GPT-4 or Claude) with a rubric and it evaluates each answer. Result: a numeric score (0-1) and a justification. Applicable to automatic tracing (all answers) or batch (a dataset selection).
User feedback: the app lets the user mark an answer as good/bad. The feedback is associated with the trace and the prompt version, which lets you see which versions have a worse rate.
Manual labeling: a UI where human labelers score answers. Useful for golden datasets and for evaluating the judge.
Custom evaluators through the API/SDK: your own evals (a unit test, a business metric) report a score through the API. They integrate with CI.

Combined, they give regression testing of the prompt: you change from v3 to v4, evaluate the golden dataset with LLM-as-judge, compare; if v4 gets worse in any of the segments, the merge fails.

Integrations

Langfuse does not compete with OpenLLMetry, LangChain or LiteLLM: it integrates them. The ones that are tested and documented:

OpenTelemetry: any OTel instrumentation emits to Langfuse over OTLP.
LangChain and LangGraph: a native callback that captures the whole chain.
LlamaIndex: native callback.
OpenAI SDK (Python and TS): a wrapper that adds tracing automatically.
LiteLLM: integration as a callback, which covers 100+ providers through LiteLLM.
OpenLLMetry / Traceloop: they emit to Langfuse like any OTel backend.
MLflow: through an OTel exporter from MLflow to Langfuse.
Vercel AI SDK: native instrumentation.

The strategy is clear: Langfuse is a backend, not an SDK. Your team chooses how it instruments; Langfuse accepts any path. The operational consequence: moving from Langfuse to another OTel backend tomorrow is viable.

When Langfuse is not the answer

So as not to present it as a silver bullet:

If you only use LangChain and have no resources to self-host: LangSmith will give you a smoother integration (it is the same team).
If your only need is a proxy with cost tracking and no evals: Helicone is simpler.
If you want an integrated commercial vendor solution: Datadog LLM Observability, New Relic AI Monitoring or Dynatrace AI are Enterprise alternatives with 24/7 support.
If your load is pure batch massive inference with no agents: you probably do not need semantic tracing; Prometheus + Grafana with OTel metrics is enough.

For everything else, your own apps with serious tracing, multi-tenant with quotas, teams iterating on prompts daily, RAG with continuous evaluation, Langfuse is the safe bet.

Quick choice summary:

LangChain → LangSmith (zero effort, automatic instrumentation).
Your own multi-framework applications with OSS → Langfuse (MIT, self-host, complete).
RAG with vector stores → Arize Phoenix (better retrieval visibility).
Simple proxy, low budget → Helicone.
Strict vendor neutrality → OpenLLMetry/Traceloop.
Pydantic AI → Logfire (same team).

Strengths and weaknesses of the instrumented model

Strengths:

Enormous depth: nested spans with all the context (chain steps, retrieval, embeddings, tool calls).
Semantic vocabulary: the SDK knows the domain (LLM, vector store, agent).
Maturity: three years of evolution, a rich ecosystem, ready-made dashboards.
Built-in evals: the top platforms combine tracing with evaluation (judge LLM, datasets, regression).

Weaknesses:

Requires control of the code: if you cannot instrument, it does not work.
Trust in the app: if the app reports badly or has a bug, so does the trace. It is not tamper-proof.
Coupling to the SDK: version changes in a library can break the instrumentation.
Uneven coverage: Python SDKs are mature; Go, Rust and JS are younger.

The zero-instrumentation approach: AgentSight

AgentSight is the project from the eunomia-bpf group that champions the opposite approach. Its paper on arxiv (2508.02736), presented at the Workshop on Practical Adoption Challenges of ML for Systems, formalises the proposal. The premise is direct:

Instead of instrumenting the agent, observe it at the system boundary.

And “system boundary” means the kernel boundary: the last point before a piece of data leaves the process towards the network or the filesystem. There, with eBPF, things are seen as they are, without the application being able to cooperate in hiding them.

Architecture: three planes

AgentSight assembles three layers:

Plane 1 — SSL/TLS uprobes. eBPF can attach programs to functions in userspace libraries (uprobes). The target functions are the encryption ones: SSL_write, SSL_read from OpenSSL/BoringSSL, and their equivalents in Rustls. AgentSight puts hooks on them that capture the arguments: the plaintext buffer the app passes to be encrypted, just before TLS processes it. On reception it does the symmetric thing: a hook after SSL_read with the freshly decrypted plaintext. Result: AgentSight sees the complete content of any HTTPS request the app makes without needing a man-in-the-middle, certificates or traffic decryption. The payload is plaintext because it was captured before being encrypted.

This works because uprobes are cheap (around 100 ns per invocation) and because apps use common TLS libraries. The few apps that implement their own TLS (rare in production) escape this hook; for those you need a different kprobe or manual instrumentation.

Plane 2 — Kernel events. In parallel, AgentSight observes relevant syscalls through tracepoints: execve (which processes start), connect/accept (network), read/write with file descriptors (filesystem and stdio), unlink, clone. Any action by the agent that has an effect outside the process passes through here. This covers, among other things, shell commands executed by the agent: if a Claude Code agent decides to run rm -rf to “clean up the project”, the execve is seen even if the LLM API does not report it.

Plane 3 — Correlation engine. The two previous planes produce streams of asynchronous events. AgentSight has a userspace component that correlates them causally cross-process: an outgoing HTTP request with bash -c rm -rf can be correlated with the previous LLM response that suggested it, through PIDs, timings and heuristics. The paper mentions the optional use of a secondary LLM (Anthropic Claude, for instance) that analyses the sequence of events and produces semantic alerts: “the agent responded with a tool call that was not in the whitelist”, “the reasoning chain has gone 47 iterations without converging”.

`stdiocap`: capturing the stdio of local MCP servers

One specific piece that deserves its own mention is stdiocap, a separate BPF tool included in the repo. The Model Context Protocol (MCP), popularised by Anthropic in 2024 and mainstream in 2025-2026, has two transport modes: HTTP/SSE (network) and stdio (between the client and the server it starts as a subprocess). Local MCP servers, the ones that run on the same machine and are started by the client as children over pipes, communicate through stdin/stdout/stderr with JSON-RPC.

stdiocap hooks read/write/dup on the stdin/stdout/stderr file descriptors of a target process and records all the JSON-RPC traffic between the MCP client and server. It is the same idea as the SSL capture but for stdio: you observe the conversation without either the client or the server knowing. Typical use case: seeing which tools of the filesystem-mcp MCP server a Claude Code agent has invoked in the last hour, what arguments it passed, what errors it received. Impossible with classic instrumentation (MCP servers are usually third-party binaries).

Guarantees: tamper-proof, kernel safety, <3% overhead

Three properties make AgentSight interesting for production:

Tamper-proof: the observation happens in the kernel (uprobes, syscalls). A malicious or compromised application cannot falsify what is seen. Compare with instrumentation: if the agent decides not to emit the span for its action, it does not appear in Langfuse. Here it has no choice.
Kernel safety: eBPF formally verifies that programs terminate and respect bounds checks. It cannot crash the kernel. The same as in the rest of the eBPF series.
<3% CPU overhead measured on real agent workloads (per the paper). The number compares favourably with SDK instrumentation, which typically adds 5-10% in intensive applications.

What it detects out of the box

The paper and the documentation highlight three classes of detection:

Real-time prompt injection: the correlation engine can apply rules or a detection model over the plaintext captured by the SSL uprobes. If the prompt contains suspicious patterns, “ignore all previous instructions”, a system prompt embedded in a user input, instructions to exfiltrate data, it raises an alert.
Reasoning loops that burn resources: agents that enter infinite loops calling tools without progressing. Detectable because the causal chain does not converge to a “final answer” and the tokens pile up. The correlation engine flags them.
Bottlenecks in multi-agent setups: when several agents coordinate, AgentSight sees the communication matrix between all of them and can detect agents that block while waiting, deadlocks, excessive fan-out.

The clash and the coexistence

The two families look like competitors, but in reality they see different things and complement each other in production.

What only the instrumented one sees

Internal agent variables that never reach the wire: the intermediate state of a LangChain chain, the values before passing them to a tool, how a prompt is built from a template with internal vars.
Deep semantic spans: retrieval > embed > vector_search > rerank > format_context > prompt_template > llm. AgentSight sees only the final call to the LLM; the path taken to build it is invisible.
Evaluations: scoring of answers, judge LLMs, quality regression. This lives only in instrumented platforms.

What only eBPF sees

Opaque binaries: Claude Code, Cursor, Gemini CLI, third-party agents. You do not have the code; you cannot instrument them. Only eBPF sees them.
System-level actions: the agent decides to run git push --force or kubectl delete. The action is seen in the execve. The agent’s instrumentation may not report it (especially if it was a command the agent generated as output without going through an explicit “tool”).
Tamper-proof audit: for regulatory compliance (HIPAA, SOC2, NIS2), having observation the app cannot dodge has formal value. eBPF provides it.
Local MCP servers over stdio: invisible to classic instrumentation unless each server emits its own spans (rare).

What both see, complementarily

Prompts and completions: the instrumented approach emits them with rich metadata; eBPF captures them off the wire. A perfect cross-check for spotting discrepancies.
Calls to external APIs: APM flags it; eBPF confirms it at kernel level.
Latency: APM per span; eBPF measures RTT at TCP level and network connectivity.

Decision matrix

Case	Instrumented	eBPF (AgentSight)
Your own app with LangChain	Yes, first	Optional
Your own multi-framework app	Yes	Optional
Third-party binary (Claude Code, Cursor)	Does not work	Yes, the only path
Tamper-proof regulatory compliance	Insufficient	Yes, required
Zero-trust multi-tenant	Insufficient	Yes, required
Local MCP servers (stdio)	Difficult	Yes, with stdiocap
Answer quality evaluation	Yes, required	No (out of scope)
Depth of the internal chain	Yes, required	No (a black box for AgentSight)
Reasoning loop detection	Possible with plumbing	Yes, built in
Real-time prompt injection	Possible (post-processed)	Yes, in stream

The natural conclusion: for your own apps, instrumented; for opaque binaries or compliance, eBPF; for everything that matters, both.

2026 reference architecture

Four recipes that cover the bulk of real cases:

Setup A — Your own application with LangChain or similar

Needs: depth, evals, a team comfortable with SDKs.

Langfuse self-host or LangSmith cloud as the backend.
OpenLLMetry SDK or LangSmith SDK instrumenting the code.
OpenTelemetry Collector between the app and the backend for routing flexibility (to Langfuse + Tempo + Loki, for instance).
Hubble for the network layer in the cluster (inter-pod latency, drop attribution).

Setup B — Productionising an opaque binary (Claude Code, Gemini CLI)

Needs: observe without touching, audit, control cost.

AgentSight deployed as a DaemonSet over the cluster (or standalone on the node).
Grafana with dashboards fed by AgentSight’s metrics.
An OTLP exporter from AgentSight to an OTel backend (Tempo, Jaeger). The spans will use the GenAI semantic conventions once they are fully standardised.
Tetragon optionally for policy over what the agent is allowed to run (Sigkill if it tries rm -rf or similar).

Setup C — Zero-trust multi-tenant platform

Needs: agents from different clients running on the same cluster, mandatory audit, nobody trusts anybody.

AgentSight as the tamper-proof audit layer. Compliance requires it.
Langfuse multi-tenant for the clients that do instrument.
Tetragon with TracingPolicyNamespaced per tenant (different policies per namespace).
Hubble with persistent flow logs for forensics.
Cilium NetworkPolicy to isolate tenants from each other on the network.

Setup D — A local MCP server on a workstation

Needs: seeing what an agent does with a stdio MCP server.

AgentSight stdiocap pointed at the PID of the client or the server.
Complete JSON-RPC capture to a file or to an OTLP endpoint.
Visualisation: Grafana, or simply jq over the log.

A real use case: if you are integrating an MCP server of your own and want to see what tool calls a Claude Code or Cursor agent makes to your server, stdiocap is the cleanest way. You need to modify neither client nor server.

Operational traps

Sensitive data in prompts (instrumented)

By default, Langfuse, LangSmith and similar tools capture the complete content of prompts and completions. If your app processes PII, secrets or medical data, that goes to your observability backend. Configuring redaction or content opt-out before going to production is mandatory. OTel GenAI has specific flags (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false) to avoid it.

Sensitive data in prompts (AgentSight)

The same problem, worse: AgentSight captures literally what goes to the wire, in plaintext. If the agent talked to api.openai.com with a prompt containing sensitive data, AgentSight has that plaintext. You have to encrypt or redact before storing.

Pinned certificates or non-standard TLS

Some high-security apps do certificate pinning or use unconventional TLS implementations (Go’s crypto/tls, custom BoringSSL). In those cases, the uprobes on libssl do not cover them. AgentSight detects when it cannot observe and reports the gap; you still have to add specific hooks for the alternative SDK.

Token volume and storage

An application with medium traffic can generate millions of tokens a day. If you store them all in Langfuse or Phoenix with long retention, the database grows fast. Strategies: aggressive sampling, short retention for normal sessions and long retention only for errors/anomalies, content redaction and keeping only metadata.

Tracing with sampling and consistency

To cut cost, many installations sample: only 1 in every N traces is persisted. Be careful with inconsistent sampling: a trace can carry several spans across multiple services, and if the sampling decision is taken per span, you end up with incomplete traces. OTel has head sampling (in the SDK, at the start), which is consistent, and tail sampling (in the collector, at the end), which allows finer rules. For LLM, tail sampling is ideal: sample everything, discard only the “normal” traces and keep the ones with errors, high latency or high cost.

Multi-agent and trace propagation

When agent A calls agent B, you have to propagate the trace context (W3C Trace Context headers) so it shows up as a single tree. If you do not, you see two disconnected traces. Modern platforms do it automatically with inject/extract, but if your transport between agents is custom (over Redis pub/sub, over a DB), you have to propagate by hand.

The cost of uprobes on critical libraries

Hooking libssl adds around 100 ns per invocation. On extreme TLS traffic loads (tens of thousands of connections/s per core), that adds up. AgentSight keeps it under 3% on typical agent workloads (which are chatty but not networking-intensive). If your use were sniffing all the HTTPS on the node, it could hurt more.

What we have not covered (next series)

Evals: the next layer after tracing. Phoenix, Langfuse, LangSmith and company offer answer evaluation (judge LLM, datasets, regression). It is a world of its own.
Guardrails and safety: NeMo Guardrails, Llama Guard, Llama Prompt Guard, evaluators specific to prompt injection and jailbreaks.
Deep MCP server observability: how the OpenTelemetry GenAI conventions are being extended to MCP servers for trace-aware tools.
eBPF + on-device inference: when the LLM runs locally through vLLM or llama.cpp, the uprobes can see the output token queue BEFORE it goes to the client. New territory.
Statistical analysis of agent flows: detecting drift, outliers, patterns that indicate degradation.

Closing the eBPF series

This series of four articles has travelled eBPF from first principles to the 2026 frontier:

eBPF from zero to Cilium — what eBPF is, networking hooks, how Cilium skips the TCP/IP stack, BGP Control Plane v2.
Tetragon: runtime security — observability and enforcement of processes in the kernel.
Hubble: network observability — L3-L7 flow logs and the frontier with AI agents.
This one — AgentSight, LLM tracing, instrumented vs zero-instrumentation.

If you have got this far you have the map to sit down with a platform, security or AI team in 2026 and recognise what each piece does, what problem it solves and where to start. That whole stack, Cilium for CNI and BGP, Tetragon for runtime security, Hubble for network observability, AgentSight for AI agents, sharing eBPF as a common substrate, Cloud Native governance and OpenTelemetry vocabulary. It is the clean architecture the industry asked for a decade ago and that finally exists.

References

AgentSight:

AgentSight GitHub (eunomia-bpf) — the project.
AgentSight: System-Level Observability for AI Agents Using eBPF (arxiv 2508.02736) — the formal paper.
AgentSight ACM workshop publication.
AgentSight blog post (eunomia.dev) — accessible description.

OpenTelemetry GenAI semantic conventions:

Instrumented platforms:

Langfuse — MIT, self-host + cloud.
LangSmith — LangChain team.
Arize Phoenix — OSS, OTel-native.
Helicone — simple proxy.
OpenLLMetry (Traceloop) — Apache 2.0, OTel SDK.
Pydantic Logfire — AI observability.

2026 comparisons:

Cross-references from the series:

eBPF from zero to Cilium.
Tetragon: runtime security.
Hubble: network observability.
LLM inference series: KV cache, vLLM on Kubernetes, PagedAttention, LLM Operators on K8s.

Agentsight on lo0 — Tech Blog