eBPF in local inference and statistical drift detection: closing the LLM observability loop in 2026

TL;DR

Tracing, evals, guardrails, MCP observability: the layers we have already covered see what is happening right now. What they do not see is what changes silently: the agent that answered well last week and this week, without anyone touching anything, answers slightly worse. What they also do not see is the fine mechanics of local inference: why a llama.cpp on an edge device takes 200 ms when it should take 100, which specific runtime function is the bottleneck. This post closes the two series of the week with the two layers that were missing: eBPF applied to local inference (uprobes on llama.cpp, vLLM, libcudart.so, integrated hardware perf counters, with <4% overhead, formalised in the ProfInfer 2026 paper, which is to local inference what Hubble is to the network) and statistical analysis of agent flows to detect drift before your user notices it (KS, PSI, MMD, embedding-space clustering, with Evidently AI, NannyML and WhyLabs as the dominant tools). The three types of LLM drift in 2026, prompt drift, model drift and eval-score drift, demand different tests. The complete stack (tracing, evals, guardrails, MCP observability, eBPF observability, drift detection) forms the loop that any serious agentic system needs in order to operate with a real SLA rather than with hope.

This post closes two series: the post-tracing series (Evals, Guardrails, MCP observability) and the eBPF series (eBPF from zero to Cilium, Tetragon, Hubble, AgentSight). It brings the two threads together: eBPF applied to the local inference engine plus statistical analysis of the flows that every layer produces.

The analogy: the agent’s cardiogram

A doctor who only looks at acute symptoms, the patient arrives with a high fever and you have to act, is practising reactive medicine. To practise preventive medicine, they need time series: blood pressure every year, cholesterol every six months, an ECG when there is a suspicion. It is not “right now” information, it is information about how something that should be stable is evolving. When a time series deviates from its baseline, you investigate before it turns into a high fever.

The LLM observability layers we have seen so far are reactive medicine: tracing tells you what happened in a specific conversation; evals tell you whether that conversation was good; guardrails tell you whether there was a specific threat; MCP observability tells you which tools were invoked and how. They all look at events, not trends.

Drift detection is preventive medicine. It looks at time series, of prompt embeddings, of evaluation scores, of distributions of generated tokens, and fires alerts when something drifts away from its normality. It does not tell you “this answer is bad”; it tells you “the distribution of prompts over the last 6 hours does not look like the distribution of the last month”. There you decide whether to investigate.

And the other half of the post, eBPF in local inference, is the equivalent of the MRI scanner: once you know there is a problem, it lets you see the inside of the model at a resolution no external wrapper gives you. To see which specific runtime function is slow, which CUDA kernel is the bottleneck, how tokens move through the internal buffers before going out to the client.

The two together close the loop: the time series detect that something is wrong, the scanner locates where.

Part 1 — eBPF applied to local inference

Why local inference changes the game

When the LLM runs locally, vLLM on a Kubernetes node, llama.cpp on an edge device, Ollama on a workstation, MLX on macOS, and not behind an external API, observability changes shape:

• You control the binary: you can attach hooks that would otherwise be impossible.
• The internal buffers exist in accessible RAM: the output-token stream, the logits, the KV caches, the scheduler structures are right there, at addresses a uprobe can read.
• There is no cable to sniff: AgentSight’s analogy with SSL hooks does not apply because there is no TLS; the model answers you with an in-process function return, not with an HTTPS response.
• The distance between kernel and model is minimal: the CUDA kernels running attention are one syscall deep; eBPF can observe both sides of that boundary with the same tracer.

This opens up a class of observability that is structurally impossible with LLM-as-a-service (the Anthropic, OpenAI or Vertex APIs). For apps serving inference on-premise or on-edge, an inference cluster, a mobile device, an RTX 4090 server in the rack, it is a new layer.

ProfInfer: the paper that formalises the pattern

ProfInfer (arxiv 2601.20755, 2026) is the reference academic piece that systematises what the ecosystem had been doing ad-hoc. The paper’s subtitle says it all: An eBPF-based Fine-Grained LLM Inference Profiler.

What it proposes:

• Attaching uprobes dynamically to runtime functions of engines such as llama.cpp (and by extension vLLM, Ollama). No recompiling, no modifying the source code. It is like bpftrace for LLM inference.
• Combining runtime events with hardware performance counters. A uprobe tells you when llama_decode runs; a hardware counter tells you how many floating-point instructions executed while it was inside. The correlation between the two is what gives the fine resolution.
• <4% measured overhead under real workloads. That is a production-grade cost.
• Visualisations in three views: operators (which tensor operations ran), graphs (how they relate) and timelines (when).

The paper focuses particularly on models on mobile platforms (Llama served on a Pixel or an iPhone), where classic observability with Prometheus and exported metrics barely exists. But the pattern applies to any local inference.

Where to hook: the map by engine

Now to the detail of the hooks. The target functions vary by engine:

llama.cpp

llama.cpp is pure C++, with symbols visible in the binary. The typical hooks:

• llama_decode: the function that runs one inference pass (processes the current batch). Spans for latency per iteration, tokens processed.
• llama_token_to_piece: converts a token ID to text. A hook here captures the stream of generated tokens before returning to the caller. It is the local equivalent of the SSL uprobes: you see the model’s output before it even reaches the consumer.
• llama_get_logits: reads the logits of the last decode. If you want to record the model’s probabilities (not just the chosen token), here.
• ggml_compute_forward_* (several functions): the operation kernels (matmul, attention, layernorm). Hooks for per-operation profiling.
• ggml_backend_*: the backend functions (CPU, Metal, CUDA, ROCm). Hooks here break down the cost per device.

An example with bpftrace:

# Latency and count of llama_decode
bpftrace -e '
uprobe:/path/to/llama-server:llama_decode {
    @start[tid] = nsecs;
}
uretprobe:/path/to/llama-server:llama_decode /@start[tid]/ {
    @decode_lat = hist((nsecs - @start[tid]) / 1000);
    delete(@start[tid]);
}
'

Output: a histogram of decode latencies in microseconds. Zero modification to the binary.

vLLM

vLLM is mostly Python. The C/CUDA symbols live in its native extensions (vllm._C, vllm._moe_C). The typical hooks:

• uprobes on vllm._C.* for custom operators (the paged attention kernel, the sampling kernel).
• uprobes on libcudart.so and libcuda.so to capture cudaMalloc, cudaLaunchKernel, cudaMemcpy. This maps the cost of host↔device transfers and kernel launches.
• Python tracepoints with bpftrace over usdt points: vLLM does not expose native static tracepoints, but they can be placed with USDT (dtrace style) at strategic points in the scheduler.

vLLM also exposes native Prometheus metrics (vllm:num_requests_running, vllm:gpu_cache_usage_perc, and so on). The added value of the eBPF approach is going down from the scheduler’s metrics to the individual functions: when a request is slow, seeing whether it was prefill, decode, scheduler overhead, transfer or synchronisation.

CUDA in general

Independently of the engine, uprobes on libcudart.so capture all the CUDA activity of the process:

• cudaMalloc(size) → tracking of device memory allocations.
• cudaLaunchKernel(func, ...) → spans for each kernel launch.
• cudaMemcpyAsync(dst, src, size, kind) → host↔device transfers.
• cudaStreamSynchronize(stream) → synchronisation points (where the host waits for the device).

This gives you a complete timeline of CUDA activity without needing NVIDIA Nsight Systems (which is excellent but heavy and oriented towards development, not continuous production).

Hardware counters: the other half

eBPF can read performance counters from the CPU/GPU’s PMU (Performance Monitoring Unit). That includes instructions executed, cache misses, branch mispredictions and, on GPUs with support, FLOPS, SM occupancy, HBM bandwidth.

Combining:

• uprobe: “llama_decode ran from T1 to T2 with tokens=4”.
• perf counter: “during that window, L2 cache misses = 15,000, instructions = 2.3 million”.

This lets you answer: why is it slow? Is it memory-bound (many cache misses), compute-bound (all the instructions in the FPU), bandwidth-bound (a lot of data movement)? State of the art for professional profiling.

Comparison with AgentSight

There are two eBPF products for LLMs today with different focuses:

• AgentSight (covered in the eBPF series): observes agents that call external APIs. It hooks SSL to see the plaintext of HTTPS calls to the remote LLM, plus stdio for local MCP servers. The client view.
• ProfInfer / the eBPF-on-local-inference pattern: observes the engine that runs the model locally. It hooks the engine’s internal functions (llama.cpp, vLLM) and the CUDA layer. The server (internal) view.

They are complementary. If your agent uses the Claude API plus your own local vLLM with Llama 3 for specific tasks, AgentSight sees the first, eBPF/ProfInfer sees the second. If everything is local, clearly the domain of the second. If everything is an external API, of the first.

Use cases for eBPF in local inference

Three cases where it is the right tool:

Fine-grained profiling for optimisation: your vLLM takes 50ms more per token than expected. With eBPF plus hardware counters you pin down which specific kernel it is. Previously this required Nsight Systems in a development session; now it is continuous in production.

Token-level observability without modifying the engine: capturing the stream of generated tokens before returning them to the client. Useful for auditing, for drift detection over the outputs, for local tracing without going through wrapper instrumentation.

Detecting specific degradation: a new version of vLLM introduces a subtle regression in paged attention. With perf counter baselines, you detect the change even if the external metric (tokens/sec) looks the same.

Part 2 — Statistical analysis of flows: detecting drift

Now to the other side of the problem: the time series.

Why tracing, evals and guardrails do not detect drift

The layers we have already seen operate on individual events:

• Tracing: a trace of one conversation.
• Evals: a score for one answer.
• Guardrails: a verdict on one prompt or answer.
• MCP observability: spans of one tool invocation.

Each one answers a point-in-time question (“is this all right?”). None of them answers the question of evolution (“is something changing over time?”).

The operational problem: drift is invisible in individual events. If the mean eval score drops from 0.92 to 0.85 over three weeks, no individual evaluation will raise an alarm, since they all continue to be “reasonable”. What changes is the distribution. And that only shows up when you look at many evaluations aggregated over time.

The three types of LLM drift in 2026

FutureAGI consolidates them like this, and the industry is converging on this vocabulary:

1. Prompt drift: someone updates the system prompt and the side effects break cases that used to work. Almost always intentional, but with unanticipated consequences. Detection: comparing response distributions before and after the change, monitoring eval scores per prompt version (linked in Langfuse, see the AgentSight post where we covered prompt management).

2. Model drift: the provider (OpenAI, Anthropic) updates the model without warning. The same prompt produces answers with a slightly different tone, similar but different quality, or degradation on some subset. Detection: comparing today’s response embeddings with the baseline; monitoring rubric scores; alerting if intra-model variance grows.

3. Eval-score drift: the rolling mean of your eval metrics (faithfulness, answer relevancy, custom rubrics) trends downwards. The root cause can be any of the above or a change in the user mix. Detection: alerts on the trends of the eval series.

To these three you can add a fourth, subtler one:

4. Persona drift / user mix shift: the population of users using the system changes. It is not that the model or the prompt got worse; it is that the new users ask different questions and the system, while still just as good at what it was good at, fails on the new material. Detection: embedding clustering of prompts, monitoring the appearance of new clusters or the growth of a minority one.

The key technical concept: embedding-space shift

Stack Pulsar puts it plainly: in LLMs, drift is best measured in embedding space. Classic distances in token space do not capture fine semantics; in embedding space they do.

The canonical pipeline:

1. Establish a baseline: during a stable period (say the first two weeks after a release), capture a large sample of prompt and response embeddings.
2. Continuous monitoring: every hour or every day, capture a new sample of production traffic.
3. Compare distributions: apply a statistical test that compares the current distribution with the baseline in embedding space.
4. Alert: if the divergence exceeds a threshold, fire an alert and an investigation workflow.

As a bonus, monitor clusters: if your baseline has 5 clusters of prompts (technical questions, general support, sales, and so on) and suddenly a sixth cluster appears that was not there, the most likely explanation is that a new user segment has arrived.

Statistical tests: KS, PSI, MMD

Three tests that any drift system uses, each with its place:

Kolmogorov-Smirnov (KS): non-parametric. It computes the maximum distance between two empirical CDFs. It returns a statistic and a p-value. Advantage: very sensitive to subtle changes, especially in the tails. Disadvantage: with large datasets it is “too sensitive”, firing alarms for changes that are real but clinically irrelevant.

Population Stability Index (PSI): you bin the reference distribution and the current one, then sum (p_actual - p_ref) × log(p_actual / p_ref) over the bins. The canonical interpretation: PSI < 0.1 stable, 0.1-0.25 mild drift, > 0.25 significant drift. Advantage: interpretable, threshold-based, with a tradition of use in credit scoring (Capital One, Goldman Sachs). Disadvantage: less sensitive than KS, so it misses drift in the tails.

Maximum Mean Discrepancy (MMD): measures the divergence between two distributions by embedding each one in a Hilbert space via a kernel. It works for complex multivariate distributions (high-dimensional embeddings). Advantage: the only one that scales reasonably to embeddings of 768/1024/4096 dimensions. Disadvantage: harder to interpret.

The recommended practice in 2026:

• PSI for simple features (prompt length, tokens, number of tools invoked).
• KS for continuous features where you want high sensitivity.
• MMD for embeddings (high-dimensional spaces).

Evidently’s analysis on real datasets showed that KS detects drift 6+ hours earlier than PSI in some incidents. The operational consequence: use KS for early warning, PSI for confirmation with an interpretable threshold.

Tools in 2026

Three products dominate the field:

Evidently AI

Evidently is open-source (Apache 2.0), Python-first. Its value:

• HTML drift reports: you generate a report comparing two datasets (reference vs current) and you get an HTML file with all the statistical tests, visualisations and conclusions. No server, no infrastructure; one shareable file.
• Native LLM support: on top of tabular, it supports text. It computes embeddings and applies the appropriate tests.
• 100+ metrics in the suite. It covers everything from a single framework.
• Integration with MLflow and kube: CI workflows with reports on every release.

from evidently import Report
from evidently.metrics import DataDriftPreset

ref = load_baseline_dataset()  # prompts from last week
cur = load_current_dataset()   # prompts from the last hour

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=ref, current_data=cur)
report.save_html("drift_report.html")

When this functionality detects drift, it also tells you which column and which test fired.

NannyML

NannyML has a different focus: estimating model performance when you have no ground truth. The techniques:

• CBPE (Confidence-Based Performance Estimation): estimates accuracy using the model’s confidence in its predictions.
• DLE (Direct Loss Estimation): estimates the loss directly.

Useful when your LLM app has no immediate human feedback but you want to know whether its quality has dropped. Apache 2.0, Python.

WhyLabs

WhyLabs is commercial (with whylogs as the underlying OSS library), aimed at enterprise production:

• Managed SaaS with SOC 2 Type 2 and HIPAA compliance.
• Real-time monitoring via continuous log ingestion.
• Embedding tracking: native support for embedding distributions, not just tabular features.
• Token probability shifts: it monitors the probability distribution of generated tokens, not just metadata.

For regulated companies that do not want to operate their own drift detection platform, it is the lowest-friction option.

Other mentions

Arize Phoenix (seen in the Evals post) includes drift detection as a module. Galileo has commercial products specialised in LLM monitoring. Fiddler AI and Alibi Detect (Seldon) are more generalist alternatives that also cover LLMs.

Part 3 — The complete stack: how everything fits

Let us recap the layers the two series have covered, ordered from closest to the individual request to closest to the aggregate trend:

Individual EVENTS                             Aggregated TRENDS
       │                                                  │
  Tracing  ──→  Evals  ──→  Guardrails  ──→  MCP obs  ──→  Drift detection
       │                                                  │
  AgentSight ──→ Tetragon ──→ Hubble  ──→  eBPF on-device
       │                                                  │
  (what happens)                                   (what changes)

Each layer answers a different question:

None of them replaces the others. Full coverage requires all seven. The practical operation:

1. Layers 1-3 (tracing, evals, guardrails) are mandatory from day one. Any LLM app in production that does not have them is flying blind.
2. Layer 4 (MCP) becomes mandatory when there are agents with tools, which is the majority in 2026.
3. Layers 5-6 (eBPF) become valuable when the scale justifies the operating cost (>10 services, >100 inference pods).
4. Layer 7 (drift) is the one that is most neglected and most expensive to ignore: it takes a day’s work to get the basic pipeline and it saves weeks of future incidents.

The operational drift pattern in 2026

The minimum recipe any serious LLM app should have:

Step 1 — Establish a baseline

During a stable post-release period (2 weeks minimum), store:

• Embeddings of every prompt (vector + metadata: timestamp, user_segment, tenant).
• Embeddings of the responses.
• Automated eval scores over a sample (e.g. 5-10% of traffic with G-Eval).
• The distribution of tools invoked (which tools, with which typical arguments, at what frequency).

Storage: any vector store plus a relational one. Reasonable cardinality at whatever scale you have.

Step 2 — Continuous comparison pipeline

Every hour (or every day depending on scale):

• Take the sample for the current period (the last hour).
• Apply the statistical tests against the baseline:
  • PSI over simple features (prompt length, tokens, num tools).
  • KS over continuous features (latency, score).
  • MMD over embeddings.
• Generate a drift report (Evidently does it in one line of Python).

Step 3 — Alerts and investigation workflow

Configure thresholds and routes:

• PSI > 0.25 over tokens consumed: moderate alert (it may be legitimate, investigate segments).
• Significant MMD over prompt embeddings: high alert (a change in user mix or a coordinated attack).
• Eval rubric score down >5% over a rolling 7d: critical alert.
• A new cluster in embedding space accounting for 10%+ of traffic: a review workflow (it may be a legitimate new segment or an anomaly).

Every alert must lead to a drill-down dashboard with the affected segments, not to an empty Slack message. The operational rule: if someone cannot investigate the alert in <5 minutes, it will not get investigated.

Step 4 — Baseline refresh

The baseline is not static. Every N weeks, refresh the baseline by incorporating the “new stable”. If over 3 months the usage pattern has legitimately changed (more international users, new languages), the baseline must reflect it. The typical cadence: quarterly.

Operational traps

A contaminated baseline

You take the baseline from a period that already contained the problem in embryo. Result: the baseline includes the bad behaviour and the tests never fire. Solution: verify the baseline against a second independent sample (for example, the first week vs the second) before blessing it.

A threshold that is too low

PSI > 0.05 fires constantly. Your team learns to ignore the alerts. Calibrate thresholds against the natural noise of your system: run the system with the baseline plus successive weekly samples and measure the PSI distribution; set the threshold a couple of deviations above normal.

Embeddings that are not representative

You use OpenAI’s text-embedding-3-small embedding model to detect drift in a system serving technical questions in Spanish about Cisco networks. Result: the embedding model does not capture the fine semantics of the domain. Solution: use embeddings finetuned for your domain, or one that is strong in multilingual and technical material.

Storage overload

Storing an embedding of every prompt in production at scale (millions of prompts/day) fills disks and increases cost. Stratified sampling: keep 5-10% of traffic, but make sure minority segments are over-represented so you do not lose them.

Confusing drift with “the system works”

Sometimes drift is good drift: new users discover the agent knows how to do X, and suddenly 30% of traffic is for X. The distribution changed because the product found a new use. Before pulling the alarm, check whether the change is desirable.

Privacy in embedding storage

Embeddings can be partially inverted back to their original text with embedding inversion techniques. If the prompts contain PII, storing embeddings for months for drift detection is a leak vector. Encrypt at rest and rotate regularly, or work with aggregated/averaged embeddings.

eBPF in production without profile guardrails

Attaching uprobes to hot-path functions such as llama_decode can hit throughput if it is not done carefully. Always test in staging and monitor the overhead. ProfInfer reports <4%; what you measure may vary with your binary and kernel.

Closing the two series

This week we have written 12 articles that walk the modern LLM inference stack in production from top to bottom:

LLM inference series (4 articles):

• KV cache: the working memory that holds up LLM inference — fundamentals.
• vLLM on Kubernetes — the engine.
• PagedAttention deep dive — how it works inside.
• LLM Operators on Kubernetes — orchestration.

eBPF series (4 articles):

• eBPF from zero to Cilium — the substrate.
• Tetragon — runtime security.
• Hubble — network observability.
• AgentSight — agent observability.

Post-tracing series (4 articles):

• Evals — reactive quality.
• Guardrails — preventive safety.
• MCP observability — the tool protocol.
• This one — drift detection and eBPF in local inference.

If you read the twelve in order you have a reasonably complete map of what it takes to operate AI agents in serious production in 2026, with enough detail not to crash into the usual problems in the first month. And, above all, with the mindset that LLM observability is a stack, not a product: each layer solves a problem, none of them solves all of them, and the combination is what separates an operable system from one that holds up until the first incident.

What is left for future series

• MLOps specific to LLMs: continuous fine-tuning, RAG over data lakes, agent training.
• Constitutional AI and alignment at runtime: how the model self-regulates with internal guardrails.
• GPU networking: InfiniBand, NCCL, GPUDirect, the angle we left untouched.
• Edge inference: llama.cpp on phones, MLX on macOS, Snapdragon NPU.
• Theoretical inference scheduling: CFS-like algorithms applied to multi-tenant LLM serving.

We will cover them in time. That is it for now, and thanks for reading these twelve posts. If they gave you something, share them with a colleague.

References

eBPF in local inference:

• ProfInfer: An eBPF-based Fine-Grained LLM Inference Profiler (arxiv 2601.20755) — the 2026 reference paper.
• Monitor LLM Inference in Production 2026 (Glukhov) — Prometheus + Grafana for vLLM/TGI/llama.cpp.
• AI Inference Server Observability in Kubernetes (ARMO) — the four signals MLOps tools do not capture.
• vLLM vs llama.cpp: Choosing the right engine (Red Hat).

Drift detection concepts:

• What is LLM Drift? Types, Detection, Mitigation 2026 (FutureAGI).
• LLM Model Drift Detection 2026 (Stack Pulsar).
• 9 Best LLM Drift Monitoring Platforms in 2026 (Galileo).
• Open-Source Drift Detection Tools in Action (arxiv 2404.18673).
• Detecting covariate drift in text data using document embeddings (arxiv 2309.10000).
• Best AI Drift Detection Tools in 2026 (FutureAGI).

Tools:

• Evidently AI (GitHub) — open-source.
• Evidently — official site.
• NannyML — performance without ground truth.
• WhyLabs — managed observability.
• Alibi Detect (Seldon) — general drift detection.
• Arize Phoenix — drift integrated with tracing.

Statistical tests:

• Data drift detection: PSI vs Kolmogorov–Smirnov (MLPipeline) — a practical comparison.
• Population Stability Index for Model Drift Detection.
• Which test is the best? 5 methods to detect data drift (Evidently) — the 6 hours of advantage for KS.

Cross-references (the three complete series):

• LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators.
• eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.
• Post-tracing series: Evals, Guardrails, MCP observability.