Llama-Cpp on lo0 — Tech Blog

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

Wed, 27 May 2026 11:30:00 +0200

This post complements KV cache: the working memory of LLM inference, where cache quantisation is mentioned as a savings lever; here we go into the whole method, model weights and cache, and into why each format does what it does.

TL;DR

Quantising is a change of representation: instead of storing every model weight as a float16 or bfloat16 (2 bytes), it is stored as a short integer (1 byte INT8, half a byte INT4) with a scale factor that reconstructs a value close to the original. The price is loss of numerical precision; the reward is 2-4× less VRAM, 2-3× more throughput and, on Hopper and Blackwell, a radically lower compute cost because FP8/FP4 units execute in fewer cycles than BF16 ones. The four dominant formats in May 2026 are FP8 (E4M3/E5M2, datacenter), INT4 GPTQ (Hessian-aware reconstruction), INT4 AWQ (activation-aware) and GGUF (the llama.cpp family). Each has a sweet spot: FP8 when the datacenter is Hopper/Blackwell and quality matters; GPTQ and AWQ when serving runs on Ampere/Ada and 4 bits are mandatory; GGUF when the target is edge or a consumer GPU. This post explains the minimum maths, the algorithms behind each format, what each one loses measured in perplexity and MMLU, and how it all applies on a 4090 versus an H100 cluster.

You are here: DEPLOY

The analogy: JPEG with an edge detector

A JPEG compresses an image by reducing the precision with which pixels are stored, but it does not reduce it uniformly. Where there is flat sky, thousands of very similar pixels, it throws away detail without anyone noticing. Where there is a sharp edge, the outline of a face, it keeps the fidelity. The trick is to detect which parts are sensitive before compressing.

Quantization of an LLM works the same way. You do not take every model weight and say “all of them in 4 bits”. Some weights are very important, projection weights that move the output a lot when they change, and others are less so. Modern techniques (GPTQ, AWQ) are basically edge detectors: they identify which weights can be quantised aggressively and which need more bits or special treatment, and they apply the quantisation with that information.

The analogy holds up in three details:

Calibration with a small dataset = the phase in which the JPEG encoder analyses the image before choosing blocks.
A block of 128 weights with a shared scale = the JPEG 8×8 block with its DCT.
Outliers preserved with more precision = the high frequencies of an edge are preserved more than the flat areas.

From there on, what follows is the maths and the operational detail.

The minimum maths: scale and zero-point

Quantising a weight vector w ∈ ℝ^n (in BF16) to INT4 means finding two things:

A scale s ∈ ℝ (in BF16 or FP16).
For each weight, an integer code q ∈ {0, 1, ..., 15} that fits in 4 bits.

And an approximate reconstruction formula:

$$\hat{w}_i \approx s \cdot (q_i - z)$$

where z is the zero-point (the integer that defines which code represents the original zero). The zero-point exists in asymmetric INT4/INT8 so that half the range is not wasted on negative values when the weight distribution is not symmetric.

The choice of s and z for a block of weights w_block:

$$s = \frac{\max(w_\text{block}) - \min(w_\text{block})}{2^{b} - 1}, \quad z = -\frac{\min(w_\text{block})}{s},$$

with b = number of bits (4 in INT4). Encoding:

$$q_i = \text{clip}\!\left(\text{round}\!\left(\frac{w_i}{s} + z\right),\, 0,\, 2^b - 1\right).$$

And decoding at inference time:

$$\hat{w}_i = s \cdot (q_i - z).$$

Numerical example

Take 8 real weights from a linear layer: w = [0.31, -0.12, 0.78, -0.05, 1.42, -0.91, 0.23, 0.66]. We want to quantise to INT4 (16 levels).

max = 1.42, min = -0.91. Range = 2.33.

$$s = \frac{2.33}{15} \approx 0.1553, \quad z = -\frac{-0.91}{0.1553} \approx 5.86 \to 6.$$

Encoding of each weight:

`w_i`	`w_i/s + z`	`round`	`q_i`	`ŵ_i = s·(q-z)`	error
0.31	8.00	8	8	0.311	+0.001
-0.12	5.23	5	5	-0.155	-0.035
0.78	11.02	11	11	0.776	-0.004
-0.05	5.68	6	6	0.000	+0.050
1.42	15.14	15	15	1.398	-0.022
-0.91	-0.06	0	0	-0.932	-0.022
0.23	7.48	7	7	0.155	-0.075
0.66	10.25	10	10	0.621	-0.039

Mean squared error: MSE ≈ 0.0015. For a single layer with millions of weights, the aggregate effect is what calibration tries to minimise.

Storage: instead of 8 values × 2 bytes = 16 bytes (BF16), we have 8 × 4 bits = 4 bytes + 2 bytes (BF16 scale) + 0.5 byte (INT4 zero-point) ≈ 6.5 bytes. 2.5× less, yet the useful data is still recoverable with a small error.

PTQ vs QAT: when quantisation happens

Two different operational regimes.

Post-Training Quantization (PTQ) is applied after training, on a model already trained in BF16/FP16. It reads a small dataset (typically 128-512 examples) to calibrate the scales, runs the quantisation algorithm (GPTQ, AWQ, and so on) and produces the quantised weights. Cost: minutes to a few hours. Typical loss: 0.05-0.3 PPL in perplexity (~0.5-2 % in MMLU) for INT4 with modern methods.

Quantization-Aware Training (QAT) introduces the quantisation operations inside the training loop. During training the model “sees” that its weights get quantised and learns to be robust to it. Cost: retraining the model (expensive), but little is needed, a short fine-tune on top of the already quantised PTQ model. Typical loss: ~0 (quantisation becomes indistinguishable from the original model).

When to use which:

PTQ = default. 90 % of production cases. The model arrives in BF16, you quantise it in 1-2 hours on one GPU, you deploy it.
QAT = when PTQ loses too much and the difference matters (typical case: INT2/INT3, or sensitive models such as specific reasoning ones).

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

FP8 is not “INT8 + sign”: these are two 8-bit floating point formats.

E4M3 (4 exponent bits, 3 mantissa bits): range ±448, reasonable precision around ±1.0. Typically used for weights and for activations in most layers.
E5M2 (5 exponent, 2 mantissa): range ±57,344, lower precision. Used for gradients during training or for activations with large outliers at inference time.

Why FP8 left INT8 behind in the datacenter: the tensor cores on H100/H200/B200 execute FP8 operations natively with 2× the throughput of BF16 and 4× that of FP16. And because FP8 preserves logarithmic dynamics (just like FP16), matrices with values spread out in magnitude, typical of transformers, quantise with less error than with INT8.

The loss measured in production is minimal: for a Llama 3.1 70B FP8 versus BF16, perplexity rises by ~0.03 and MMLU drops by ~0.5 points. It is the default option in any modern deployment on H100/B200.

Microscaling: NVFP4 and MXFP4

Blackwell (B100/B200, 2025) introduces NVFP4 and MXFP4, 4-bit formats with small-block scaling (typically 16 or 32 elements per scale, against 128 in INT4 GPTQ). The scale is FP8 instead of FP16/BF16, which reduces storage further.

The result: 4 bits with quality close to FP8. In 2026 NVFP4 is becoming the default option for very large models (200B+) on Blackwell clusters. For 4×H100 SXM, Hopper rather than Blackwell, FP8 remains the sweet spot.

INT4: GPTQ vs AWQ

The two algorithms that dominate 4-bit quantisation solve the same problem with different strategies.

GPTQ (Frantar et al. 2022)

The idea: layer-by-layer quantisation, explicitly minimising the error at the output of each linear layer using information from the Hessian matrix (the second derivative of the loss). For each layer:

Estimate the Hessian H = X^T X where X are the calibration activations.
Quantise one weight at a time in order (typically the most sensitive one first).
Update the remaining weights to compensate for the error of the weight just quantised.

Step 3 is what makes GPTQ better than naive round-to-nearest: weights compensate for their neighbours’ errors. The official implementation quantises a Llama 3 70B in ~3-4 hours on one H100 with 128 calibration samples.

AWQ (Lin et al. 2023)

The AWQ observation: within a layer, not all channels (weight columns) are equally important. Roughly 1 % of the channels account for most of the impact on the activations. AWQ identifies them by measuring the mean magnitude of the activations that multiply them, and scales them before quantising so they are better preserved.

Concretely: if a channel c has large mean activations, AWQ multiplies that channel’s weights by a factor s_c before quantising, and when quantising the input to the next layer it divides by s_c. The maths cancels out, but the important weights end up with more resolution inside the INT4 range. No retraining, no Hessian, faster than GPTQ (~1-2 h for 70B on an H100).

Which one to choose

Criterion	GPTQ	AWQ
Quantisation speed	Slower	Faster
Quality preserved in INT4	Good	Slightly better (~0.05-0.1 PPL)
Supported hardware	Broad (Ampere+)	Broad (Ampere+)
Ecosystem	Mature, widely integrated	More recent, gaining ground
Default in 2026	When GPTQ artefacts already exist	Default for new quantisations

The practical rule in May 2026: AWQ by default for new INT4. GPTQ when a GPTQ-Int4 artefact published by the community already meets your requirements.

GGUF: the llama.cpp ecosystem

GGUF is not a quantisation algorithm, it is a file format, and a whole tooling ecosystem, around the llama.cpp runtime.

Its value: universal compatibility. The same GGUF file runs on:

Pure CPU (Intel/AMD x86, ARM).
Apple Silicon (M1/M2/M3/M4) with Metal acceleration.
Consumer GPUs (RTX, AMD Radeon) with layer offload to VRAM.
Edge devices (Jetson, ARM phones).

That is what llama.cpp allows and vLLM/TensorRT-LLM do not. The trade-off: lower peak throughput on datacenter GPUs than vLLM.

The most used GGUF sub-formats in 2026:

Sub-format	Effective bits	Relative quality	Typical use
`Q8_0`	8.5	Almost lossless	Baseline validation
`Q6_K`	6.6	Very small loss	High quality + savings
`Q5_K_M`	5.7	Small loss	Quality/size sweet spot
`Q4_K_M`	4.8	Moderate loss	Consumer default
`Q4_K_S`	4.5	Moderate-to-high loss	When Q4_K_M does not fit
`Q3_K_M`	3.9	Noticeable loss	Very constrained hardware
`Q2_K`	3.3	Large loss	Last resort

The _K_M suffix indicates the degree of mixing: inside the file, certain layers (typically attention.wv, feed_forward.w2) are stored with more bits than others. It is the equivalent of the JPEG “edge detector” applied layer by layer with a pre-established heuristic.

KV cache quantization

Quantising the model weights is half the problem. The KV cache, covered in detail in KV cache: the working memory, typically consumes 20-50 % of VRAM in production with concurrency. Quantising the cache is a lever too:

--kv-cache-dtype=auto (BF16/FP16, the default). 2 bytes per dimension × num_heads × head_dim × 2 (K and V).
--kv-cache-dtype=fp8 (E4M3 or E5M2 depending on hardware). 1 byte. Halves the cache with a typical loss of < 0.5 % on quality benchmarks.
--kv-cache-dtype=int4 (with blocks of 128). 0.5 bytes plus scale overhead. Divides the cache by ~3.5. Measurable loss (1-2 %) but acceptable with long contexts.

KV cache quantisation is orthogonal to weight quantisation: you can have BF16 weights and an FP8 cache, or INT4 weights and an FP8 cache, and so on. The dominant combination in 2026 on H100: FP8 weights + FP8 cache, which is almost indistinguishable from BF16 in quality and doubles concurrency capacity.

Measured quality loss (Llama 3.1 70B Instruct, reference)

A representative table for Llama 3.1 70B Instruct with a WikiText-2 calibration dataset (128 samples). Figures aggregated from public sources; they may vary by ±0.05 PPL and ±0.5 MMLU depending on implementation and seed.

Format	Model VRAM	Perplexity (WikiText-2)	MMLU (5-shot)	Relative speed (H100)
BF16 (baseline)	140 GB	4.85	82.1	1.00×
FP8 (E4M3)	70 GB	4.87 (+0.02)	81.6 (-0.5)	1.85×
INT8 SmoothQuant	70 GB	4.92 (+0.07)	81.0 (-1.1)	1.65×
INT4 AWQ	35 GB	4.99 (+0.14)	80.4 (-1.7)	2.50×
INT4 GPTQ	35 GB	5.05 (+0.20)	80.0 (-2.1)	2.40×
GGUF Q5_K_M	49 GB	4.94 (+0.09)	81.1 (-1.0)	n/a (llama.cpp)
GGUF Q4_K_M	42 GB	5.08 (+0.23)	79.8 (-2.3)	n/a (llama.cpp)
GGUF Q3_K_M	33 GB	5.45 (+0.60)	77.5 (-4.6)	n/a (llama.cpp)

Three lessons worth keeping:

FP8 is almost free in quality terms. If your hardware supports it, there is no debate.
INT4 AWQ is noticeably better than INT4 GPTQ in preserved quality, at comparable speed.
Q3 is already in measurable-loss territory; Q2 should no longer be used except for experiments or extreme demos.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

Llama 3.1 8B Instruct fits comfortably in BF16 (16 GB), but that leaves little headroom for a KV cache under concurrency. The usual sweet spot:

Llama 3.1 8B AWQ-INT4: ~5 GB of weights, 19 GB free for the KV cache → 4-8 concurrent sessions with a moderate context.
Llama 3 70B GGUF Q4_K_M: ~42 GB. It does not fit on a whole 4090; it requires CPU offload with llama.cpp (slow decode but workable for a single user).
Llama 3 70B AWQ-INT4 with TP=2 (two 4090s): ~17 GB per GPU → it fits and leaves headroom.

The 4090 does not support native FP8 (Ada Lovelace has the instruction but not Hopper’s accelerated throughput). In practice, FP8 on a 4090 works but without the speed gain: the sensible choice is INT4 AWQ.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here FP8 shines:

Llama 3.1 70B FP8 with TP=2: ~35 GB/GPU. Comfortable, leaving a huge amount of room for the KV cache → dozens of concurrent sessions.
Llama 3.1 405B FP8 with TP=4: ~200 GB/GPU. It just fits, with prefill+decode in the same pool.
Llama 3.1 405B INT4 AWQ with TP=2: ~100 GB/GPU. It allows serving the large model without saturating the cluster; there is headroom left for the cache and for serving another model at the same time.

The rule of thumb on an H100 cluster in 2026: FP8 if quality matters and the model fits; INT4 AWQ if the model does not fit in FP8 or if you want more concurrency at the cost of 1-2 MMLU points.

What we have not covered (upcoming articles)

Speculative decoding: the other big acceleration lever in inference. Orthogonal to quantization, it multiplies the speedup.
MoE quantization: Mixture-of-Experts models (Mixtral, DeepSeek V3, Qwen3-235B-A22B) have different quantisation patterns, experts are not quantised uniformly and there is dynamic routing.
Calibration dataset matters: how to choose the 128-512 calibration samples. The common mistake of grabbing a random dataset off the internet, and how to avoid it.
Multimodal quantization: vision-language models have heterogeneous layers (a CNN vision encoder, a transformer language model) that need separate treatment.

References

Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023).
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024).
Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023). Introduces NF4 and double quantization.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023).
NVIDIA. FP8 Formats for Deep Learning — E4M3/E5M2 white paper: https://arxiv.org/abs/2209.05433.
Rouhani, B. et al. Microscaling Data Formats for Deep Learning — MXFP4/MXFP8: https://arxiv.org/abs/2310.10537.
llama.cpp GGUF spec: https://github.com/ggerganov/llama.cpp/blob/master/docs/gguf.md.
vLLM quantization docs: https://docs.vllm.ai/en/latest/quantization/.

eBPF in local inference and statistical drift detection: closing the LLM observability loop in 2026

Wed, 20 May 2026 09:00:00 +0200

TL;DR

Tracing, evals, guardrails, MCP observability: the layers we have already covered see what is happening right now. What they do not see is what changes silently: the agent that answered well last week and this week, without anyone touching anything, answers slightly worse. What they also do not see is the fine mechanics of local inference: why a llama.cpp on an edge device takes 200 ms when it should take 100, which specific runtime function is the bottleneck. This post closes the two series of the week with the two layers that were missing: eBPF applied to local inference (uprobes on llama.cpp, vLLM, libcudart.so, integrated hardware perf counters, with <4% overhead, formalised in the ProfInfer 2026 paper, which is to local inference what Hubble is to the network) and statistical analysis of agent flows to detect drift before your user notices it (KS, PSI, MMD, embedding-space clustering, with Evidently AI, NannyML and WhyLabs as the dominant tools). The three types of LLM drift in 2026, prompt drift, model drift and eval-score drift, demand different tests. The complete stack (tracing, evals, guardrails, MCP observability, eBPF observability, drift detection) forms the loop that any serious agentic system needs in order to operate with a real SLA rather than with hope.

This post closes two series: the post-tracing series (Evals, Guardrails, MCP observability) and the eBPF series (eBPF from zero to Cilium, Tetragon, Hubble, AgentSight). It brings the two threads together: eBPF applied to the local inference engine plus statistical analysis of the flows that every layer produces.

The analogy: the agent’s cardiogram

A doctor who only looks at acute symptoms, the patient arrives with a high fever and you have to act, is practising reactive medicine. To practise preventive medicine, they need time series: blood pressure every year, cholesterol every six months, an ECG when there is a suspicion. It is not “right now” information, it is information about how something that should be stable is evolving. When a time series deviates from its baseline, you investigate before it turns into a high fever.

The LLM observability layers we have seen so far are reactive medicine: tracing tells you what happened in a specific conversation; evals tell you whether that conversation was good; guardrails tell you whether there was a specific threat; MCP observability tells you which tools were invoked and how. They all look at events, not trends.

Drift detection is preventive medicine. It looks at time series, of prompt embeddings, of evaluation scores, of distributions of generated tokens, and fires alerts when something drifts away from its normality. It does not tell you “this answer is bad”; it tells you “the distribution of prompts over the last 6 hours does not look like the distribution of the last month”. There you decide whether to investigate.

And the other half of the post, eBPF in local inference, is the equivalent of the MRI scanner: once you know there is a problem, it lets you see the inside of the model at a resolution no external wrapper gives you. To see which specific runtime function is slow, which CUDA kernel is the bottleneck, how tokens move through the internal buffers before going out to the client.

The two together close the loop: the time series detect that something is wrong, the scanner locates where.

Part 1 — eBPF applied to local inference

Why local inference changes the game

When the LLM runs locally, vLLM on a Kubernetes node, llama.cpp on an edge device, Ollama on a workstation, MLX on macOS, and not behind an external API, observability changes shape:

You control the binary: you can attach hooks that would otherwise be impossible.
The internal buffers exist in accessible RAM: the output-token stream, the logits, the KV caches, the scheduler structures are right there, at addresses a uprobe can read.
There is no cable to sniff: AgentSight’s analogy with SSL hooks does not apply because there is no TLS; the model answers you with an in-process function return, not with an HTTPS response.
The distance between kernel and model is minimal: the CUDA kernels running attention are one syscall deep; eBPF can observe both sides of that boundary with the same tracer.

This opens up a class of observability that is structurally impossible with LLM-as-a-service (the Anthropic, OpenAI or Vertex APIs). For apps serving inference on-premise or on-edge, an inference cluster, a mobile device, an RTX 4090 server in the rack, it is a new layer.

ProfInfer: the paper that formalises the pattern

ProfInfer (arxiv 2601.20755, 2026) is the reference academic piece that systematises what the ecosystem had been doing ad-hoc. The paper’s subtitle says it all: An eBPF-based Fine-Grained LLM Inference Profiler.

What it proposes:

Attaching uprobes dynamically to runtime functions of engines such as llama.cpp (and by extension vLLM, Ollama). No recompiling, no modifying the source code. It is like bpftrace for LLM inference.
Combining runtime events with hardware performance counters. A uprobe tells you when llama_decode runs; a hardware counter tells you how many floating-point instructions executed while it was inside. The correlation between the two is what gives the fine resolution.
<4% measured overhead under real workloads. That is a production-grade cost.
Visualisations in three views: operators (which tensor operations ran), graphs (how they relate) and timelines (when).

The paper focuses particularly on models on mobile platforms (Llama served on a Pixel or an iPhone), where classic observability with Prometheus and exported metrics barely exists. But the pattern applies to any local inference.

Where to hook: the map by engine

Now to the detail of the hooks. The target functions vary by engine:

llama.cpp

llama.cpp is pure C++, with symbols visible in the binary. The typical hooks:

llama_decode: the function that runs one inference pass (processes the current batch). Spans for latency per iteration, tokens processed.
llama_token_to_piece: converts a token ID to text. A hook here captures the stream of generated tokens before returning to the caller. It is the local equivalent of the SSL uprobes: you see the model’s output before it even reaches the consumer.
llama_get_logits: reads the logits of the last decode. If you want to record the model’s probabilities (not just the chosen token), here.
ggml_compute_forward_* (several functions): the operation kernels (matmul, attention, layernorm). Hooks for per-operation profiling.
ggml_backend_*: the backend functions (CPU, Metal, CUDA, ROCm). Hooks here break down the cost per device.

An example with bpftrace:

# Latency and count of llama_decode
bpftrace -e '
uprobe:/path/to/llama-server:llama_decode {
 @start[tid] = nsecs;
}
uretprobe:/path/to/llama-server:llama_decode /@start[tid]/ {
 @decode_lat = hist((nsecs - @start[tid]) / 1000);
 delete(@start[tid]);
}
'

Output: a histogram of decode latencies in microseconds. Zero modification to the binary.

vLLM

vLLM is mostly Python. The C/CUDA symbols live in its native extensions (vllm._C, vllm._moe_C). The typical hooks:

uprobes on vllm._C.* for custom operators (the paged attention kernel, the sampling kernel).
uprobes on libcudart.so and libcuda.so to capture cudaMalloc, cudaLaunchKernel, cudaMemcpy. This maps the cost of host↔device transfers and kernel launches.
Python tracepoints with bpftrace over usdt points: vLLM does not expose native static tracepoints, but they can be placed with USDT (dtrace style) at strategic points in the scheduler.

vLLM also exposes native Prometheus metrics (vllm:num_requests_running, vllm:gpu_cache_usage_perc, and so on). The added value of the eBPF approach is going down from the scheduler’s metrics to the individual functions: when a request is slow, seeing whether it was prefill, decode, scheduler overhead, transfer or synchronisation.

CUDA in general

Independently of the engine, uprobes on libcudart.so capture all the CUDA activity of the process:

cudaMalloc(size) → tracking of device memory allocations.
cudaLaunchKernel(func, ...) → spans for each kernel launch.
cudaMemcpyAsync(dst, src, size, kind) → host↔device transfers.
cudaStreamSynchronize(stream) → synchronisation points (where the host waits for the device).

This gives you a complete timeline of CUDA activity without needing NVIDIA Nsight Systems (which is excellent but heavy and oriented towards development, not continuous production).

Hardware counters: the other half

eBPF can read performance counters from the CPU/GPU’s PMU (Performance Monitoring Unit). That includes instructions executed, cache misses, branch mispredictions and, on GPUs with support, FLOPS, SM occupancy, HBM bandwidth.

Combining:

uprobe: “llama_decode ran from T1 to T2 with tokens=4”.
perf counter: “during that window, L2 cache misses = 15,000, instructions = 2.3 million”.

This lets you answer: why is it slow? Is it memory-bound (many cache misses), compute-bound (all the instructions in the FPU), bandwidth-bound (a lot of data movement)? State of the art for professional profiling.

Comparison with AgentSight

There are two eBPF products for LLMs today with different focuses:

AgentSight (covered in the eBPF series): observes agents that call external APIs. It hooks SSL to see the plaintext of HTTPS calls to the remote LLM, plus stdio for local MCP servers. The client view.
ProfInfer / the eBPF-on-local-inference pattern: observes the engine that runs the model locally. It hooks the engine’s internal functions (llama.cpp, vLLM) and the CUDA layer. The server (internal) view.

They are complementary. If your agent uses the Claude API plus your own local vLLM with Llama 3 for specific tasks, AgentSight sees the first, eBPF/ProfInfer sees the second. If everything is local, clearly the domain of the second. If everything is an external API, of the first.

Use cases for eBPF in local inference

Three cases where it is the right tool:

Fine-grained profiling for optimisation: your vLLM takes 50ms more per token than expected. With eBPF plus hardware counters you pin down which specific kernel it is. Previously this required Nsight Systems in a development session; now it is continuous in production.

Token-level observability without modifying the engine: capturing the stream of generated tokens before returning them to the client. Useful for auditing, for drift detection over the outputs, for local tracing without going through wrapper instrumentation.

Detecting specific degradation: a new version of vLLM introduces a subtle regression in paged attention. With perf counter baselines, you detect the change even if the external metric (tokens/sec) looks the same.

Part 2 — Statistical analysis of flows: detecting drift

Now to the other side of the problem: the time series.

Why tracing, evals and guardrails do not detect drift

The layers we have already seen operate on individual events:

Tracing: a trace of one conversation.
Evals: a score for one answer.
Guardrails: a verdict on one prompt or answer.
MCP observability: spans of one tool invocation.

Each one answers a point-in-time question (“is this all right?”). None of them answers the question of evolution (“is something changing over time?”).

The operational problem: drift is invisible in individual events. If the mean eval score drops from 0.92 to 0.85 over three weeks, no individual evaluation will raise an alarm, since they all continue to be “reasonable”. What changes is the distribution. And that only shows up when you look at many evaluations aggregated over time.

The three types of LLM drift in 2026

FutureAGI consolidates them like this, and the industry is converging on this vocabulary:

1. Prompt drift: someone updates the system prompt and the side effects break cases that used to work. Almost always intentional, but with unanticipated consequences. Detection: comparing response distributions before and after the change, monitoring eval scores per prompt version (linked in Langfuse, see the AgentSight post where we covered prompt management).

2. Model drift: the provider (OpenAI, Anthropic) updates the model without warning. The same prompt produces answers with a slightly different tone, similar but different quality, or degradation on some subset. Detection: comparing today’s response embeddings with the baseline; monitoring rubric scores; alerting if intra-model variance grows.

3. Eval-score drift: the rolling mean of your eval metrics (faithfulness, answer relevancy, custom rubrics) trends downwards. The root cause can be any of the above or a change in the user mix. Detection: alerts on the trends of the eval series.

To these three you can add a fourth, subtler one:

4. Persona drift / user mix shift: the population of users using the system changes. It is not that the model or the prompt got worse; it is that the new users ask different questions and the system, while still just as good at what it was good at, fails on the new material. Detection: embedding clustering of prompts, monitoring the appearance of new clusters or the growth of a minority one.

The key technical concept: embedding-space shift

Stack Pulsar puts it plainly: in LLMs, drift is best measured in embedding space. Classic distances in token space do not capture fine semantics; in embedding space they do.

The canonical pipeline:

Establish a baseline: during a stable period (say the first two weeks after a release), capture a large sample of prompt and response embeddings.
Continuous monitoring: every hour or every day, capture a new sample of production traffic.
Compare distributions: apply a statistical test that compares the current distribution with the baseline in embedding space.
Alert: if the divergence exceeds a threshold, fire an alert and an investigation workflow.

As a bonus, monitor clusters: if your baseline has 5 clusters of prompts (technical questions, general support, sales, and so on) and suddenly a sixth cluster appears that was not there, the most likely explanation is that a new user segment has arrived.

Statistical tests: KS, PSI, MMD

Three tests that any drift system uses, each with its place:

Kolmogorov-Smirnov (KS): non-parametric. It computes the maximum distance between two empirical CDFs. It returns a statistic and a p-value. Advantage: very sensitive to subtle changes, especially in the tails. Disadvantage: with large datasets it is “too sensitive”, firing alarms for changes that are real but clinically irrelevant.

Population Stability Index (PSI): you bin the reference distribution and the current one, then sum (p_actual - p_ref) × log(p_actual / p_ref) over the bins. The canonical interpretation: PSI < 0.1 stable, 0.1-0.25 mild drift, > 0.25 significant drift. Advantage: interpretable, threshold-based, with a tradition of use in credit scoring (Capital One, Goldman Sachs). Disadvantage: less sensitive than KS, so it misses drift in the tails.

Maximum Mean Discrepancy (MMD): measures the divergence between two distributions by embedding each one in a Hilbert space via a kernel. It works for complex multivariate distributions (high-dimensional embeddings). Advantage: the only one that scales reasonably to embeddings of 768/1024/4096 dimensions. Disadvantage: harder to interpret.

The recommended practice in 2026:

PSI for simple features (prompt length, tokens, number of tools invoked).
KS for continuous features where you want high sensitivity.
MMD for embeddings (high-dimensional spaces).

Evidently’s analysis on real datasets showed that KS detects drift 6+ hours earlier than PSI in some incidents. The operational consequence: use KS for early warning, PSI for confirmation with an interpretable threshold.

Tools in 2026

Three products dominate the field:

Evidently AI

Evidently is open-source (Apache 2.0), Python-first. Its value:

HTML drift reports: you generate a report comparing two datasets (reference vs current) and you get an HTML file with all the statistical tests, visualisations and conclusions. No server, no infrastructure; one shareable file.
Native LLM support: on top of tabular, it supports text. It computes embeddings and applies the appropriate tests.
100+ metrics in the suite. It covers everything from a single framework.
Integration with MLflow and kube: CI workflows with reports on every release.

from evidently import Report
from evidently.metrics import DataDriftPreset

ref = load_baseline_dataset() # prompts from last week
cur = load_current_dataset() # prompts from the last hour

report = Report(metrics=[DataDriftPreset()])
report.run(reference_data=ref, current_data=cur)
report.save_html("drift_report.html")

When this functionality detects drift, it also tells you which column and which test fired.

NannyML

NannyML has a different focus: estimating model performance when you have no ground truth. The techniques:

CBPE (Confidence-Based Performance Estimation): estimates accuracy using the model’s confidence in its predictions.
DLE (Direct Loss Estimation): estimates the loss directly.

Useful when your LLM app has no immediate human feedback but you want to know whether its quality has dropped. Apache 2.0, Python.

WhyLabs

WhyLabs is commercial (with whylogs as the underlying OSS library), aimed at enterprise production:

Managed SaaS with SOC 2 Type 2 and HIPAA compliance.
Real-time monitoring via continuous log ingestion.
Embedding tracking: native support for embedding distributions, not just tabular features.
Token probability shifts: it monitors the probability distribution of generated tokens, not just metadata.

For regulated companies that do not want to operate their own drift detection platform, it is the lowest-friction option.

Other mentions

Arize Phoenix (seen in the Evals post) includes drift detection as a module. Galileo has commercial products specialised in LLM monitoring. Fiddler AI and Alibi Detect (Seldon) are more generalist alternatives that also cover LLMs.

Tool	Licence	Focus	Typical stack
Evidently AI	Apache 2.0	Drift reports + LLM	OSS Python, HTML reports
NannyML	Apache 2.0	Performance without GT	OSS Python, batch
WhyLabs	Commercial (whylogs OSS)	Enterprise SaaS, embeddings	Continuous logs, compliance
Arize Phoenix	ELv2	Tracing + drift unified	OSS, OTel-native
Galileo	Commercial	Premium LLM monitoring	SaaS, ML expert team
Alibi Detect	Apache 2.0	General drift detection	OSS Python, Seldon ecosystem
Fiddler AI	Commercial	Explainability + monitoring	Enterprise SaaS

Part 3 — The complete stack: how everything fits

Let us recap the layers the two series have covered, ordered from closest to the individual request to closest to the aggregate trend:

Individual EVENTS Aggregated TRENDS
│ │
Tracing ──→ Evals ──→ Guardrails ──→ MCP obs ──→ Drift detection
│ │
AgentSight ──→ Tetragon ──→ Hubble ──→ eBPF on-device
│ │
(what happens) (what changes)

Each layer answers a different question:

Layer	Question it answers	Granularity
Tracing (Langfuse, AgentSight)	What exactly did the agent do?	One session
Evals	Was the answer good?	One answer
Guardrails	Is this prompt/answer safe?	One message
MCP observability	Which tools did it invoke, at what cost?	One tool call
eBPF on agent/network (AgentSight, Hubble)	How did the system behave?	Per process/connection
eBPF on the local engine (ProfInfer-like)	How did the model execute?	Per runtime function
Drift detection	Is something changing silently?	Distribution

None of them replaces the others. Full coverage requires all seven. The practical operation:

Layers 1-3 (tracing, evals, guardrails) are mandatory from day one. Any LLM app in production that does not have them is flying blind.
Layer 4 (MCP) becomes mandatory when there are agents with tools, which is the majority in 2026.
Layers 5-6 (eBPF) become valuable when the scale justifies the operating cost (>10 services, >100 inference pods).
Layer 7 (drift) is the one that is most neglected and most expensive to ignore: it takes a day’s work to get the basic pipeline and it saves weeks of future incidents.

The operational drift pattern in 2026

The minimum recipe any serious LLM app should have:

Step 1 — Establish a baseline

During a stable post-release period (2 weeks minimum), store:

Embeddings of every prompt (vector + metadata: timestamp, user_segment, tenant).
Embeddings of the responses.
Automated eval scores over a sample (e.g. 5-10% of traffic with G-Eval).
The distribution of tools invoked (which tools, with which typical arguments, at what frequency).

Storage: any vector store plus a relational one. Reasonable cardinality at whatever scale you have.

Step 2 — Continuous comparison pipeline

Every hour (or every day depending on scale):

Take the sample for the current period (the last hour).
Apply the statistical tests against the baseline:
- PSI over simple features (prompt length, tokens, num tools).
- KS over continuous features (latency, score).
- MMD over embeddings.
Generate a drift report (Evidently does it in one line of Python).

Step 3 — Alerts and investigation workflow

Configure thresholds and routes:

PSI > 0.25 over tokens consumed: moderate alert (it may be legitimate, investigate segments).
Significant MMD over prompt embeddings: high alert (a change in user mix or a coordinated attack).
Eval rubric score down >5% over a rolling 7d: critical alert.
A new cluster in embedding space accounting for 10%+ of traffic: a review workflow (it may be a legitimate new segment or an anomaly).

Every alert must lead to a drill-down dashboard with the affected segments, not to an empty Slack message. The operational rule: if someone cannot investigate the alert in <5 minutes, it will not get investigated.

Step 4 — Baseline refresh

The baseline is not static. Every N weeks, refresh the baseline by incorporating the “new stable”. If over 3 months the usage pattern has legitimately changed (more international users, new languages), the baseline must reflect it. The typical cadence: quarterly.

Operational traps

A contaminated baseline

You take the baseline from a period that already contained the problem in embryo. Result: the baseline includes the bad behaviour and the tests never fire. Solution: verify the baseline against a second independent sample (for example, the first week vs the second) before blessing it.

A threshold that is too low

PSI > 0.05 fires constantly. Your team learns to ignore the alerts. Calibrate thresholds against the natural noise of your system: run the system with the baseline plus successive weekly samples and measure the PSI distribution; set the threshold a couple of deviations above normal.

Embeddings that are not representative

You use OpenAI’s text-embedding-3-small embedding model to detect drift in a system serving technical questions in Spanish about Cisco networks. Result: the embedding model does not capture the fine semantics of the domain. Solution: use embeddings finetuned for your domain, or one that is strong in multilingual and technical material.

Storage overload

Storing an embedding of every prompt in production at scale (millions of prompts/day) fills disks and increases cost. Stratified sampling: keep 5-10% of traffic, but make sure minority segments are over-represented so you do not lose them.

Confusing drift with “the system works”

Sometimes drift is good drift: new users discover the agent knows how to do X, and suddenly 30% of traffic is for X. The distribution changed because the product found a new use. Before pulling the alarm, check whether the change is desirable.

Privacy in embedding storage

Embeddings can be partially inverted back to their original text with embedding inversion techniques. If the prompts contain PII, storing embeddings for months for drift detection is a leak vector. Encrypt at rest and rotate regularly, or work with aggregated/averaged embeddings.

eBPF in production without profile guardrails

Attaching uprobes to hot-path functions such as llama_decode can hit throughput if it is not done carefully. Always test in staging and monitor the overhead. ProfInfer reports <4%; what you measure may vary with your binary and kernel.

Closing the two series

This week we have written 12 articles that walk the modern LLM inference stack in production from top to bottom:

LLM inference series (4 articles):

KV cache: the working memory that holds up LLM inference — fundamentals.
vLLM on Kubernetes — the engine.
PagedAttention deep dive — how it works inside.
LLM Operators on Kubernetes — orchestration.

eBPF series (4 articles):

eBPF from zero to Cilium — the substrate.
Tetragon — runtime security.
Hubble — network observability.
AgentSight — agent observability.

Post-tracing series (4 articles):

Evals — reactive quality.
Guardrails — preventive safety.
MCP observability — the tool protocol.
This one — drift detection and eBPF in local inference.

If you read the twelve in order you have a reasonably complete map of what it takes to operate AI agents in serious production in 2026, with enough detail not to crash into the usual problems in the first month. And, above all, with the mindset that LLM observability is a stack, not a product: each layer solves a problem, none of them solves all of them, and the combination is what separates an operable system from one that holds up until the first incident.

What is left for future series

MLOps specific to LLMs: continuous fine-tuning, RAG over data lakes, agent training.
Constitutional AI and alignment at runtime: how the model self-regulates with internal guardrails.
GPU networking: InfiniBand, NCCL, GPUDirect, the angle we left untouched.
Edge inference: llama.cpp on phones, MLX on macOS, Snapdragon NPU.
Theoretical inference scheduling: CFS-like algorithms applied to multi-tenant LLM serving.

We will cover them in time. That is it for now, and thanks for reading these twelve posts. If they gave you something, share them with a colleague.

References

eBPF in local inference:

ProfInfer: An eBPF-based Fine-Grained LLM Inference Profiler (arxiv 2601.20755) — the 2026 reference paper.
Monitor LLM Inference in Production 2026 (Glukhov) — Prometheus + Grafana for vLLM/TGI/llama.cpp.
AI Inference Server Observability in Kubernetes (ARMO) — the four signals MLOps tools do not capture.
vLLM vs llama.cpp: Choosing the right engine (Red Hat).

Drift detection concepts:

Tools:

Evidently AI (GitHub) — open-source.
Evidently — official site.
NannyML — performance without ground truth.
WhyLabs — managed observability.
Alibi Detect (Seldon) — general drift detection.
Arize Phoenix — drift integrated with tracing.

Statistical tests:

Data drift detection: PSI vs Kolmogorov–Smirnov (MLPipeline) — a practical comparison.
Population Stability Index for Model Drift Detection.
Which test is the best? 5 methods to detect data drift (Evidently) — the 6 hours of advantage for KS.

Cross-references (the three complete series):

LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.
Post-tracing series: Evals, Guardrails, MCP observability.

Llama-Cpp on lo0 — Tech Blog

Quantization for LLM inference: FP8, INT4 (GPTQ, AWQ) and GGUF — the model's accounting zoom

TL;DR

You are here: DEPLOY

The analogy: JPEG with an edge detector

The minimum maths: scale and zero-point

Numerical example

PTQ vs QAT: when quantisation happens

The dominant formats in 2026

FP8: the Hopper/Blackwell datacenter format

Microscaling: NVFP4 and MXFP4

INT4: GPTQ vs AWQ

GPTQ (Frantar et al. 2022)

AWQ (Lin et al. 2023)

Which one to choose

GGUF: the llama.cpp ecosystem

KV cache quantization

Measured quality loss (Llama 3.1 70B Instruct, reference)

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace, no native FP8)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered (upcoming articles)

See also

References

eBPF in local inference and statistical drift detection: closing the LLM observability loop in 2026

TL;DR

The analogy: the agent’s cardiogram

Part 1 — eBPF applied to local inference

Why local inference changes the game

ProfInfer: the paper that formalises the pattern

Where to hook: the map by engine

llama.cpp

vLLM

CUDA in general

Hardware counters: the other half

Comparison with AgentSight

Use cases for eBPF in local inference

Part 2 — Statistical analysis of flows: detecting drift

Why tracing, evals and guardrails do not detect drift

The three types of LLM drift in 2026

The key technical concept: embedding-space shift

Statistical tests: KS, PSI, MMD

Tools in 2026

Evidently AI

NannyML

WhyLabs

Other mentions

Part 3 — The complete stack: how everything fits

The operational drift pattern in 2026

Step 1 — Establish a baseline

Step 2 — Continuous comparison pipeline

Step 3 — Alerts and investigation workflow

Step 4 — Baseline refresh

Operational traps

A contaminated baseline

A threshold that is too low

Embeddings that are not representative

Storage overload

Confusing drift with “the system works”

Privacy in embedding storage

eBPF in production without profile guardrails

Closing the two series

What is left for future series

References