Opentelemetry on lo0 — Tech Blog

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

Mon, 14 Sep 2026 08:00:00 +0200

Fourth article in a series of ten about operating Langfuse v4 in production. The previous three looked at the client side: what goes into a trace, what an agent turn costs and two well-known agents instrumented. This is the first one on the server side. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

The default write mode is the destination, not the origin. LANGFUSE_MIGRATION_V4_WRITE_MODE ships as events_only (packages/shared/src/env.ts:337). That is right for a new installation and wrong for a migration: in that mode the traces and observations tables stop being written.

The worker refuses to start on three combinations. legacy with direct OTel writes, events_only without the preview opt-in, and events_only with OTel behaviour set to dual. The code comment names them: combinations that would silently lose data (worker/src/env.ts:722).

In events_only twenty-four REST routes return 404. Traces, observations, sessions, v1 and v2 scores, metrics, spans, generations and the dataset-run ones. Not a bug: a deliberate short circuit so stale data from tables nobody writes is never served.

History is backfilled by a chain of five background migrations, not by the ClickHouse migrations. Not one of the ten new ClickHouse migrations copies a single old row.

A failed background migration does not block the next one. The manager has no dependency model and skips them. The chain protects itself with predecessor guards that the first two steps do not declare, so a failure of step 1 does not stop step 2 from starting.

The old API’s ten-minute delay has an exact explanation. Three-minute partitions in the staging table, a lock four minutes after creation, and a per-minute cron that only touches partitions older than ten minutes, with a global concurrency of one.

That staging table has a forty-eight-hour TTL. If the propagator stops over a long weekend, those partitions disappear and the hole in events_full is permanent. No code detects it and none repairs it.

The point of no return is in Postgres, not in ClickHouse. A Prisma migration runs DROP TABLE on traces, observations and scores, and Prisma Migrate has no down migrations.

You are here: OBSERVE, day 2

The first three articles measured what an agent produces. This one starts on operating the backend that receives it, and it comes before the capacity article because you cannot size what has not been migrated yet.

The analogy: changing the track gauge without closing the line

Changing the gauge of a line in service is not done in one go. The third rail is laid, rolling stock of both gauges runs for a while, and only when every train is of the new gauge is the old rail lifted.

The Langfuse migration has exactly that shape. dual mode is the third rail: the old tables and the new ones are written at the same time. And as with the track, the danger is not the day of the change, it is at both ends. Laying the new rail and lifting the old one on the same day strands trains. And once the old rail is up, going back is not a matter of will, it is a matter of the material no longer being there.

The three-rung ladder

Everything turns on one variable with three values, defined three times over in the three packages that have to agree, with the same enum and the same default (packages/shared/src/env.ts:337, worker/src/env.ts:586, web/src/env.mjs:610):

LANGFUSE_MIGRATION_V4_WRITE_MODE: z
 .enum(["legacy", "dual", "events_only"])
 .default("events_only"),

The two helpers that read it say what each rung does better than any explanation (worker/src/env.ts:705):

export const v4WritesToEventsTable = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "legacy";

export const v4WritesToLegacyTables = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "events_only";

What gets written in each mode:

Table	`legacy`	`dual`	`events_only`
`traces`, `observations`	yes	yes	no
`scores`	yes	yes	yes
`dataset_run_items_rmt`	yes	yes	no
`observations_batch_staging`	no	yes	no
`events_full`	no	yes	yes
`events_core`	no	yes (through the materialised view)	yes

The middle rung is the only one where both coexist, and therefore the only place you can step back from without losing anything. The code itself offers it as the remedy when something breaks in events_only (web/src/pages/api/public/ingestion.ts:296):

As a temporary migration bridge, set LANGFUSE_MIGRATION_V4_WRITE_MODE=dual on both the web and worker services and redeploy.

The both in that sentence matters. All three packages read the variable and the code comment insists three separate times on keeping them in sync. Deploying the worker with one value and the web with another is the first way to break this.

The three combinations that refuse to start

The worker validates its configuration at boot and throws in three cases, under a comment that leaves no doubt (worker/src/env.ts:718):

 // Hard errors: combinations that would silently lose data.
 if (mode === "legacy" && otel === "direct") {
 throw new Error(
 "Invalid V4 config: LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct " +
 "requires LANGFUSE_MIGRATION_V4_WRITE_MODE in {dual, events_only}. " +
 "Direct OTel writes target events_full, which is not read in legacy mode.",
 );
 }

The second one bites in a real migration: events_only also requires LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true, because otherwise the web reads would target the legacy tables that mode no longer writes. The third forbids events_only with OTel behaviour set to dual.

A failed boot is the good news. A failed boot shows up in thirty seconds. The combinations it does not validate take days to surface.

What breaks in each mode

This is the table to have in front of you before moving the variable, because the two ends break different things.

In legacy the event tables are not written, so whatever depends on them disappears. Monitor routes return 404 through a dedicated middleware (web/src/server/api/trpc.ts:413). Metrics v2 is unavailable. Dashboard widgets stay at version 1 (DashboardService.ts:39). The v4 preview is forced off and cannot be switched on.

In events_only the legacy tables are not written, so twenty-four REST routes short-circuit with a 404 (web/src/features/public-api/server/createAuthedProjectAPIRoute.ts:319):

 // Short-circuit routes that read from legacy traces/observations tables
 // when the deployment is in events_only mode — those tables are no longer
 // populated, so the response would be stale or empty.

The list covers traces, traces/{id}, observations, observations/{id}, sessions, sessions/{id}, scores and scores/{id} in v1 and v2, metrics, metrics/daily, spans, generations, events, dataset-run-items and the two datasets/{name}/runs routes. If you have your own integrations against those routes, they break on cutover day, and returning 404 rather than stale data is the right call even though it stings.

One detail that is usually reported wrongly: /api/public/ingestion is never shut down. The deprecation notice says so itself (deprecations.ts:104):

This route is never removed: scores (and sdk-log) stay accepted.

In events_only that route stays alive and keeps accepting two event types (web/src/pages/api/public/ingestion.ts:274):

const EVENTS_ONLY_ALLOWED_TYPES = new Set<string>([
 eventTypes.SCORE_CREATE,
 eventTypes.SDK_LOG,
]);

The rest are rejected per event, not per batch: the batch comes back as a 207 with 400s inside. An old client sending traces and scores mixed together will see the scores go through and the rest fail, which is exactly the kind of partial failure that takes a long time to notice.

The three steps, in order

Given all of the above, the order of a no-window migration is this:

Finish the v3 background migrations before upgrading. This comes before everything else and there are no shortcuts. v4 deletes the rows of seven old migrations, and the comment explains why (prisma/migrations/20260723124010_drop_dataset_run_items_table/migration.sql:7): their worker scripts are deleted alongside, so an unfinished row would leave the manager with no script to resolve. If one was halfway, that data is lost and the row disappears quietly.
Upgrade to v4 with LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy, apply the ClickHouse and Prisma migrations. Here the installation behaves like v3 and you verify everything still stands.
Move to dual. From then on both table families are written and the propagator starts. Let it run for as long as it takes and launch the historic backfill.
Once history is complete and verified, move to events_only with the preview opt-in enabled, and only then lift the old rail.

All the work is between 3 and 4, and it can take weeks. There is no rush: dual is a stable state.

The ClickHouse migrations backfill nothing

There are 96 files in packages/shared/clickhouse/migrations/canonical/, 48 up and 48 down, which have to be materialised before running (pnpm ch:migrations:materialize) and cleaned afterwards, including after a failed migration. The runner is golang-migrate.

The ten v4 ones are 0039 through 0048. The first three create events_full, events_core and the materialised view that feeds the second from the first. The three drop migrations remove event_log, project_environments and dataset_run_items. The ones in between add columns and indexes.

What matters for planning is what they do not do. None copies an old row. The materialised view only sees rows inserted into events_full after it exists. The indexes in migrations 0043 and 0047 are added without materialising historical parts, and 0047 says so in writing:

Skip indexes are added without materializing historical parts to keep this migration metadata-only.

Translated into operations: during and after the migration, searches over older data do full scans. Not a bug, the price of a migration that does not block.

And one cluster trap: migrations 0042, 0043, 0047 and 0048 carry alter_sync = 2, meaning every ALTER waits for all replicas. One slow or down replica hangs the whole migration. On a single-replica installation you never notice; on three, you do.

The five-step chain that does backfill history

The backfill is done by five background migrations running in the worker, registered in Postgres with names that carry the order inside them:

20260701_v4_step_1_create_root_spans_from_traces: creates virtual root spans in events_full from existing traces, with the identifier concat('t-', t.id). It skips traces referenced by a dataset run item, which belong to step 4. Chunked by monthly partition.
20260701_v4_step_2_rewrite_observations_to_pid_tid_sorting: copies observations into a scratch table with a different key order. Chunked by monthly partition.
20260701_v4_step_3_backfill_events_full_from_observations: the bulk of it. Chunked by ClickHouse part, not by partition, and the comment explains why that is the only safe granularity:

One todo per part keeps each INSERT bounded to a single ClickHouse part (single-digit GBs in most cases), which is the only granularity that’s safe to assume self-hoster hardware can chew through without OOM or memory-limit failures.
20260701_v4_step_4_backfill_events_full_from_dataset_run_items: walks the dataset run items by cursor and writes the root and its descendants. Batches of 200, a cap of 50,000 descendants per item, a 90-day window.
20260701_v4_step_5_drop_pid_tid_sorting_tables: drops step 2’s scratch tables.

There are only two switches for the five. The first four share LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL, which ships true, and the fifth has its own, LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES, which ships false on purpose so the scratch tables stay around until the operator confirms the backfill went well.

One migration at a time across the whole cluster. The manager takes a lock in Postgres, refreshes it with a heartbeat every 15 seconds and treats a lock older than 60 seconds as expired, all inside a serialisable transaction. Adding worker replicas does not speed the backfill up by one minute. That is the first thing that surprises anyone used to scaling out to go faster.

They are resumable, and rather better than this usually is. Progress lives in a JSONB field on the row, and on resume the worker re-attaches to the queries the previous worker left running, by query id, instead of relaunching them:

In-flight queries keep running server-side; the next run re-attaches to them via the persisted queryIds.

On how long they take, the code says nothing. Not the files, not the folder’s README, not the migrations. The only time threshold anywhere is a design one: “A good threshold is something that takes more than 5 minutes to run”. Any figure in hours or days you read elsewhere, including the one I repeated myself before checking, is not in the code.

The failure that does not block

This is the finding that changes a runbook the most. When a background migration fails, the manager marks it with failedAt and moves on. And the predicate it uses to pick the next one excludes exactly the rows carrying failedAt, so a failed migration is skipped and does not block the ones after it.

The code acknowledges the gap and explains the patch (worker/src/backgroundMigrations/utils/backfillBase.ts:107):

The BackgroundMigrationManager has no dependency model — it runs every eligible row in name order and marks each finished/failed independently, so a failed upstream step does NOT by itself stop a downstream step from running on partial data. Each downstream step calls this from its validate() […] Because the next step in turn guards on its own predecessor, a single failure transitively halts the rest of the chain.

The guard exists, but only in steps 3 and 4. Steps 1 and 2 declare no predecessor. A failure of step 1 does not stop step 2 from starting; what saves the situation is that step 3 does check step 2.

Practical consequence: the only reliable signal that the backfill is going well is to look at the background_migrations table in Postgres and check that all five rows have finishedAt and none has failedAt. And to restart a failed one you clear failedAt by hand and relaunch with --retry-failed, as the error message itself says.

Two concrete failure modes that deserve a place in the runbook:

Step 3 breaks if ClickHouse merges a part midway. The message says it all: “Part … no longer active after processing — its rows are in a merged successor part that is not in this run’s todo list. Clear failedAt and re-run with state.chunksLoaded=false”. Recovery is manual, editing the JSONB.
Step 4 stops dead on one giant trace, if it exceeds 50,000 descendants. The three ways out the error offers are raising the cap, skipping the item (and losing that trace entirely) or fixing it by hand in SQL.

And an asymmetry worth knowing before comparing data: the historic backfill does not copy trace metadata down to child spans, deliberately, whereas the live path does concatenate them. History and new data are not identical.

Where the ten minutes come from

The v3 API warns that its data runs about ten minutes behind (deprecations.ts:16):

Data on this API is delayed by about 10 minutes. Real-time is only OpenTelemetry writes plus v2 observations and v2 metrics reads.

That delay is the sum of four things, all visible in the code:

The staging table partitions on three-minute intervals.
A partition is treated as closed four minutes after creation: “We ’lock’ partitions 4min after their creation, i.e. the 15:00:00 partition should stop receiving updates at 15:04:00.”
The propagation cron runs every minute and only looks at partitions older than LANGFUSE_EXPERIMENT_EVENT_PROPAGATION_PARTITION_DELAY_MINUTES, which defaults to 10.
That cron has a global concurrency of one across the cluster and processes a single partition per invocation, so a backlog makes the delay grow rather than recover.

The way out of the delay is not to lower that variable, it is to have clients write over OTLP with direct writes, which is the real-time route. The header that enables it is x-langfuse-ingestion-version: 4, sent automatically by the Python SDK from 4.0.0 and the JavaScript one from 5.0.0, and a value above 4 is rejected with a 400.

The forty-eight-hour TTL

This is the frightening one, and it appears in no migration guide.

The staging table carries TTL toDateTime(s3_first_seen_timestamp) + INTERVAL 48 HOUR with ttl_only_drop_parts = 1. If the propagator stops for more than forty-eight hours, a long bank holiday for instance, those partitions delete themselves. The data is already in the legacy tables, so an installation in dual keeps working, but the hole in events_full is permanent, and no code detects it and none repairs it. The health probe detects stuck, not lost.

The defence is an alert, and one exists for this. The worker health endpoint accepts ?failIfEventPropagationStuck=true and returns 503 if the propagator’s heartbeat has not been refreshed for more than LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES, which defaults to 35. The comment carries a condition to read before putting it in a Kubernetes probe:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

Another detail of the same mechanism: the propagation cursor lives in Redis, not in Postgres. If Redis is flushed, the cursor resets and every live partition of the staging table is reprocessed.

What can and cannot be undone

The ClickHouse down migrations exist, and they are honest about what they do. The one for 0039 warns in capitals:

WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this table may predate this migration […] Rolling back drops that data regardless of which path created the table.

The ones for the dropped tables recreate the schema and nothing else: “Any data held in event_log before the up migration dropped it is NOT restored.”

But the real point of no return is not in ClickHouse, it is in Postgres. There is a Prisma migration called 20260723150000_drop_legacy_tracing_tables that does exactly what it says:

DROP TABLE IF EXISTS "traces";
DROP TABLE IF EXISTS "observations";
DROP TABLE IF EXISTS "scores";

Prisma Migrate has no down migrations. It applies, and that is that.

Summarising the real reversibility: the write mode can go down from events_only to dual whenever you want, and that is the panic button. What does not come back is what was not written while you were up there: the legacy tables receive nothing retroactively. And step 5 of the chain, the one that drops the scratch tables, ships disabled precisely so you can do forensics if something went wrong.

One last note on the ground you are standing on. The comment in the export-source policy drops something worth reading twice: “legacy-writes-disabled, which in practice only surfaces on self-hosted (Cloud does not run events_only)”. The events_only path is the one you will use and not the one the vendor’s managed service uses. It is less battle-tested in production than dual.

How this looks in code

The minimum runbook, in variables:

# Step 2: upgrade to v4 behaving like v3
LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy
LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=dual_write # "direct" will not start here
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=false

# Step 3: third rail. Both services, web and worker, together.
LANGFUSE_MIGRATION_V4_WRITE_MODE=dual
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=true
LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES=35

# Step 4: only once all five background_migrations rows have finishedAt
LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only
LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true # without this it will not start
LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES=true # last of all

And the query that answers the only question that matters for weeks:

SELECT name, started_at, finished_at, failed_at, failed_reason,
 state->>'chunksLoaded' AS chunks
FROM background_migrations
WHERE name LIKE '20260701_v4_%'
ORDER BY name;

Migration checklist

Every v3 background migration is finished before the image is upgraded.
There is a Postgres and a ClickHouse backup from before the first DROP TABLE.
The write mode has the same value in the worker and in the web, and both change together.
The upgrade went to legacy first, and the service was verified unchanged.
There is an alert on ?failIfEventPropagationStuck=true, with initialDelaySeconds of at least 60 seconds.
Somebody looks at the background_migrations table every day while the backfill runs.
The in-house integrations calling the twenty-four routes that will return 404 have been inventoried.
Client SDKs are on Python 4.0.0 or JavaScript 5.0.0 at minimum before moving to events_only.
Step 5 of the chain is enabled last, and only after verifying the backfill.

Traps

The default is the end state. A v3 installation booting v4 without touching the variable lands in events_only and stops writing the old tables from the first second.

A failed background migration does not block the next one. And the first two steps of the chain declare no predecessor.

More worker replicas do not speed up the backfill. One migration at a time across the cluster, by design.

Forty-eight hours of stopped propagator is a permanent hole. No detection and no repair.

New indexes do not cover old data. Historical searches scan until somebody materialises the indexes by hand.

On a cluster, every ALTER waits for all replicas. One down replica hangs the migration.

Classic ingestion rejects per event. A mixed batch returns 207 and half of it is lost without anyone reading the response body.

History and the live path do not produce quite the same thing. Trace metadata does not descend to children in the backfill.

The route you will take is not the one the vendor takes. Cloud does not run events_only.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window (this article): the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/env.ts, worker/src/env.ts, web/src/env.mjs, web/src/features/public-api/server/deprecations.ts, web/src/pages/api/public/ingestion.ts, web/src/features/public-api/server/createAuthedProjectAPIRoute.ts, worker/src/backgroundMigrations/, worker/src/features/eventPropagation/, packages/shared/clickhouse/migrations/canonical/, packages/shared/prisma/migrations/.
Langfuse compatibility and migration, checked on 14 September 2026.
OpenTelemetry integration migration to v4, checked on 14 September 2026.

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

Sun, 13 Sep 2026 07:00:00 +0200

Second article in a series of ten about operating Langfuse v4 in production. The first one, what really goes into a trace, walked through the data model by reading the server code. This one measures the other side. Verified against Langfuse 4.15.2 (Python SDK), LangGraph 1.2.11 and langchain-core 1.6.3, with figures taken on 13 September 2026 on a bench with no network.

TL;DR

One agent turn is four observations plus five per tool call. Measured with an in-memory exporter over a minimal ReAct graph: zero tools, 4 observations; one, 9; five, 29; ten, 54; forty, 204. The formula holds exactly across the whole range.

The byte volume grows quadratically, not linearly. The same turn goes from 12.2 KB with one tool call to 1.87 MB with forty. 88 % of those bytes is the input field, because every observation re-serialises the full message history as it stands at that moment.

Tool definitions travel attached to every model call. The handler appends them to the input field as messages with role tool. Measured: 364 bytes per tool. With twenty tools that is 7.3 KB on every single step of the loop.

The AGENT observation type is assigned by string match on the name. A node called planificador comes out as CHAIN; the same node called agente_planificador comes out as AGENT. The logic is literally looking for agent in the class path or in the node name (langfuse/langchain/CallbackHandler.py:397).

An interrupt with a resume produces two separate traces. Verified with invoke, with stream and with subgraphs, and also reusing the same handler object. The SDK ships machinery to reattach the trace, but it only fires if the root run finishes with a control-flow exception, and in LangGraph 1.2.11 the interrupt comes back through the return value.

The fix fits in one line and is measured. Passing trace_context={"trace_id": Langfuse.create_trace_id(seed=thread_id)} when building the handler leaves the interrupt and the resume in the same trace, even with different handlers and therefore with different replicas.

The span processor drops by default whatever it does not recognise. It only exports spans from the SDK’s own tracer, spans carrying some gen_ai.* attribute, or spans from a closed list of thirty-five instrumentation prefixes (langfuse/_client/span_filter.py:11). Your own code’s spans do not arrive unless they come through one of those three doors.

You are here: OBSERVE, day 2

The series was planned with eight articles and has grown to ten. The reason is ordering: the next two are about migrating and about sizing ClickHouse, and both need an input figure that nobody publishes, which is how many observations and how many bytes a real unit of work produces. A chat produces a trace with two observations. An agent produces two hundred. Sizing with the chat figure leads to a cluster that falls over the day someone connects the agent.

So before migrating, measure. And to measure there has to be in front what people actually deploy.

The analogy: the file that gets photocopied whole at every signature

A procurement file passes across six desks. At each desk somebody reads what is there, adds a sheet and passes it on.

There are two ways to leave a record of that route. The first is to write in a register which desk touched it, when, and which sheet it added. Six short entries. The second is to photocopy the whole file on the way into each desk and on the way out, and to archive the twelve copies. The register takes one sheet. The copies take, at the sixth desk, six times what they took at the first, and in total they take something on the order of the square of the number of desks.

LangChain’s callback instrumentation does the second thing. It does not record the delta of each step, it records the complete state on the way into and out of each step. With a chat that makes no difference, because the state is short and there is one step. With a forty-step agent, the archive weighs almost two megabytes per file processed.

This is not a defect that needs fixing. It is what lets you open a trace and see exactly what the model saw at step 17 without reconstructing anything. But it drives the sizing, and the number has to be known before signing the contract.

Which agentic platform is the most used

The question has two answers depending on the metric, and both are defensible.

By PyPI downloads over the last thirty days, checked on 13 September 2026:

Package	Downloads / 30 days
`langgraph`	52.3 M
`strands-agents`	35.2 M
`openai-agents`	22.9 M
`crewai`	19.1 M
`google-adk`	14.5 M
`pydantic-ai`	8.0 M
`llama-index`	5.0 M
`agno`	1.9 M
`autogen-agentchat`	0.6 M

By GitHub stars the order flips at the top: CrewAI is at 58.4 k, LangGraph at 38.1 k and the OpenAI Agents SDK at 28.6 k.

PyPI downloads are inflated by continuous integration and by images that reinstall on every build, so they indicate presence in pipelines rather than agents in production. Stars measure interest, not deployment. With both together, LangGraph is the one that shows up in more pipelines and the one with the most worked-through instrumentation path in Langfuse, so it is the one instrumented here. What follows about volume applies equally to any platform instrumented through LangChain callbacks, because the cost comes from the callback model, not from the graph.

How LangGraph gets into Langfuse

The route is worth stating, because the previous article left two open and only one is used here.

langchain-core 1.6.3 does not import OpenTelemetry anywhere. The check is direct against the installed wheel and returns not one file. LangGraph does not emit OTLP spans on its own either. What exists is LangChain’s callback system, and the Langfuse CallbackHandler hooks into it.

That handler does not talk to the classic ingestion API. It creates observations with the v4 SDK, which underneath is OpenTelemetry: LangfuseSpanProcessor sets up an OTLPSpanExporter against {base_url}/api/public/otel/v1/traces (langfuse/_client/span_processor.py:123). So it is OTLP, but generated by the SDK, not by third-party auto-instrumentation.

The difference matters for one concrete reason left hanging in the previous article. Because the data passes through the SDK calls, the classic masking function does apply: input, output and metadata all go through _process_media_and_apply_mask before becoming attributes (langfuse/_client/span.py:534). With auto-instrumentation of the OpenInference or OpenLLMetry kind, the messages travel in span events and masking does not touch them. For an agent handling customer data inside its tools, that difference decides the route.

The trade-off is that a callback handler only sees what goes through LangChain. HTTP calls a tool makes on its own, the database query, the subprocess, none of that shows up. And if you instrument your code with OpenTelemetry so that it does show up, you run into the filter described further down.

The observation type is decided by the node name

The first article insisted that v4 has ten observation types and that using the right one changes what the interface knows how to group. The LangChain handler emits six of those ten: tool, retriever, generation, agent, chain and span. It never emits event, embedding, evaluator or guardrail.

The assignment is direct for four of them: tool callback, type tool; retriever callback, type retriever; model callback, type generation. For chain callbacks, which is what every node of a LangGraph graph is, the criterion is this (langfuse/langchain/CallbackHandler.py:393):

elif callback_type == "chain":
 # Detect if it's an agent by examining class path or name
 if serialized and "id" in serialized:
 class_path = serialized["id"]
 if any("agent" in part.lower() for part in class_path):
 return "agent"

 name = self.get_langchain_run_name(serialized, **kwargs)
 if "agent" in name.lower():
 return "agent"

 return "chain"

Measured over the same graph with the node renamed and everything else identical:

Node name	Tool calls	Observations	Types
`planificador`	10	54	11 generation, 33 chain, 10 tool
`agente_planificador`	10	54	11 generation, 22 chain, 11 agent, 10 tool

Eleven observations change type because of eleven characters in a function name. This has two practical consequences. The first is that any dashboard counting AGENT observations is counting naming conventions. The second is that the interface draws the agent graph when the trace contains any observation of a type other than span, event or generation, and chain qualifies, so the drawing comes out either way. What changes is what you can filter and aggregate afterwards.

The operational recommendation is boring and it works: name the nodes that represent a model decision with a name containing agent, and leave the rest as chains. It is a naming convention promoted to data schema, which is not pretty, but it is what there is and it is stable.

What a turn costs, measured

The setup is a minimal ReAct graph: one node that calls the model, a ToolNode with one tool, a conditional edge with tools_condition and an in-memory checkpointer. The model is langchain-core’s GenericFakeChatModel, which returns a fixed list of messages, so neither network nor GPU is needed and the experiment reproduces on any laptop. The exporter is OpenTelemetry’s InMemorySpanExporter, passed to the client through the span_exporter parameter, so observations are counted with no Langfuse server in front.

Result, with the node named planificador:

Tool calls	Observations	generation	chain	tool	Attribute bytes
0	4	1	3	0	4.2 KB
1	9	2	6	1	12.2 KB
5	29	6	18	5	64.1 KB
10	54	11	33	10	173.5 KB
20	104	21	63	20	541.4 KB
30	154	31	93	30	1,108 KB
40	204	41	123	40	1,874 KB

The observation count is exactly 4 + 5n. The five per loop iteration are: the model node, the model call, the conditional edge, the tools node and the tool. The four fixed ones are the root graph, the first partial pass and the close.

The bytes do not follow that line. From 10 to 20 steps the volume multiplies by 3.1; from 20 to 40, by 3.5. That is the quadratic growth expected when each of the n steps drags along a history whose length is proportional to n.

The breakdown by field confirms it. With forty steps, out of 1,709 KB measured in that run, 1,502 KB sit in langfuse.observation.input and only 56 KB in the output. 88 % of what is ingested is the same history repeated at different stages of progress. The largest single observation in that turn is 22.2 KB, well below the 9.5 MB per OTLP span ceiling, so the problem is not the size limit: it is the aggregate.

And inside the input there is a second multiplier. The handler appends each tool’s definition to the generation’s input field, as messages with role tool (langfuse/langchain/CallbackHandler.py:1190). Measured with three-parameter tools and a one-line description:

Bound tools	Input field bytes of one generation
0	34
1	398
5	1,854
10	3,674
20	7,334

That is 364 bytes per tool, on every generation, always. An agent with twenty tools and forty steps ingests 7.3 KB × 41 generations = 300 KB in tool schemas alone, which never change over the course of the turn.

What this means for ClickHouse

A honest caveat belongs here. I compressed the full payload of those runs with ZSTD level 3, which is what events_full uses, and got ratios between 29× and 86×, improving the longer the turn. That number is inflated by how synthetic the bench is: my tool always returns the same string, so the redundancy is higher than in production.

What does hold is the shape. Repeating the history is exactly the pattern columnar compression absorbs well, so the ClickHouse disk is not where it hurts. Where it hurts is earlier: in the bandwidth towards the OTLP endpoint, in the worker CPU that decompresses and flattens, and in the ingestion queue, all of which see uncompressed bytes. The first article noted that the queue that saturates first is langfuse.queue.ingestion.depth, and this is why it saturates.

Quick sizing arithmetic, with the measured figures and an average ten-step turn:

1,000 turns a day is 54,000 observations and around 170 MB uncompressed a day.
10,000 turns a day is 540,000 observations and 1.7 GB uncompressed a day.
If the average turn goes from ten steps to twenty, observations double and bytes triple.

Sensitivity to the number of steps is the parameter to watch, more than the number of users. A prompt change that makes the agent take three more turns around the loop multiplies the observability bill with nobody touching the infrastructure. Article 6 of the series comes back to this with ClickHouse system tables to measure the real bytes per observation in your own installation, which is the only thing that replaces this estimate.

Interrupts: two traces where there should be one

A production agent with human approval uses interrupt(), hands control back, waits for a person and resumes with Command(resume=...). The operational question is whether the trace survives that pause.

The SDK ships explicit machinery for the case. There is a store of pending trace contexts keyed by thread_id, capped at 1,024 entries with eviction of the oldest (MAX_PENDING_RESUME_TRACE_CONTEXTS), and a function that recognises a Command carrying resume and recovers the stored context.

Measured, it does not fire in this combination of versions:

Case	Same trace?
Same handler object, `invoke`	No
Same handler object, `stream`	No
Interrupt inside a subgraph	No
Fresh handler for the resume	No

The reason is in the code and it is consistent. The pending context is only stored from on_chain_error when the failing run is the root one and the resulting level is DEFAULT, which is what happens with LangGraph’s control-flow exceptions (CallbackHandler.py:866). In LangGraph 1.2.11 the interrupt does not leave the root as an exception: the graph returns normally, with the interrupt in the return value. The root run ends through on_chain_end and the branch that stores the context never runs. The interrupted node’s observation does come out right: level DEFAULT and the Interrupt in the status message, not marked as an error, which is the correct behaviour.

The fix is verified and does not depend on the SDK changing:

from langfuse import Langfuse
from langfuse.langchain import CallbackHandler

trace_id = Langfuse.create_trace_id(seed=thread_id)
handler = CallbackHandler(trace_context={"trace_id": trace_id})

With that, the interrupt and the resume land in the same trace even if the handler is a different object and even if another replica serves the resume, because the identifier derives from the thread_id and not from in-memory state. It is the same technique the documentation recommends for joining several agents into one trace, applied to joining an agent with itself.

With one nuance to decide deliberately. Seeding with the bare thread_id puts every turn of that conversation into a single trace, which grows without bound while the conversation stays alive. If what you want is one trace per turn with the resume attached to the right turn, seed with thread_id plus the turn number, and leave conversation grouping to the session identifier, which is what it is for. Since v4 has no trace table, a huge trace is not a huge row: it is a filter returning many rows, and what suffers is the interface when opening it.

Thread, session, user, and what is only read at the root

The handler recognises five metadata keys: langfuse_session_id, langfuse_user_id, langfuse_trace_name, langfuse_tags and langfuse_prompt. The first four are read only when parent_run_id is None, that is, only on the root run (CallbackHandler.py:587). Putting them in the config used to invoke a subgraph does nothing.

The recommended route, and the one that works without surprises, is the context manager:

from langfuse import propagate_attributes

with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["production", "support-l1"],
):
 graph.invoke(payload, config=cfg)

What LangGraph contributes on its own arrives anyway. In the measured run these show up as observation metadata: langgraph_node, langgraph_step, langgraph_triggers, langgraph_path, langgraph_checkpoint_ns, checkpoint_ns and thread_id, plus ls_provider, ls_model_type and ls_integration. The thread_id is also promoted to trace metadata.

That is fine, but keeping in mind the first article: in events_core what is indexed is metadata_names, that is the key names, not the values. Filtering by langgraph_node = "planificador" is a scan. If you are going to segment by node routinely, the right place is the observation name, which is where the node already appears, or a tag, assuming that tags are not indexed either. The cheap field for slicing by node in v4 is the observation name itself.

The session identifier deserves an explicit decision. LangGraph’s thread_id and Langfuse’s session_id are the same concept and they do not connect on their own. Mapping them one to one is the reasonable thing, and it is also what makes the whole conversation navigable in the interface even when each turn is a separate trace.

The filter that drops your own spans

This is the finding that surprises most people while setting it up. The Langfuse span processor does not export everything that passes through the TracerProvider. It exports what satisfies one of these three conditions (langfuse/_client/span_filter.py:104):

It was created by the Langfuse SDK’s own tracer.
It carries at least one attribute starting with gen_ai.
Its instrumentation scope matches one of thirty-five prefixes in a closed list.

The list covers the expected and a bit more: openinference, litellm, haystack, langsmith, strands-agents, pydantic-ai, autogen-core, vllm and a couple of dozen opentelemetry.instrumentation.* entries.

The consequence for an agent is direct. If you instrument your tools with OpenTelemetry by hand so that the ERP call or the Postgres query shows up inside the agent’s trace, those spans are dropped silently, with no log and no error. There are three ways out, in order of cleanliness:

Create those observations with the Langfuse SDK, with start_as_current_observation(as_type="tool"), which is what the rest of this article assumes.
Put some gen_ai.* attribute on the span, which is abusing the convention to smuggle the span through, so only if the span does describe an interaction with a model.
Send those spans to your normal OpenTelemetry collector instead of Langfuse, and correlate by trace identifier. For a sovereign platform with its own collector, this is the sensible option: Langfuse keeps the model side, the collector keeps the system side, and the trace identifier joins them.

The behaviour being silent is what makes this take an afternoon to find. It belongs in the runbook.

The model and the cost with a gateway in the middle

One detail that affects anyone putting LiteLLM in front, which is the setup described in the operational pair and in GPU multi-tenancy.

The model name the handler puts on the generation comes first from the ls_model_name metadata and, failing that, from the component’s serialisation (CallbackHandler.py:1297). With a ChatOpenAI pointed at the gateway, that name is whatever alias you defined in LiteLLM, for example qwen-32b-internal. Langfuse computes cost from its own model table, which does not know that alias, so the generation arrives with tokens and without cost.

It is fixed by defining the model in Langfuse under that same name with its price per million tokens, which is also the only way for the cost to reflect what your GPU costs rather than what somebody else’s API costs. It is an afternoon of work and it stops the agentic platform’s cost panel from sitting at zero forever.

How this looks in code

The minimum to have in place for a LangGraph agent to produce useful traces in Langfuse v4:

import os
from typing import Annotated, TypedDict

from langfuse import Langfuse, get_client, propagate_attributes
from langfuse.langchain import CallbackHandler
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition

# 1. Client. Masking applies because the data goes through the SDK.
def mask(*, data, **kwargs):
 return redact_pii(data)

Langfuse(
 public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
 secret_key=os.environ["LANGFUSE_SECRET_KEY"],
 host=os.environ["LANGFUSE_HOST"],
 mask=mask,
 environment="prod", # lowercase, dashes, 40 chars, must not start with "langfuse"
 sample_rate=float(os.getenv("LANGFUSE_SAMPLE_RATE", "1.0")),
)

# 2. Graph. The decision node carries "agent" in its name on purpose.
class State(TypedDict):
 messages: Annotated[list, add_messages]

graph = StateGraph(State)
graph.add_node("agent_planner", model_node)
graph.add_node("tools", ToolNode(TOOLS))
graph.add_edge(START, "agent_planner")
graph.add_conditional_edges("agent_planner", tools_condition)
graph.add_edge("tools", "agent_planner")
app = graph.compile(checkpointer=checkpointer)

# 3. Per turn: deterministic trace seeded with thread and turn, session = thread.
def run_turn(thread_id: str, turn: int, text: str, user: str):
 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {
 "callbacks": [handler],
 "configurable": {"thread_id": thread_id},
 }
 with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["agent", "support"],
 ):
 out = app.invoke({"messages": [HumanMessage(text)]}, config=cfg)
 return trace_id, out

# 4. Resume after human approval: same seed, same trace.
def resume_turn(thread_id: str, turn: int, decision):
 from langgraph.types import Command

 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {"callbacks": [handler], "configurable": {"thread_id": thread_id}}
 return app.invoke(Command(resume=decision), config=cfg)

# 5. Score the turn. Scores do not travel over OTLP: they have their own route.
get_client().create_score(
 trace_id=trace_id,
 name="resolved",
 value=1,
 data_type="BOOLEAN",
)

Two notes on what is not in the example. sample_rate is head sampling and per trace, so it samples whole turns, which is what you want: half an agent trace is no use to anybody. And if the process is a web server, the client is built once at startup, not per request, whereas the handler can be per turn with no problem, because with the seeded identifier it no longer holds state worth preserving.

Agent instrumentation checklist

Nodes representing a model decision carry agent in the name. The rest do not.
Every turn has a deterministic trace identifier seeded with thread and turn number.
LangGraph’s thread_id goes as Langfuse’s session_id, always.
Trace attributes are set with propagate_attributes, not as metadata on a subgraph.
The gateway’s model alias is registered in the Langfuse model table, with a price.
Tools doing work outside LangChain are instrumented with the Langfuse SDK, not with bare OpenTelemetry.
There is an alert on ingestion queue depth, and the average number of steps per turn is known.
Someone has measured, on the actual installation, how many observations and how many bytes a real turn produces. The figures in this article come from a synthetic bench.
Sampling is set through an environment variable, so it can be lowered without deploying code.

Traps

Type AGENT does not mean there is an agent. It means somebody wrote agent in the name. And the other way around: a whole graph without that string produces not one agent-type observation.

Putting langfuse_session_id in a subgraph’s metadata does nothing. It is only read on the root run.

An interrupt splits the trace in two. Unless you seed the identifier. And with several replicas, the handler’s in-memory state is never going to save you, because the resume can land on another pod.

Your OpenTelemetry spans do not arrive. The default filter drops them without warning if the scope is not in the list and there are no gen_ai.* attributes.

The number to watch is not users, it is steps per turn. Volume grows with the square of the steps, and the steps are decided by the prompt, not by the infrastructure.

You store megabytes and the interface shows you a thousand characters. The materialised view’s truncation to two hundred characters and the thousand-character read ceiling are still there. You pay for the full ingestion and see a clipping, unless you open the specific observation.

Cost comes out at zero with the gateway in front. Model alias unknown to the price table.

flush_at above the queue size blows up when constructing the client. With an OpenTelemetry ValueError that does not mention Langfuse anywhere.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front (this article): instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Own measurement of 13 September 2026: LangGraph 1.2.11, langchain-core 1.6.3 and Langfuse Python SDK 4.15.2, with OpenTelemetry’s InMemorySpanExporter and GenericFakeChatModel, no network.
Langfuse SDK 4.15.2 source: langfuse/langchain/CallbackHandler.py, langfuse/_client/span_filter.py, langfuse/_client/span_processor.py, langfuse/_client/span.py.
PyPI downloads over the last thirty days via pypistats, checked on 13 September 2026.
GitHub stars for LangGraph, CrewAI and the OpenAI Agents SDK, checked on 13 September 2026.

Langfuse v4, day 2 (1 of 10): the data model changed underneath, and what really goes into a trace

Sat, 12 Sep 2026 07:00:00 +0200

First article in a series of ten about operating Langfuse v4 in production. The architecture of version 3 and its ten performance levers were already covered in Langfuse from the inside; the seam with the gateway, in the operational pair. This series is about day 2. Verified against Langfuse 4.35.0, commit of 11 September 2026.

TL;DR

Version 4 is observations first. There is no trace table. A trace is a set of observations that share an identifier, and the trace attributes live denormalised in every row: name, user, session, tags, version, release and environment (clickhouse/migrations/canonical/0039_create_events_full.up.sql). The whole intuition of “first I create the trace and then I hang things off it” stops describing the storage.

There are ten observation types, not three. Alongside the classic SPAN, EVENT and GENERATION come AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (packages/shared/src/domain/observations.ts:5). Using the right type changes what the interface knows how to group.

The real limits are in the code and they do not match what people assume. The body of a request to the classic ingestion route is capped at 4.5 MB (api/public/ingestion.ts:40). The OTLP body, at 512 MiB, compressed and decompressed, with a 413 when it is exceeded. An OTLP span above 9.5 MB is only written to the log, not rejected. Identifiers accept 800 characters. The trace name, 1,000.

The ingester looks for the user in five places and the session in six. With strict precedence order, starting from Langfuse’s own attributes and ending in the metadata of other frameworks. Among the session ones sits gen_ai.conversation.id, which is the OpenTelemetry standard, so a conformant instrumentation works without translation.

Scores do not travel over OTLP. Zero occurrences of the word across the 3,864 lines of the OTLP ingestion processor. They come in through the classic ingestion route, through the scores endpoint or through the interface. Any design that assumes an evaluator can write its score down the same channel as the trace is wrong.

The classic masking function does not cover what almost everyone thinks it covers. It only acts on data that passes through the SDK calls. The attributes produced by a third-party automatic instrumentation come out unmasked. There is a new function that does act on the batch attributes, but it leaves out span events, which is where the recent conventions put the messages.

Filtering by tags or by status message is a scan. In the listings table, the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the dates, the model and the metadata names. The name, the tags, the version and the status message have no index.

You are here: OBSERVE, day 2

The previous series about this pair covered selection and setup. The six-service architecture explained version 3; tracing with OpenTelemetry explained the pipeline; the pair with the gateway explained where the trace gets lost between the two pieces.

What is left uncovered is the work of the next three years: migrating, sizing ClickHouse, retaining and deleting, taking backups, alerting, getting the data out. This series is about that, and it starts with the question that conditions everything else, which is what gets stored.

The analogy: the archive that changed system

An archive with a folder per case file and documents inside each folder is easy to understand. You open the folder, you look at what is there. The problem shows up when you have to find every document of one kind over the last two years: you have to open every folder.

Large archives solve that the other way around. There is no folder: each document carries, copied into its header, the case reference, the client name and the date. Searching by client is immediate. Reconstructing a case file is grouping by reference. In exchange, each document weighs a little more, and what gets written into that header matters a great deal, because everything you can search for comes from there.

Langfuse v4 made exactly that change. What used to be a folder with attributes of its own now travels repeated in the header of every document. The two operational consequences of this article come from there: which fields are worth filling in and which fields are not worth filling with content.

The data model, as it is in the code

The tables

Three new pieces in ClickHouse:

events_full (migration 0039): the table of record, with the full input and output, compressed with ZSTD level 3 and indexed with full-text indexes over the lowercase versions. ReplacingMergeTree engine ordered by event timestamp and deletion flag, partitioned by month of the start time.
events_core (migration 0040): the listings table, lighter.
events_core_mv (migration 0041): the materialised view that fills the previous one, and which truncates to two hundred characters the input, the output and each metadata value.

That truncation to two hundred characters is the first thing to know. What you see in a listing is not what is stored; it is a clipping. And when reading a full trace a second cap applies on top, LANGFUSE_SERVER_SIDE_IO_CHAR_LIMIT, with a default value of a thousand (packages/shared/src/env.ts:513).

In the events_full table there is a configuration line that says a lot: index_granularity_bytes = '64Mi', with the comment that it avoids very small granules because of large rows. It is the schema itself acknowledging that the rows of this system are bulky.

The ten observation types

SPAN, EVENT, GENERATION, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (domain/observations.ts:5), with their corresponding ingestion events (server/ingestion/types.ts:279).

For a platform with agents, the difference between marking a tool call as a generic SPAN or as a TOOL is the difference between being able to count tool calls per session and having to reconstruct it by hand. It is worth reviewing the instrumentation with this list in front of you.

The level is still DEBUG, DEFAULT, WARNING and ERROR (domain/observations.ts:31), with a free-form status message.

The limits, with their source

Field	Limit	File
Trace, span and observation identifiers	800 characters, no carriage return	`ingestion/types.ts:10`
Trace name	1,000 characters	`ingestion/types.ts:428`
Environment	40 characters, pattern of lowercase, digits, hyphen and underscore	`ingestion/types.ts:226`
Text-type score	500 characters	`domain/scores.ts:44`
Score config name	35 characters	`domain/score-configs.ts:11`
Classic ingestion body	4.5 MB per request	`api/public/ingestion.ts:40`
OTLP body	512 MiB, adjustable	`web/src/env.mjs:583`
OTLP span considered large	9.5 MB, only logged	`packages/shared/src/env.ts:193`
Name, status message, version, release, tags	no declared limit	`ingestion/types.ts:424`
Metadata	no declared limit	`ingestion/types.ts:425`

The last two are the dangerous ones, because they have no schema cap and the real ceiling ends up being the one on the request body. A three-megabyte metadata value passes validation and reaches ClickHouse.

A note on the langfuse prefix in the environment field: it is reserved for the product’s own internal traces and is stripped idempotently from public inputs (ingestion/types.ts:228), including the case of someone trying to sneak it in duplicated. The default value is default.

On rate limits: in the cloud there are per-plan quotas, but in self-hosted they are not applied. The service returns directly that there is no limiting (features/public-api/server/RateLimitService.ts). It is good news for bulk exports and bad news for anyone relying on that mechanism as protection.

The two instrumentation routes

What gets lost over OTLP, and it is one specific thing

The check is simple and the result is blunt: there is not a single occurrence of the word score in the OTLP ingestion processor, a file of 3,864 lines. Scores only come in through three places: the score-create event of the classic ingestion route, the POST /api/public/scores endpoint, or the interface.

This has a design consequence worth being clear about before building evaluators: the trace can travel through the OpenTelemetry collector with everything good that brings, on-disk queue, retries and sampling, but the score an evaluator puts on it has to go down a different channel and with the project keys.

Everything else does travel over OTLP. Whatever is not in the mapping lists is kept as metadata converted to a string, visible but outside the indexed fields.

The precedence chain, which is the map you need to have

Verified in packages/shared/src/server/otel/OtelIngestionProcessor.ts:

User identifier (line 2545), in this order: langfuse.user.id, user.id, the observation metadata, the trace metadata, and lastly the Vercel SDK metadata.

Session identifier (line 2565): langfuse.session.id, session.id, gen_ai.conversation.id, observation metadata, trace metadata, Vercel metadata.

Environment (line 2383): langfuse.environment, deployment.environment.name, deployment.environment; for each key the span attributes are checked first and the resource attributes after.

Model (line 2753): langfuse.observation.model.name, gen_ai.response.model, ai.model.id, gen_ai.request.model, llm.response.model, llm.model_name, model.

Level (line 1250): the native attribute, with accepted aliases where SUCCESS and OK become DEFAULT, WARN becomes WARNING and FATAL or CRITICAL become ERROR; if there is no attribute and the span status is an error, the result is ERROR.

Tags (line 3227): langfuse.trace.tags, langfuse.tags, observation and trace metadata, Vercel metadata, and tag.tags.

Input and output (line 1790): a chain of some twenty frameworks in strict order, starting from Langfuse’s own attributes and going through Genkit, the Vercel SDK, the standard inference operation details event of the conventions from 1.37 onwards, the per-role message events, the prompt and completion attributes, Logfire, MLflow, TraceLoop, OpenInference and the gen_ai.input.messages and gen_ai.output.messages attributes.

That this chain has twenty entries says something about the real state of the conventions: it was already noted that the generative AI conventions were extracted into their own repository without any published version. While that remains the case, the backend has to guess, and it guesses with a precedence list.

The practical part: if Langfuse’s own attributes are emitted, they always win. If you want backend-neutral instrumentation, gen_ai.conversation.id and deployment.environment.name are the two standard attributes that Langfuse does understand.

Metadata, and how it ends up in the table

A metadata value is flattened into two parallel arrays, one of names and one of values, with dot-separated paths and everything converted to a string (otel/utils.ts:239, applied in worker/src/services/IngestionService/index.ts:363). A nested object with twenty keys is twenty entries in each array.

There is a cap of 10,001 positions when reconstructing arrays from OTel attributes (OtelIngestionProcessor.ts:271), which is high but it exists.

Sessions, users and cardinality

A session is a text column with a Bloom filter index at 1 %. The same goes for the user. The Postgres table that accompanies sessions stores only whether it is bookmarked, whether it is public and its environment; there is no counter and no cap. There is no limit on traces per session in the code, and the documentation recommends session identifiers below 200 ASCII characters.

The cardinality difference matters and it is not intuitive: user and session are plain text, so high cardinality does not break the schema, it only inflates the Bloom filters. The environment, by contrast, is declared low cardinality. Generating a dynamic environment value per tenant degrades ClickHouse measurably.

Aggregated cost comes out of a cost details map with materialised input, output and total columns. The classic sessions endpoint is deprecated; the replacement is the version 2 observations one with a filter.

Scores: five types and three sources

The types are NUMERIC, CATEGORICAL, BOOLEAN, CORRECTION and TEXT (domain/scores.ts:45). The boolean requires a value of zero or one. The correction does not accept an associated config. The text one is capped at 500 characters.

The sources are API, EVAL and ANNOTATION. The EVAL source is reserved for the internal evaluators: the public API only accepts the other two. Annotation scores require an associated config except for the correction one.

On what costs money, a clarification that corrects a widespread belief: in 4.35.0 the entitlement for the number of model-based evaluators is unlimited on every plan, including the open edition (features/entitlements/constants/entitlements.ts:60). What is limited by plan is the number of annotation queues and the days of data access. And there is a version nuance: trace-level evaluators are discontinued in version 4.

Masking, and what it does not cover

This is the part to read twice if the deployment has to comply with anything.

The classic Python SDK function has the signature masking_function(*, data, **kwargs) and covers only the data that passes through the SDK calls: observation creation, update, and setting the trace input and output. It does not cover the raw OTel attributes produced by a third-party automatic instrumentation.

The replacement is a function that acts on the attributes and the resource attributes in the export batch. That one does reach automatic instrumentation, with a limitation declared in the documentation: it cannot change the span name, the identifiers, the parent relationship, the resource attributes, the events or the links.

And there is the gap that matters: the generative AI conventions from 1.37 onwards put messages and responses into span events. That is, the route by which prompts arrive in a modern instrumentation is precisely the one the new masking does not touch.

For a deployment with personal data, the conclusion is that client-side masking is not enough and a redaction processor has to go into the collector. It is the same reasoning that led in the previous article to recommending the collector in the middle, and it adds to the already documented problem of MCP tool arguments being written in clear, bypassing the message redaction switch.

On the server side there is no safety net: Langfuse has no server-side personal data masking. The encryption key that gets configured protects secrets, that is, model provider credentials, integration headers and SSO configurations. Trace inputs and outputs are not encrypted at application level.

Volume control

Three levers, in order of how high up they act:

In the SDK: sample_rate, or the LANGFUSE_SAMPLE_RATE variable, with head sampling at trace level. If the trace is not sampled, neither its observations nor its scores travel. LANGFUSE_TRACING_ENABLED set to false switches the whole thing off.

In the collector: probabilistic sampling and tail sampling. It is the only layer where you can decide based on the outcome, that is, keep every error and a fraction of the rest.

In the server: a deterministic per-project sampling, with a SHA-256 hash of the trace identifier (server/ingestion/sampling.ts). Deterministic per trace, so it does not cut traces in half.

What not to put into a trace

With the schema in front of you, the list is concrete.

Do not filter by tags or by status message. In the listings table the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the creation and update dates, the model name, the experiment identifier and the metadata names. The name, the tags, the version, the release and the status message are not. Filtering by them scans.

Do not use metadata as a store. It gets flattened into string arrays, truncated to two hundred characters in the listings table but stored whole in the full table, and on top of that it is indexed with a full-text index segmented on non-alphabetic characters. Bulky metadata multiplies the inverted index, not just the data.

Do not store full prompts without thinking about it. The input and the output carry ZSTD level 3 compression and two full-text indexes over their lowercase versions. Storing the entire context of a 142,000-token agentic call on every turn inflates the inverted index disproportionately. Client-side truncation is the cheap lever.

Do not use the environment to separate tenants. It is low cardinality, it has a 40-character cap and it isolates nothing: same project, same keys, same limits, same retention. It is there to segment dashboards. Real isolation means separate projects, as was already argued in the article on the seams.

Version 3 and the calendar date

The constant is in the code: V3_SUNSET_DATE = "2026-11-16" (features/public-api/server/deprecations.ts:11).

It is worth being precise about what it means, because the public documentation and the code do not say exactly the same thing. The code comment is literal: the classic ingestion route is never switched off and continues to accept score events; trace and observation events fail only in version 4’s exclusive write mode, not in the mixed or classic modes. And when they fail, they fail per event, not per batch.

The x-langfuse-ingestion-version: 4 header activates direct writing to the events table, which is the real-time route. Without it the deferred path is taken. Recent SDKs send it by themselves. Any value above four is rejected.

As for names, trace-level input and output attributes are deprecated in favour of the observation ones on the root span. The classic read endpoints carry a deprecation marker in the response body, and their replacements are version 2 of observations, version 3 of scores and version 2 of metrics. And dataset runs are renamed experiments.

There is a curious piece that deserves a mention because it shows up in the Redis dashboards and is misleading: a queue called version 4 classic API usage, which ingests nothing. It is a periodic task every fifteen minutes that scans the ClickHouse query log to detect which projects are still using the old API, and materialises the result into Redis for the interface’s migration notice.

How this looks in code

Two pieces, the client one and the collector one.

On the application side, what changes compared with the usual instrumentation is the observation type and where the identity comes from:

from langfuse import get_client, propagate_attributes

lf = get_client()

# identity propagates by context, not observation by observation
with propagate_attributes(
 session_id=id_conversacion, # the same across every turn
 user_id=sujeto_del_token, # from the claim, not from the body
 environment="produccion",
 tags=["soporte", "n2"],
):
 # the type matters: the interface groups by it
 with lf.start_as_current_observation(
 as_type="agent", name="resolver-incidencia"
 ) as raiz:

 with lf.start_as_current_observation(
 as_type="tool", name="buscar_en_inventario"
 ) as herramienta:
 resultado = mcp.call("buscar", args)
 # truncate here, not on the server
 herramienta.update(output=str(resultado)[:4000])

 # the score goes down its own channel, never over OTLP
 lf.score_current_trace(name="resuelta", value=1, data_type="BOOLEAN")

The as_type parameter accepts nine literals: span, generation, embedding, agent, tool, chain, retriever, evaluator and guardrail. The tenth type in the model, the event, has its own constructor because it does not open a scope.

On the collector side, the redaction processor that compensates for what the SDK masking does not reach:

processors:
 redaction:
 allow_all_keys: true
 blocked_values:
 - '\b\d{8}[A-HJ-NP-TV-Z]\b' # national ID number
 - '\b[\w.+-]+@[\w-]+\.[\w.]+\b' # email
 summary: silent
 tail_sampling:
 policies:
 - name: errores
 type: status_code
 status_code: {status_codes: [ERROR]}
 - name: resto
 type: probabilistic
 probabilistic: {sampling_percentage: 5}

exporters:
 otlphttp/langfuse:
 endpoint: http://langfuse-web.observabilidad.svc:3000/api/public/otel
 headers:
 Authorization: "Basic ${LANGFUSE_BASIC}"
 x-langfuse-ingestion-version: "4"
 sending_queue:
 enabled: true
 storage: file_storage
 retry_on_failure:
 enabled: true

The endpoint carries the base path without the signal suffix: the exporter adds its own. And the on-disk queue is the main reason for putting the collector in the middle, because the SDK’s batch processor drops when it fills up.

Instrumentation checklist

Review the type of every observation against the list of ten. Tool calls as TOOL, agents as AGENT, guardrails as GUARDRAIL.
Decide where the user identifier comes from and set it in a single place, high up the precedence chain.
Emit gen_ai.conversation.id if you want neutral instrumentation, or Langfuse’s own attribute if coupling does not matter.
Set the environment with deployment.environment.name, with few distinct values and never one per tenant.
Get scores out through their own channel, with the project keys, and do not expect them to travel with the trace.
Truncate input and output on the client before sending them, especially with long agentic contexts.
Put a redaction processor in the collector, because the SDK masking leaves out span events.
Do not filter by tags in dashboards that get queried often.
Send the ingestion version header, or use recent SDKs that send it by themselves.
Review the size of the metadata: there is no schema cap, the cap is the one on the request body.

Traps and things that are not what they look like

What you see in a listing is truncated to two hundred characters by the materialised view, and the full read has another cap of a thousand. The data is whole in the full events table.

Metadata has no size limit in the schema. A blob of megabytes passes validation.

An OTLP span over 9.5 MB is not rejected, it is only logged. There is no signal back to the client.

The environment is declared low cardinality. One value per tenant degrades ClickHouse.

Scores do not travel over OTLP. It is by absence in the code, not by configuration.

The classic masking does not cover automatic instrumentation, and the new one does not cover span events, which is where the recent conventions put the messages.

Langfuse does not encrypt trace inputs or outputs. The encryption key protects integration credentials.

In self-hosted there are no rate limits on the API. It is no use as protection.

The classic ingestion route is not switched off on 16 November, it changes behaviour depending on the write mode, and it fails per event instead of per batch.

Model-based evaluators are not limited by plan in this version, contrary to what is usually assumed. What is limited are the annotation queues and the days of data access.

The series: the ten articles

This is the first. The order is designed so that each one can be read on its own and so that together they cover the whole of day 2.

What goes into a trace (this article): version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Anuncio de Langfuse v4, 17 de agosto de 2026.
Compatibilidad y migración, consultado el 12 de septiembre de 2026.
Sesiones, muestreo y enmascarado, consultados el 12 de septiembre de 2026.
Integración nativa con OpenTelemetry, consultado el 12 de septiembre de 2026.

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

Mon, 07 Sep 2026 18:00:00 +0200

This post opens the operational track of the control layer. The previous articles left the pieces chosen and assembled: choosing the OSS gateway and the L7 inference router explain why LiteLLM ends up in front of the fleet, and Langfuse from the inside takes apart the backend that receives the traces. Here we deal with the seam between the two, which is where the work of the next three years lives.

TL;DR

The LiteLLM plus Langfuse pair is configured with a list of callbacks and two API keys, and from there four matters decide whether the dashboard is good for operating or only for showing in a demo.

There are three integration routes and they are not equivalent. The native langfuse callback is tied to the Python SDK v2 (langfuse>=2.59.7,<3.0 in LiteLLM’s pyproject.toml), which writes against the legacy ingestion that Langfuse Cloud retires on 16 November 2026. The langfuse_otel preset exports over OTLP and is the path recommended in the current documentation of both projects. The OTel v2 route, behind the LITELLM_OTEL_V2=true flag, changes span names, leaves the success status at UNSET and, above all, its Langfuse mapper does not read the request metadata: session_id, trace_user_id, tags and trace_id stop arriving.

The cost of a self-hosted model comes out as zero unless a price is registered. get_model_info() raises an exception for a model that is not in the pricing map, completion_cost() propagates it, the logger catches it, stores it in a debugging field and returns None, and the payload turns it into 0.0. There is no warning at the default log level. A team’s budget is never exhausted because its spend is always zero.

Client-gateway-engine correlation does not come out of the box. LiteLLM does continue the client’s trace if a traceparent arrives, and it also extracts from it the 32-hex trace-id to use as litellm_trace_id, which makes the Langfuse identifier equal to the W3C one. Upstream the story changes: forward_traceparent_to_llm_provider is False by default, the function that applies it returns early if no otel logger is loaded, and when it does act it forwards the client’s original header, so the vLLM span ends up a sibling of LiteLLM’s rather than a child. Add to that the fact that vLLM exports OTLP over gRPC by default and Langfuse does not accept gRPC.

Between the request and the dashboard there are four bounded queues that drop events. LiteLLM’s logging thread pool (100 threads, 10,000 pending tasks, dropping once full), the asynchronous LoggingWorker (50,000, and once full it aggressively flushes half), OTel’s BatchSpanProcessor (2,048 spans by default) and Langfuse’s ClickhouseWriter, which drops rows after exhausting the retries with no dead-letter queue. None of the four applies back pressure on the request, which is exactly what you want, and none of them guarantees delivery, which is exactly what an auditor asks about.

The rest of the article develops the four points with configuration applicable to a generic 4×H100 cluster with vLLM, the overhead figures the project itself publishes, and the honest limit of what this data sustains before ENS or ISO/IEC 42001.

You are here: the seam between DEPLOY and OBSERVE

In the seven-layer stack, LiteLLM lives in the gateway layer and Langfuse in the observability one. The separation is clean on the diagram and false in operation: the gateway is the only point in the system that sees the complete request before chopping it up and the complete response after reassembling it, so it is also the only reasonable place from which to emit the trace. The inference engine sees tokens and latencies; it does not see who is asking or how much that team has spent this month.

That turns LiteLLM into a telemetry producer as well as a traffic distributor, and moves onto the gateway a responsibility that was not in the original design: if the callback fails, the request is served all the same and nobody notices that the dashboard has been incomplete for two days.

The analogy: the toll booth that issues the ticket

A motorway toll booth does two independent jobs. It raises the barrier, which is its visible function, and it issues a ticket with number plate, time, section and amount, which is the function the concession lives off. The two share a booth and do not share a destination: the barrier responds in a hundred milliseconds because there is a car waiting, and the ticket travels to a processing centre over a slower channel that batches thousands of records before sending them.

The design is well thought out as long as its consequence is understood. If the processing centre goes down, cars keep going through. If the ticket queue fills up, tickets are thrown away, traffic is not stopped. The concession accepts losing records rather than causing a jam, and charges on a statistical sample it knows to be imperfect.

LiteLLM and Langfuse work the same way. The four queues in the corresponding section are the slow channel, and all of them prefer to drop rather than block. Whoever builds the pair inheriting the complete analogy gets the operation right. Whoever thinks the ticket is an accounting entry ends up explaining to an auditor why traces from a Tuesday afternoon are missing.

Three routes, and one with an expiry date

As of September 2026 three paths coexist for getting a request from LiteLLM to Langfuse. LiteLLM’s documentation already labels the first as legacy.

Route	How it is enabled	Transport	Status
Native `langfuse` callback	`success_callback: ["langfuse"]`	Langfuse Python SDK v2, legacy ingestion	Maintained for compatibility. Tied to `langfuse<3.0`
`langfuse_otel` preset (v1)	`callbacks: ["langfuse_otel"]`	OTLP/HTTP against `/api/public/otel`	Recommended by both projects
OTel v2 route	`callbacks: ["langfuse_otel"]` + `LITELLM_OTEL_V2=true`	OTLP/HTTP, new mapper	Opt-in. Changes semantics and loses metadata

The date that orders the decision is 16 November 2026. Langfuse’s compatibility matrix marks the legacy batch ingestion of the Python SDK v2 as retired on that date for Cloud, at which point trace ingestion through that route stops working. On self-hosted the endpoint survives, but a v4 server in events_only mode returns 400 for every event type except score-create and sdk-log.

Since LiteLLM’s native callback is pinned to langfuse>=2.59.7,<3.0 in its own pyproject.toml, and its code calls APIs that only exist in SDK v2 (self.Langfuse.trace(...), trace.generation(...)), there is no way to move up an SDK without changing route. Anyone who today has success_callback: ["langfuse"] against Langfuse Cloud has a migration with a date on it. Anyone who has it against a self-hosted Langfuse has one with some slack, because v3 receives security patches until the end of January 2027, but they have one all the same.

One detail of the OTLP route worth checking in older deployments: the x-langfuse-ingestion-version: 4 header is the one that enables real-time ingestion. Without it the traces arrive, but with a delay that Langfuse’s documentation puts at around ten minutes (the compatibility page says fifteen, and that discrepancy remains unresolved in their own documents). LiteLLM started sending it in version 1.95.0, of August 2026. A proxy older than that version against a Langfuse v4 produces the classic symptom: the dashboard works, but it is always behind and nobody knows why.

The configuration that works today

For a generic 4×H100 cluster with vLLM behind it and Langfuse self-hosted in the same Kubernetes, the OTLP route in its v1 variant is the one that gives the most functionality with the fewest surprises.

model_list:
 - model_name: llama-70b
 litellm_params:
 model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
 api_base: http://vllm-llama70b.inference.svc.cluster.local:8000/v1
 api_key: os.environ/VLLM_API_KEY
 # Price derived from the cluster's €/GPU-hour. Without this, spend = 0.0
 input_cost_per_token: 0.00000031
 output_cost_per_token: 0.00000089

litellm_settings:
 callbacks: ["langfuse_otel"]
 turn_off_message_logging: false # a conscious decision, see the PII section
 langfuse_default_tags: ["model_group", "user_api_key_team_alias", "cache_hit"]

general_settings:
 disable_error_logs: true # provider errors inflate SpendLogs
 proxy_batch_write_at: 60

The credentials travel through the environment. LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST are enough: the preset builds the Authorization: Basic base64(public:secret) header and adds the ingestion version one, so there is no need to touch OTEL_EXPORTER_OTLP_HEADERS by hand. The preset’s default host points to the US region of Langfuse Cloud, a relevant detail for anyone building a platform with data sovereignty requirements: if the variable is not defined, the telemetry leaves Spain without anyone having decided it.

For real multi-tenancy, LiteLLM supports per-team Langfuse credentials (POST /team/<id>/callback with callback_vars) and per-key ones (metadata.logging[] when generating the key), and the key’s wins over the team’s without mixing. There is a limitation that breaks the architecture of anyone who planned to separate tenants across different Langfuse instances: under OTel v2, a per-key or per-team langfuse_host does not move that tenant to another host, because the exporter fixes the destination at startup and the only thing that varies per tenant are the headers.

And one operational warning with years of history behind it: MAX_LANGFUSE_INITIALIZED_CLIENTS is 50 by default because each Langfuse client starts a thread. The comment in LiteLLM’s code is explicit about the incident that motivated it, with the CPU at 100% from repeated initialisations. A deployment with per-team credentials and many teams hits that ceiling.

What is lost when moving to OTel v2

The LITELLM_OTEL_V2=true flag brings a step up in quality in the proxy’s general instrumentation, and at the same time a concrete regression on the path to Langfuse that the documentation acknowledges without underlining it.

The Langfuse mapper of the v2 route emits langfuse.observation.type, langfuse.observation.model.name, the observation identifier, the model parameters, input and output, usage_details and cost_details, plus langfuse.trace.metadata.team_id and team_alias. It does not emit session.id, nor user.id, nor langfuse.trace.tags, nor langfuse.trace.id, nor the generation name, nor parent_observation_id. LiteLLM’s documents put it this way: those attributes are set by the preset from the request and the response, not from metadata supplied by the client.

The practical consequence is that an application that today groups conversations by sending metadata: {"session_id": "...", "trace_user_id": "..."} in the request body stops grouping them when the flag is enabled, with no error and no warning. The traces keep arriving, loose.

The other v2 changes are semantic and affect alerts already written. The root span goes from being called Received Proxy Server Request to being named after the route (POST /v1/chat/completions), the inference span goes from litellm_request to {operation} {model}, the guardrail spans stop hanging off the inference span and hang off the root instead, and the success status stays at UNSET instead of OK. Any rule that fires on status == OK stops firing. The way back is to remove the flag and restart.

Cost: why the trace says zero

This is the part that takes the most time to diagnose and the least to fix.

LiteLLM computes the cost of a request from a per-model pricing map. For commercial provider models the map comes filled in. For hosted_vllm/whatever, which is all there is in an in-house inference factory, the map is empty, and the failure path is as follows:

litellm.get_model_info() raises an exception with the message This model isn't mapped yet.
completion_cost() and response_cost_calculator() propagate the exception.
Logging._response_cost_calculator() catches it, writes a StandardLoggingModelCostFailureDebugInformation into response_cost_failure_debug_information, logs it at debug level and returns None.
The standard logging payload does llm_response_cost = raw_response_cost or 0.0.

The system delivers a spend of zero, with no visible error at the default log level, and writes it into the spend column of LiteLLM_SpendLogs. The per-key, per-team and per-organisation budgets are evaluated against that zero and are never exhausted. The Langfuse dashboard shows a total cost of zero for everything self-hosted, which is precisely the part you wanted to measure.

Registering the price

The pricing keys are accepted both in litellm_params and in model_info, and those in litellm_params win when they are in both places. The complete set has 91 fields, but for a deployment with vLLM two are enough, and there are three things to know about them.

The first is that the price is per individual token, in USD. There are no per-thousand variants: searching for per_1k or per_1000 in the pricing model returns nothing. A three-orders-of-magnitude error here is easy to make and hard to see, because it produces plausible figures.

The second is that input_cost_per_request exists in the schema and is not applied in chat calls: a model registered with that field returns a cost of 0.0. To charge per request it has to be converted into a cost per token.

The third is that input_cost_per_second needs the response time and is computed as price times response_time_ms / 1000, so outside the proxy’s request path it silently returns zero.

The number that goes into those two keys comes from the article on FinOps and multi-tenancy with LiteLLM and the one on cost per token and per request: amortisation plus energy divided by tokens served. The example arithmetic, to set the order of magnitude: a cluster whose fully loaded cost is 9.60 USD per hour sustaining 3,000 output tokens per second produces 10.8 million tokens an hour, that is 0.00000089 USD per output token. That is the value that goes into output_cost_per_token. If the cluster’s average utilisation is 30 % and not 100 %, the divisor changes and the price triples, which is the same lever that article develops.

Before sending traffic, the /spend/calculate endpoint accepts the same parameters as completion_cost and returns the computed cost. It is the cheap way to check that the price resolves.

How that number reaches Langfuse, and who wins

The two integration routes write the cost into different fields. The native callback sends it as usage.total_cost of the v2 schema, and duplicates it in the trace metadata as litellm_response_cost. The OTLP preset emits langfuse.observation.cost_details as a JSON string of the form {"total": cost}, and langfuse.observation.usage_details with input, output and total.

The rule on the Langfuse side is documented and is the right one for this case: when there is ingested cost and inferred cost, the ingested one takes priority. Langfuse only computes from its own pricing table when the emitter sends no cost, and changes to a model definition apply to new generations, with no retroactive recalculation. A self-hosted model therefore needs no pricing definition in Langfuse: it is enough for LiteLLM to send cost_details.

One detail remains that explains a puzzling symptom. Langfuse stores the ingested cost and the computed one in separate columns, totalCost and calculatedTotalCost, and the session header totals add up only the first. A mixed deployment where part of the traffic brings cost and part lets Langfuse compute it shows sessions at zero while the individual generations show cost. There is an open issue about it.

The tokens of a streaming response

Almost all of an assistant’s traffic is streamed, and there the OpenAI usage object only appears if the client asks for stream_options.include_usage. LiteLLM solves this by injecting the option on its own: on the asynchronous completion routes with stream: true, if the caller did not set it and all the candidate deployments support it, it adds it and marks _litellm_strip_stream_usage so the usage block is used in the cost calculation and not returned to the client. The general_settings.always_include_stream_usage flag forces or disables that behaviour, and is worth setting to True when capability detection does not recognise the vLLM deployment.

When the usage is missing anyway, LiteLLM reassembles the response with stream_chunk_builder() and counts tokens with the tokeniser. It is an approximation, and it produces the second path towards a cost of zero. The two paths to zero are told apart by looking at total_tokens in LiteLLM_SpendLogs: zero tokens with a correct price is a usage problem, correct tokens with zero cost is a price problem.

Correlation: from the client to the engine

The operational question that justifies the whole setup is being able to take a user complaint, find their request and see where the time went. That requires the trace identifier to survive three hops.

What does work

If the client sends traceparent, LiteLLM’s spans hang off the existing trace. In addition, and independently of the OTel subsystem, litellm_pre_call_utils.py extracts by regular expression the 32-character hexadecimal trace-id and assigns it to data["litellm_trace_id"] as a last resort, and takes the session_id from the W3C baggage header. That makes the Langfuse trace identifier equal to the W3C trace identifier, which is the most robust correlation key available: with it, an identifier copied from the application log opens the trace in Langfuse with no translation.

For the routes that are not OTel v2, moreover, headers prefixed with langfuse_ inject per-request metadata. langfuse_trace_id, langfuse_trace_user_id and langfuse_trace_metadata are documented, but the code strips the prefix and writes the rest into the metadata, so any body key works as a header. Two warnings about this: header values arrive as strings and are not interpreted as JSON, and Nginx drops headers with underscores by default (underscores_in_headers off), which produces the symptom that the metadata works locally and disappears behind the ingress.

What does not work without effort

Towards the engine, propagation is off. The flag is litellm.forward_traceparent_to_llm_provider, it is False, and its implementation has three behaviours to know about before enabling it:

The function returns immediately if open_telemetry_logger is None. A deployment with callbacks: ["langfuse_otel"] and without the generic otel callback propagates nothing, whatever the flag is set to.
When it does propagate, it copies the client’s traceparent header as is, instead of injecting the context of LiteLLM’s active span. The vLLM span ends up hanging off the client’s span-id, a sibling of the gateway call’s span. The trace is one, the hierarchy is wrong.
If the client sends no traceparent, none is created.

Add to that the fact that LiteLLM’s general policy is not to forward client headers to the provider, and that the forward_client_headers_to_llm_api allowlist covers headers prefixed with x-, among which traceparent is not. This is the only route.

The vLLM side, and why the Collector stops being optional

vLLM enables traces with --otlp-traces-endpoint, and --collect-detailed-traces with model, worker or all adds forward and execute times, with a performance impact its own documentation warns about. The attributes it emits are in vllm/tracing/utils.py and are the most useful part of the setup: gen_ai.latency.time_in_queue, time_to_first_token, time_in_scheduler, time_in_model_prefill, time_in_model_decode, time_in_model_forward, time_in_model_execute, time_in_model_inference and e2e, plus the token counters. That breakdown is what turns a “it’s slow” into “the time goes in the queue, not in decode”, which is the conversation you want to have during an incident. It fits with what was already covered in instrumenting vLLM with OTel and with the metrics in DCGM and vLLM.

There are three constraints that decide the architecture:

vLLM exports over gRPC unless OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf is set, and its gRPC exporter is built with insecure=True hard-coded. Langfuse, for its part, does not support gRPC. And vLLM emits the old convention names (gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens), renamed in the specification to input_tokens and output_tokens, plus a gen_ai.latency.* family that was never standard and that the file itself documents as its own attributes until they are normalised.

An OpenTelemetry Collector in between solves all three at once: it receives gRPC, exports HTTP with the Basic header towards /api/public/otel/v1/traces, and renames attributes with a transform. It is also the only place where tail sampling can be done, which is the next point. And add the warning from the llm-d community, which holds for any deployment with Envoy or sidecars in between: OTEL_SERVICE_NAME has to be set per engine, because vLLM sends no service name and without it every span arrives as unknown_service.

The four serial queues

A request’s telemetry crosses four bounded buffers before it is queryable. All four drop once full and none blocks the request.

Queue	Capacity	What it does once full
LiteLLM logging thread pool	100 threads, 10,000 pending tasks	Drops the new ones with a rate-limited warning
Asynchronous `LoggingWorker`	50,000 events, concurrency 100	Aggressively flushes 50 % of the queue; `enqueue()` documents that it never blocks
OTel’s `BatchSpanProcessor`	2,048 spans (`OTEL_BSP_MAX_QUEUE_SIZE`), batch 512, sent every 5 s	Drops spans silently
Langfuse’s `ClickhouseWriter`	Bounded retries	Drops rows after exhausting retries, with no dead-letter queue

The first is documented in LiteLLM’s own code with a sentence that saves half an hour of discussion: logging is best-effort and, once the cap is reached, new entries are dropped with a rate-limited warning instead of being queued forever. The third is a barely visible consequence: LiteLLM builds the BatchSpanProcessor without adjusting max_queue_size, so the OTel SDK defaults govern, and no LiteLLM documentation mentions the OTEL_BSP_* variables that change them. The Langfuse SDK does the same on its own account: it sets batch size and interval, and leaves the queue size at the default.

On top of this comes a fifth loss point that is not a queue: LiteLLM does not call flush() per request on the native route, so a SIGKILL on the pod takes the last batch with it. On Kubernetes, this is an argument for a generous terminationGracePeriodSeconds and a preStop that gives some slack.

None of this is a defect. It is the right design decision for a piece that sits on the critical path of inference. What changes is what can be asserted with the resulting data.

What cannot be promised to the auditor

A dashboard built on this pipeline answers statistical questions well: what is the p95 of time to first token for the model group, which team consumes the most, what proportion of requests fails, how cost per request evolves. That is what it is designed for, and that is what it is good for.

It does not answer completeness questions well, which are the ones an ENS auditor or an ISO/IEC 42001 management system auditor asks. Faced with “show me that I have a record of every interaction with the AI system during March”, the honest answer is that the traces are a best-effort record with four documented points of silent dropping, with no dead-letter queue on the last hop and no durability guarantee on any of them.

The clean separation is the one already proposed in technical controls ENS, 42001 and the AI Act: the regulatory audit record is a different artefact, with a different write path, durability guarantees and retention of its own. In the pair this article deals with, the LiteLLM_SpendLogs table in Postgres is a much better candidate than Langfuse for that function, because it is a transactional write against a relational database, and because it already contains request_id, api_key in hashed form, team_id, organization_id, end_user, model, tokens, startTime, endTime, status and requester_ip_address. Writing the audit event from the same place and in the same transaction as the spend is the sensible route; delegating it to the tracing pipeline is not.

Here an operational tension appears that has to be resolved deliberately. LiteLLM’s performance recommendation for production includes disable_error_logs: true and, in very loaded deployments, disable_spend_logs: true, because provider errors inflate the table and concurrent writes against the same rows produce locks and exhaust the Postgres connections. Disabling spend_logs to gain performance and at the same time leaning on that table for the audit record are incompatible decisions. Above a thousand requests per second or more than ten instances, the way out is use_redis_transaction_buffer: true, which batches the spend updates in Redis before pushing them down to Postgres, watching the litellm_in_memory_spend_update_queue_size and litellm_redis_spend_update_queue_size metrics.

PII: switching off the content before it leaves

An LLM trace contains, by definition, the user’s prompt. In an on-premise deployment for a regulated customer, that content may be exactly what cannot leave the application’s domain, not even towards the observability backend in the same cluster.

The controls on the proxy side, from most global to most fine-grained:

litellm_settings.turn_off_message_logging: true cuts messages and responses in every callback, and keeps the metadata and the spend. The redaction replaces the content with the REDACTED_BY_LITELLM sentinel in messages, choices, reasoning_content, streaming deltas, tool call arguments and in the standard logging object.
The x-litellm-enable-message-redaction: true header enables redaction per request, and LiteLLM-Disable-Message-Redaction: true disables it. The second is checked first and wins over everything else, including the global configuration, which makes it a control to block at the ingress if the policy is that content is never logged.
redact_user_api_key_info: true removes the hashed token, the user_id and the team_id from the log.
Per request and per callback, metadata: {"mask_input": true, "mask_output": true}.
The body parameter "no-log": true disables logging of that call. It is a body parameter, not a header, and it can be disabled globally with global_disable_no_log_param: true so a client cannot make itself invisible.

On the OTel v2 route the policy is inverted towards the safe side: content is off by default and is opened with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, with values no_content, span_only, event_only and span_and_event. The documentation underlines that the gate is applied centrally and that a user’s request cannot force their prompt towards the backend while capture is disabled. It is also the stance the OpenTelemetry specification recommends, whose preferred pattern for production is to store the content in external storage and leave only the reference in the span.

One specific version to check in the inventory: below LiteLLM 1.98.0, with per-team Langfuse credentials, the team’s own credentials ended up inside its traces. That version changed the origin of the metadata emitted to the StandardLoggingPayload to cut it off. The same change removed a couple of dozen metadata fields (model_group, deployment, queue_time_seconds, attempted_retries and others), so it breaks saved dashboards and alerts that filter on them. It is an upgrade with work attached, and it has to be done anyway. The same applies to the general hardening of the stack, covered in hardening and secrets of the sovereign stack.

The price in latency

LiteLLM publishes two overhead figures that differ by a factor of thirty, and both are theirs.

The benchmarks page, with Locust, a thousand users with think time and a fake OpenAI endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms, at 1,170 requests per second. For two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures hold with around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, of July 2026, measures the Python proxy at 257.7 ms of added p99 with 329.5 MB of peak memory, against 0.7 ms for the Rust variant in beta. That measurement has no callbacks, no spend tracking and no persistence.

The operational reading is that the first figure is the best case and the second the order of magnitude to plan for agentic load, which is the one that arrives in a closed loop against fast responses. To watch it there are two always-on headers: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms, and the documentation sets the useful threshold of the second at 100 ms, above which the diagnosis is that the payloads are too large.

Three production settings that appear in LiteLLM’s documentation and that are wrong by default:

request_timeout is 6,000 seconds. A hundred minutes holding a connection against a hung upstream. The recommendation is to lower it to 600.

LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and the documentation puts the cost of that serialisation at 2 to 5 seconds with payloads over 2 MB. It is the first cause of latency their troubleshooting guide mentions.

Background jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every periodic job. The LITELLM_JOB_ROLE variable separates the pods that serve traffic from the replica that runs jobs.

And one failure mode with an open issue that deserves to be in the runbook: under sustained 429s from the upstream, pods have been reported unable to answer readiness probes during startup, with the ensuing restart loop. It fits with what was covered in incident response runbooks.

Start-up checklist

Ten checks before signing off the installation of the pair:

callbacks: ["langfuse_otel"] instead of success_callback: ["langfuse"], unless you depend on prompt management, which the OTLP preset does not support.
LANGFUSE_HOST defined explicitly. The preset’s default value points to the US region.
LiteLLM at 1.98.0 or higher, because of the team credential leak into the traces, and at 1.95.0 or higher because of the v4 ingestion header.
A price registered for every self-hosted model, per token and in USD, verified with /spend/calculate before opening traffic.
A test request with a known traceparent, checking that the same identifier appears in Langfuse.
An OpenTelemetry Collector between vLLM and Langfuse, with OTEL_SERVICE_NAME per engine and renaming of the old token attributes.
request_timeout: 600 and LITELLM_LOG out of DEBUG.
Tail sampling in the Collector, with a latency and status code policy, to keep 100 % of what is slow and what failed.
A written decision about prompt content, and blocking of the LiteLLM-Disable-Message-Redaction header at the ingress if the policy is not to log.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms and Langfuse’s ingestion queue depth (langfuse.queue.ingestion.depth), which is the autoscaling signal for its workers.

Traps and things that are not what they look like

The trace arrives, but ten minutes late. The x-langfuse-ingestion-version: 4 header is missing, which LiteLLM sends from 1.95.0. It is not a ClickHouse capacity problem.

The metadata works in development and disappears in production. Nginx drops headers with underscores by default, and all of this integration’s have one.

Session cost comes out as zero and generation cost does not. Langfuse adds up only the ingested cost in the session totals, not the one it computes itself.

Enabling OTel v2 breaks nothing visible and ungroups the conversations. The v2 mapper does not read session_id or trace_user_id from the request.

A team’s budget is never exhausted. Its spend is zero because the model has no price, and the calculation failure is only logged at debug level.

The alerts stop firing after the migration to v2. The success status becomes UNSET; no rule on OK is satisfied.

Forwarding the traceparent is enabled and it does not reach the engine. The generic otel callback is missing, without which the function that applies it returns before looking at the flag.

All the engines’ spans appear under unknown_service. vLLM emits no service name; OTEL_SERVICE_NAME has to be set per deployment.

Closing

The gateway and the tracing backend are the two pieces of the inference stack that take the least time to install and the longest to operate well. The installation fits in a ConfigMap. The operation consists of four decisions no getting-started guide raises: which integration route to take knowing that one expires in November, where the price per token of a model nobody invoices comes from, what to do with the traceparent at each hop, and what to tell an auditor about data the system drops by design when things get tight.

The answer to the fourth is the one that orders the other three. This pipeline is operational instrumentation, with statistical precision and best-effort guarantees, and separating it from day one from the regulatory audit record avoids the awkward conversation two years from now. With that separation made, the LiteLLM and Langfuse pair is the pair of pieces that turns a fleet of GPUs into a platform with an owner, a known cost and explainable latency.

Sources

LiteLLM, Langfuse integration (rutas de integración, metadata aceptada, cabeceras langfuse_*): https://docs.litellm.ai/docs/observability/langfuse_integration.
LiteLLM, OpenTelemetry v2 y Migración a OTel v2 (preset, captura de contenido, cambios de semántica): https://docs.litellm.ai/docs/observability/opentelemetry_v2 · https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.
LiteLLM, Logging y Config settings (redacción, turn_off_message_logging, variables de entorno): https://docs.litellm.ai/docs/proxy/logging · https://docs.litellm.ai/docs/proxy/config_settings.
LiteLLM, Custom pricing y Spend tracking (claves de precio, SpendLogs, endpoints de gasto): https://docs.litellm.ai/docs/proxy/custom_pricing · https://docs.litellm.ai/docs/proxy/cost_tracking.
LiteLLM, Production best practices (topología, request_timeout, buffer de transacciones en Redis, LITELLM_JOB_ROLE): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Benchmarks y Latency overhead (cifras de sobrecarga y cabeceras de diagnóstico): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, Rust AI Gateway benchmarks (la cifra de 257,7 ms de p99 del proxy Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, notas de versión v1.95.0, v1.98.0 y v1.100.0: https://docs.litellm.ai/release_notes/v1.95.0/v1-95-0 · https://docs.litellm.ai/release_notes/v1.98.0/v1-98-0 · https://docs.litellm.ai/release_notes/v1.100.0/v1-100-0.
Langfuse, OpenTelemetry (endpoint, autenticación, tabla de mapeo de atributos, ausencia de gRPC): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, Compatibility y Migrate v3 to v4 (la fecha del 16 de noviembre de 2026 y los modos de escritura): https://langfuse.com/docs/compatibility · https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4.
Langfuse, Token and cost tracking (prioridad del coste ingerido sobre el inferido): https://langfuse.com/docs/observability/features/token-and-cost-tracking.
Langfuse, Scaling y Sampling (dimensionado, señal de autoescalado, muestreo de cabecera): https://langfuse.com/self-hosting/configuration/scaling · https://langfuse.com/docs/observability/features/sampling.
Langfuse, issue #13468 — descarte de filas en ClickhouseWriter sin cola de mensajes muertos: https://github.com/langfuse/langfuse/issues/13468. Issue #15109 — coste de sesión a cero: https://github.com/langfuse/langfuse/issues/15109. Issue #5173 — Nginx y cabeceras con guión bajo: https://github.com/langfuse/langfuse/issues/5173.
vLLM, vllm/tracing/utils.py y vllm/tracing/otel.py en la etiqueta v0.28.0 (atributos emitidos, protocolo por defecto, insecure=True): https://github.com/vllm-project/vllm/blob/v0.28.0/vllm/tracing/utils.py.
Langfuse, vLLM integration (la advertencia de que vLLM solo exporta contadores y latencias): https://langfuse.com/integrations/model-providers/vllm.
LiteLLM, litellm/constants.py y litellm/litellm_core_utils/logging_worker.py (capacidades de las colas y política de descarte): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
OpenTelemetry, Semantic Conventions for Generative AI y el repositorio semantic-conventions-genai (estado de desarrollo, renombrados, captura de contenido): https://github.com/open-telemetry/semantic-conventions-genai.
OpenTelemetry Collector, tailsamplingprocessor (políticas de muestreo por cola y la restricción de instancia única por traza): https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md.
llm-d, End-to-end and fine-grained tracing in llm-d (OTEL_SERVICE_NAME y los spans huérfanos tras Envoy): https://llm-d.ai/blog/end-to-end-and-fine-grained-tracing-in-llm-d.

Langfuse from the inside: the sorting centre that must not become the bottleneck it came to observe

Sat, 06 Jun 2026 06:00:00 +0200

This post closes a trilogy of the Observe layer: in LLM tracing with OpenTelemetry GenAI the SDK → Collector → backend pipeline was set up and Langfuse was treated as a black box that receives spans; in Prompt versioning with Langfuse and MLflow its prompt management layer was used. Here we open the box: what is inside Langfuse, why v3 stopped being a monolith on Postgres, and how it is operated so it holds the traffic of an inference cluster without becoming the problem.

TL;DR

Langfuse v3 (stable since December 2024) is not an application, it is six services: two containers of its own (Web and Worker) and four stateful dependencies (Postgres, ClickHouse, Redis/Valkey and an S3-compatible blob store). The key architectural change compared with v2, which was a Next.js monolith on Postgres, is the asynchronous ingestion pipeline: traces are received in batches, written immediately to S3, only a reference is queued in Redis, and a Worker ingests them into ClickHouse in the background. This decouples reception speed (limited only by Redis write latency, ~1-5 ms) from the cost of persisting and merging into the analytical database. The result: the Web container sustains hundreds of events per second without a spike blocking the client that serves the inference. But that design only performs with the right settings. This post covers the architecture, its interaction with the rest of the on-premise stack, and ten backend knobs, from batching to ClickHouse to queue sharding, from the FINAL modifier to system log table hygiene, that decide the real throughput and the storage cost. And it marks where the async design hides data loss windows worth knowing about before promising “full traceability”.

You are here: OBSERVE (and the layer that holds up the rest)

The analogy: the postal sorting centre

Picture the central mail sorting office of a large city at rush hour. Lorries loaded with sacks (batches of letters) arrive at a rate that does not stop. If the counter clerk had to open every sack, read every letter, decide its destination and file it before accepting the next lorry, the queue of lorries would go round the block in ten minutes. No serious sorting centre works like that.

What they do is decouple reception from processing:

The reception counter accepts the sack, stamps a receipt on it, leaves it in a pigeonhole in the warehouse and drops a ticket on a conveyor belt. Time per sack: seconds. The counter never blocks.
Further back, in the sorting room, a team of operators picks tickets off the belt, retrieves the sack from its pigeonhole, opens it, sorts the letters and files them in the permanent archive, ordered, indexed, searchable.

Langfuse v3 is exactly this sorting centre:

Postal centre	Langfuse v3	Function
Reception counter	Web container (ingestion endpoint)	Accepts event batches, gives an immediate receipt (HTTP 207)
Pigeonhole warehouse	S3 / Blob store (MinIO on-prem)	Stores the raw sack (the full event)
Ticket on the belt	Redis / Valkey (BullMQ queue)	Only the reference to the object in S3, not the content
Sorting room	Worker container	Takes tickets, reads S3, transforms and files
Indexed permanent archive	ClickHouse (OLAP)	Traces, observations and scores, queryable by project+time
Administrative register	Postgres (OLTP)	Users, projects, API keys, prompts, datasets, config

The thesis of the whole post follows from this analogy: the value of Langfuse lies in the counter never blocking the client that serves the inference. An observability tool that adds latency or outages to the token-serving path is worse than having no observability, because it degrades precisely the system it set out to look after. The whole of the v3 design, and every knob in this post, exists to keep that promise under load.

The mechanism itself: six services, two planes

Langfuse v3 separates two planes that in v2 were fused together:

Ingestion and query plane (the two containers of its own, stateless, horizontally scalable): Web and Worker.
State plane (four dependencies, each with its own load profile): Postgres (transactional OLTP), ClickHouse (analytical OLAP), Redis/Valkey (queue + cache), Blob store (raw objects).

What to take away from this diagram:

Web and Worker are interchangeable and stateless. They store nothing locally. You can run 1 or 20 replicas of each; the state lives in the four dependencies. This is what allows scaling by load without choreography.
Redis never carries the event content, only the reference to the object in S3. That is why Redis holds the spike: a Redis write is ~1-5 ms and moves bytes, not kilobytes. The bottleneck of the Web container is, literally, Redis write speed.
Postgres and ClickHouse have opposite profiles. Postgres is OLTP: many small transactional reads and writes (is this API key valid? which version does the production label point to?). ClickHouse is OLAP: few enormous batch writes and analytical queries over billions of rows (give me the p95 TTFT of project X over the last 7 days). Putting traces in Postgres, which is what v2 did, works until it does not: at production volume, Postgres drowns in a workload it was not designed for. That was the reason for the redesign.

The ingestion flow step by step (and the maths of the decoupling)

The heart of the design is the ingestion route. Seen in detail, a POST /api/public/ingestion request with a batch of events does this:

The mathematical point is the early ACK. The latency the client perceives when sending traces is:

$$ t_{\text{client}} = t_{\text{S3 write}} + t_{\text{Redis enqueue}} \approx 10\text{–}40\,\text{ms} $$

whereas the real cost of persisting, reading S3, transforming, merging against the previous version, inserting into ClickHouse, letting the background merges compact, happens off that path, in the Worker, and can take hundreds of ms or seconds without the client caring. The decoupling turns a system whose throughput would be limited by ClickHouse speed into one limited by Redis speed. And Redis, on modest hardware, sustains on the order of 50,000 operations/second.

This has an important sizing consequence. If your inference load generates $E$ events/second (a chat with RAG + 2 tool calls easily produces 6-10 spans = events per request), the Web container absorbs them while $E \ll 50{,}000$. The Worker, by contrast, scales with the cost of processing: that is the component to watch and replicate, and the first knob of the post.

Honest scepticism. The early ACK has a flip side: between the HTTP 207 and persistence in ClickHouse there is a potential loss window. If the event is in S3 and the reference in Redis, and Redis goes down without persistence (AOF/RDB) before the Worker processes it, the reference is lost; the data is still in S3 but nobody claims it any more. More subtly: the Worker buffers writes to ClickHouse in memory and flushes them in batches; a Worker crash with a full buffer loses that batch. There is a reported bug where the ClickhouseWriter discards rows after exhausting flush retries with no dead-letter queue. For observability this is usually tolerable (losing 0.01 % of the traces breaks nothing). For regulatory auditing, where the trace is evidence, it is not, and Langfuse is better treated as “best-effort” and not as a ledger. We will come back to this at the close.

Interaction with the rest of the stack: Langfuse in the example 4×H100 cluster

Langfuse does not live in isolation. In the seven-layer stack it occupies the LLM-aware observability layer, and it relates to almost every other one. On the generic reference cluster we use throughout the blog, 4×H100 SXM 80 GB (320 GB aggregate VRAM), NVLink, 640 GB of system RAM, NVMe-oF, 25/100 GbE network, the telemetry flow looks like this:

Three ideas from this topology:

Langfuse receives from the OTel Collector, not from the application directly (in the recommended pattern). The app SDK or vLLM emit spans with the gen_ai.* semantic conventions; the Collector does batch, tail-sampling (preserving 100 % of errors and high latencies, sampling the rest) and enriches with its own attributes (tenant_id, priority_tier); and it distributes: LLM traces go to Langfuse, infrastructure spans to Tempo, metrics (GPU DCGM, vLLM metrics) to Prometheus. Langfuse is one more exporter, not the only destination. This is covered in detail in the OTel tracing post.
Langfuse runs off the GPUs. It is a consumer of CPU, RAM, disk and network (ClickHouse wants memory, MinIO wants disk, Redis wants CPU for networking) but it does not touch VRAM. On the 4×H100 cluster, Langfuse lives on a CPU node (or on the GPU nodes but with nodeSelector/taints keeping it away from the vLLM pods). Mixing ClickHouse with vLLM on the same node without resource limits is asking for an ingestion spike to steal memory bandwidth from inference. Isolation by design.
The telemetry route is “cold” and the data route is “hot”. The data plane (left) serves tokens with a millisecond latency budget; the telemetry plane (right) tolerates seconds. The early ingestion ACK is what keeps these two clocks apart: the app does not wait for Langfuse to file anything before returning the response to the user.

The 10 backend knobs that move the needle most

These are, in approximate order of impact/frequency, the settings that decide whether your self-hosted Langfuse ingests 50 events/s or 5,000, and whether your disk grows sustainably or explodes in three weeks. They are all environment variables or config injected into the Web and Worker containers (except the ClickHouse ones, which go in its server-side config). The canonical detail is in the Langfuse scaling doc.

Knob 1 — Scale the Worker by load (the first lever, always)

The Worker is the component that saturates first, because it does the expensive work: reading S3, transforming, merging, inserting into ClickHouse. The Langfuse operational rule is simple: a 2-CPU Worker container above 50 % CPU use is saturated; add replicas. Better than CPU, the Worker publishes via statsd the metric langfuse.queue.ingestion.length (ingestion queue length), which is the direct signal for autoscaling: if the queue grows without draining, there are not enough Workers.

# The ideal autoscaler watches queue depth, not just CPU.
# (KEDA ScaledObject over the statsd metric → Prometheus)
triggers:
 - type: prometheus
 metadata:
 query: langfuse_queue_ingestion_length
 threshold: "10000" # if the queue goes past 10k refs, scale

In AWS deployments there is ENABLE_AWS_CLOUDWATCH_METRIC_PUBLISHING=true to push these metrics to CloudWatch. On-premise, the path is statsd → Prometheus → KEDA, slotting into the autoscaling on Kubernetes with KEDA we already covered for vLLM. Always start here: most “Langfuse is slow” problems are simply insufficient Workers, not fine tuning.

Knob 2 — Separate the ingestion deployment from the UI one

When ingestion is heavily loaded, UI queries and the public API become slow because they share the same Web container. The solution is to split langfuse-web into two identical deployments and route by path: everything under /api/public/ingestion*, /api/public/media* and /api/public/otel* goes to the ingestion deployment; the rest (UI, read API) to the interface one.

# Ingress / gateway rule
location ~ ^/api/public/(ingestion|media|otel) {
proxy_pass http://langfuse-web-ingest; # replicas dedicated to writing
}
location / {
proxy_pass http://langfuse-web-ui; # replicas dedicated to reading
}

It is the same idea as the read/write separation of any system with mixed loads: a storm of writes should not starve whoever is trying to look at the dashboard right in the middle of the incident, which is precisely when they need it most.

Knob 3 — Batching writes to ClickHouse (interval + batch size)

ClickHouse hates small, frequent inserts: every INSERT creates a part on disk that then has to be merged, and thousands of tiny inserts generate thousands of parts and a storm of background merges that saturates the disk. The defence is to accumulate in an in-memory buffer in the Worker and flush in large batches:

# Worker: fewer flushes, larger batches → fewer parts, fewer merges
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=1000 # raise e.g. to 2000-5000
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_BATCH_SIZE=10000 # raise if there is throughput

Raising the interval and the batch size reduces the flush frequency and improves sustained throughput. The trade-off is direct and has to be understood: larger, less frequent batches mean more data in the Worker’s volatile buffer, that is, a bigger loss window if the Worker goes down (a knob coupled to the scepticism at the close). Langfuse also uses ClickHouse async_insert, which accumulates server-side before confirming; it adds another layer of buffering to keep in mind.

Knob 4 — Skip the ClickHouse read-before-write in ingestion

By default, when ingesting an event the Worker reads the existing event from ClickHouse and merges it with the incoming one (necessary when legacy SDKs send partial events: a start, then an end, then an update of the same observation). That per-event read loads ClickHouse on the write path and limits total throughput.

If your projects are not migrated from an old version, because the full history already lives in S3, you can disable that read:

# Date earlier than the creation of your first project
LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE=2025-01-01

With modern Langfuse SDKs or with ingestion via OpenTelemetry, this does not affect you negatively and removes one read per event. A warning from the doc itself: if you combine this with aggressive deletion (lifecycle) rules in S3 plus late event updates, you can generate duplicates in the history. Know that before enabling it.

Knob 5 — Write concurrency to S3/Blob storage

In high-throughput scenarios, the S3 client can exhaust its sockets and start queueing and throttling writes. The symptom is unmistakable in the logs of the Web container that handles ingestion:

@smithy/node-http-handler:WARN - socket usage at capacity=150
and 387 additional requests are enqueued.

…accompanied by a rise in memory in that container (queued requests pile up in RAM). The cure is to raise the concurrent write limit from its default of 50:

LANGFUSE_S3_CONCURRENT_WRITES=100 # raise gradually from 50

Each additional socket has a small memory cost, so the official advice is to raise it gradually while observing the behaviour, not to jump to 1000 in one go.

Knob 6 — Redis queue sharding + concurrency per shard

If Redis goes past 90 % CPU, first the obvious: an instance with at least 4 CPUs (so Redis spreads networking and background tasks across different cores) and Redis Cluster mode enabled. If CPU still does not come down, you can shard the queues Langfuse uses:

# Advanced: only if Redis is genuinely drowning and you already did the above
LANGFUSE_INGESTION_QUEUE_SHARD_COUNT=6 # ~2-3× the number of Redis cluster shards
LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT=6
# Concurrency counts PER SHARD; target ~20 per worker
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=3 # 6 shards × ~3 ≈ 18
LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY=3

Two traps the doc underlines and that are worth tattooing: once you shard, do not reduce the number of shards (it breaks the distribution); and concurrency counts per shard, not globally, so if you have 10 shards and want concurrency 20 per worker, set 2, not 20. It is an advanced knob: most on-premise deployments never need it.

Knob 7 — The `FINAL` modifier for OTel-only projects

Langfuse stores observations in a ClickHouse ReplacingMergeTree and, by default, adds the FINAL modifier to API queries so the last version of each row wins at read time. FINAL is necessary when ingestion produces several versions of the same observation (legacy SDKs with their start/end/update events), but it adds merge work to every read and slows it down.

Projects that ingest exclusively over OpenTelemetry write each observation as a single immutable row, so FINAL is redundant for them:

# Recommended in mixed deployments: per-project, flagged in Redis with a 24h TTL
LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true

# Only if ALL projects are OTel-only: global, no Redis lookup
LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL=true

Since on the example cluster the instrumentation is 100 % OTel (gen_ai.* via Collector), this knob is free money in dashboard read latency. Careful with the global version: do not enable it if some project is still using legacy ingestion, or reads may return duplicate or stale rows.

Knob 8 — Separate analytical reads from the write path (compute-compute)

Heavy dashboard queries (percentiles over millions of spans) compete with ingestion inserts and with background merges on the same ClickHouse. If your deployment supports compute-compute separation (ClickHouse Cloud or BYOC), you can route reads to a read-only compute group:

CLICKHOUSE_URL=http://clickhouse-primary:8123 # writes, migrations, ingestion
CLICKHOUSE_READ_ONLY_URL=http://clickhouse-reader:8123 # UI + public API reads

Critical nuance for on-premise, and here it is time to be sceptical about the usefulness of this knob in our context: on a single-node ClickHouse or on a self-managed cluster without compute separation, this variable adds nothing, because the read endpoint would be the same as the write one. It is a knob for cloud architectures with storage separated from compute. On an on-premise 4×H100 cluster with ClickHouse on one node, the real alternative is to scale ClickHouse vertically (the doc recommends ≥16 GiB of RAM for large deployments; ClickHouse scales vertically well) and to make sure every query filters by projectId and time, which is how the tables are indexed. Without a time filter, even the fattest ClickHouse suffers.

Knob 9 — Data retention: TTL in ClickHouse + lifecycle in S3

Disk is the cost that grows on its own. LLM traces carry whole inputs and outputs (sometimes prompts of tens of KB), and ClickHouse also accumulates its own system tables. The first-order lever is a retention policy that nightly deletes traces, observations, scores and media older than N days, coordinating ClickHouse and blob storage. Where the retention feature is not available, it is done by hand:

-- ClickHouse: TTL over the tracing tables
ALTER TABLE traces MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE observations MODIFY TTL toDateTime(start_time) + INTERVAL 90 DAY;
ALTER TABLE scores MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE event_log MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY;

# S3/MinIO: lifecycle rule, e.g. 30 days for the raw event bucket
# WATCH OUT! Do NOT apply retention to the MEDIA bucket:
# - it breaks the files referenced in traces
# - it breaks future uploads (state is tracked by hash in Postgres)

Two operational parameters that avoid nasty surprises on large deletions:

LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS=600000 # default 10 min; raise it if deletions time out
# ClickHouse 25.7+: less mutation pressure on bulk deletions
CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE=lightweight_update
CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE=true

The mental rule: short retention for raw events (S3, 30 days is usually enough, they are recoverable/recomputable), retention by business value for the ClickHouse tables (90 days, 180, whatever compliance asks for), and never touch the media bucket with blind lifecycle rules.

Knob 10 — Hygiene of the ClickHouse system log tables (the silent disk killer)

This is the knob nobody configures and that fills the disk without showing up in any Langfuse metric, because it is not Langfuse data: they are the system tables of ClickHouse itself (trace_log, text_log, opentelemetry_span_log, asynchronous_metric_log, metric_log, latency_log). By default they have no TTL, and the query profiler writes to system.trace_log continuously. On a ClickHouse with traffic, these tables can dominate disk usage while you look for the problem in your traces. Langfuse does not read from them, so they can be trimmed without fear. Two options:

<!-- Option A — disable the ones Langfuse never reads
 (file in /etc/clickhouse-server/config.d/) -->
<clickhouse>
 <trace_log remove="1"/>
 <text_log remove="1"/>
 <opentelemetry_span_log remove="1"/>
 <asynchronous_metric_log remove="1"/>
 <metric_log remove="1"/>
 <latency_log remove="1"/>
</clickhouse>
<!-- Keep query_log, part_log and error_log: useful for debugging and small -->

-- Option B — aggressive TTL + turn off the profiler, if you want to keep them for debugging
-- (in config: query_profiler_real_time_period_ns = 0)
SET max_table_size_to_drop = 0;
TRUNCATE TABLE system.trace_log;
ALTER TABLE system.trace_log MODIFY TTL event_date + INTERVAL 7 DAY;
-- repeat for each log table to cap

To identify which table is eating the disk, the golden query:

SELECT table, formatReadableSize(sum(bytes)) AS size, sum(rows) AS rows
FROM system.parts WHERE active GROUP BY table ORDER BY sum(bytes) DESC;

If you take only one knob from this post to your first real deployment, make it this one: the difference between a ClickHouse that grows 2 GB/day of useful data and one that grows 20 GB/day of system logs nobody looks at.

Summary table of the 10 knobs

#	Knob	Variable / action	When
1	Scale Worker	replicas by CPU>50 % / `langfuse.queue.ingestion.length`	always, first
2	Separate ingestion/UI	route `/ingestion`,`/media`,`/otel*` to a dedicated replica	slow UI under load
3	Batching to ClickHouse	`LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS` / `_BATCH_SIZE`	high throughput
4	Skip CH read-before-write	`LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE`	non-migrated projects
5	S3 concurrency	`LANGFUSE_S3_CONCURRENT_WRITES` (def. 50)	“socket usage at capacity”
6	Redis queue sharding	`LANGFUSE__QUEUE_SHARD_COUNT` + `_CONCURRENCY` (per shard)	Redis CPU >90 %
7	Remove `FINAL` (OTel)	`LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true`	100 % OTel instrumentation
8	CH read/write split	`CLICKHOUSE_READ_ONLY_URL` (cloud/BYOC only)	compute-compute available
9	Retention + TTL	TTL in CH + S3 lifecycle + `LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS`	always (disk cost)
10	CH system log hygiene	`<trace_log remove="1"/>` or aggressive TTL	always (hidden disk)

How to maximise Langfuse on the example 4×H100 cluster

With the architecture and the knobs clear, this is a concrete sizing to get the most out of Langfuse on the generic reference cluster (4×H100 SXM, 320 GB VRAM, 640 GB RAM, NVMe-oF, 25/100 GbE), without stealing a single GB of VRAM from inference.

Component distribution

Langfuse is 100 % CPU/RAM/disk/network load, so its natural place is off the GPU nodes or, if it cohabits, with taints/nodeSelector confining it away from the vLLM pods. Suggested distribution:

nodo-cpu-01 (control + observability, no GPU)
├── langfuse-web-ingest ×3 (2 CPU / 4 GiB each) ← ingestion, scales with load
├── langfuse-web-ui ×2 (2 CPU / 4 GiB each) ← dashboard/read API
├── langfuse-worker ×4 (2 CPU / 4 GiB each) ← the one that scales most
├── redis/valkey ×1 (4 CPU / 4 GiB, cluster mode)
└── postgres ×1 (2 CPU / 8 GiB, replica for HA)
nodo-storage-01 (heavy state, local NVMe)
├── clickhouse ×1 (8 CPU / 32 GiB / NVMe) ← ≥16 GiB is the minimum; 32 is comfortable
└── minio (S3) ×1 (4 CPU / 8 GiB / HDD+NVMe cache)
nodo-gpu-01..02 (4×H100 SXM each) → inference ONLY
└── vLLM, embeddings, reranker, guardrails (emit spans, do not host Langfuse)

Sizing by real load

Let us put numbers on an example load. Suppose the cluster serving 300 requests/second of chat-with-RAG, where each request generates on the order of 8 spans (request, retrieval, rerank, 2× tool, guardrail in, llm, guardrail out):

$$ E = 300\,\tfrac{\text{req}}{\text{s}} \times 8\,\tfrac{\text{spans}}{\text{req}} = 2,400\ \text{events/s} $$

Against the Redis ceiling (~50,000 ops/s), $E = 2{,}400$ leaves the reception counter at ~5 % of its capacity: enormous headroom. The component to watch is the Worker. With a target of ~20 concurrency per Worker and batches of 10,000 events every ~1-2 s, 4 Workers drain 2,400 ev/s with margin; the langfuse.queue.ingestion.length metric should stay flat near zero. If it grows, knob 1 (more Workers) is the answer before any fine tuning.

Tail-sampling is the multiplier that changes the economics. If the Collector preserves 100 % of errors/high latencies but samples normal traffic at, say, 10 %, the 2,400 ev/s you store in ClickHouse drop to ~240-300 effective ev/s without losing the signal that matters. The rule: sample in the Collector, not in Langfuse. Langfuse should receive, already filtered, what deserves to be persisted. This is developed in the OTel tracing post; here it is enough to note that upstream sampling is, in effect, the knob 0 that multiplies all the others.

Storage estimate

An LLM observation with full input+output weighs, compressed in ClickHouse, on the order of 1-3 KB (ClickHouse compresses text very well, 5-10×). With sampling at 10 % over 2,400 ev/s:

$$ 240\,\tfrac{\text{ev}}{\text{s}} \times 2\,\text{KB} \times 86,400\,\tfrac{\text{s}}{\text{day}} \approx 41\ \text{GB/day (raw)} \;\xrightarrow{\text{compression}}\; \sim 5\text{–}8\ \text{GB/day in CH} $$

At 90 days of retention (knob 9), the permanent archive settles at around 500-700 GB in ClickHouse, comfortable on the NVMe of the storage node, plus the raw events in MinIO with a 30-day lifecycle. Without the system log hygiene (knob 10), add easily as much again in rubbish nobody queries. The two disk knobs together are the difference between planning storage once a year and fighting a full disk every month.

“Maximum use” checklist

Sampling in the Collector (tail: 100 % errors + N % normal), before touching anything in Langfuse.
Workers scaled by queue length via KEDA (knob 1), not fixed.
Ingestion separated from UI (knob 2) so the dashboard responds during incidents.
SKIP_FINAL_FOR_OTEL_PROJECTS enabled (knob 7) because the instrumentation is 100 % OTel.
Generous CH batching (knob 3) tuned to the throughput, accepting the loss window.
Retention + TTL + system log hygiene (knobs 9 and 10) configured on day 1, not when the disk screams.
ClickHouse with ≥16 GiB and every query filtering by projectId+time (knob 8 in its on-premise version: vertical scaling).
Langfuse isolated from the GPUs by taints/nodeSelector: not a MB of VRAM, no memory bandwidth contention with vLLM.

Traps and things that are not what they look like

“Langfuse guarantees me full traceability.” No: the design is high-performance best-effort, not a ledger. Between the HTTP 207 and the row in ClickHouse there are volatile buffers (Redis without hard persistence, the Worker’s in-memory buffer, ClickHouse’s server-side async_insert). There is a known bug where the writer discards rows with no dead-letter queue after exhausting retries. For operational observability, losing 0.01 % of spans is irrelevant. For ENS/EU AI Act audit evidence, where the trace is the proof, Langfuse should not be the only record; the regulatory audit log needs durability guarantees this pipeline does not promise. A distinction covered in the technical controls for ENS/42001/EU AI Act.

Raising ClickHouse batching “to go faster” and nothing else. Knob 3 improves throughput at the cost of enlarging the loss window and the latency of data appearing in the dashboard. Batches of 50,000 every 10 s perform wonderfully… until the Worker restarts with 50,000 events in the buffer. Tune with awareness of the trade-off, not by blind maximisation.

Putting ClickHouse on the same node as vLLM with no limits. ClickHouse is voracious with memory bandwidth during merges. Sharing a node with vLLM without resources.limits or NUMA isolation means an ingestion spike can degrade inference TTFT, exactly the original sin this whole architecture wanted to avoid. Isolate.

Forgetting the time filter in your own queries. The ClickHouse tables are indexed by projectId and time. A custom dashboard or an API query without a time filter scans the whole history and takes performance down for everyone. It is not Langfuse “being slow”: it is a badly written query.

Applying lifecycle to the media bucket. It breaks the files referenced in traces and blocks future uploads (state is tracked by hash in Postgres). The media bucket is managed only with the Langfuse retention feature, never with blind S3 rules.

Treating queue sharding as a routine optimisation. It is an advanced knob for a genuinely drowning Redis, irreversible (do not reduce shards) and with per-shard concurrency semantics that are easy to misread. In the vast majority of on-premise deployments it is not needed; if you enable it “just in case”, you complicate your life without gaining anything.

Conclusion

Langfuse v3 solved the structural problem of LLM observability, that the observer should not suffocate the observed, by moving from a monolith on Postgres to a six-service sorting centre with asynchronous ingestion. That design is what allows a cluster serving thousands of tokens per second to be instrumented end to end without the app ever waiting for a trace to be filed. But the design is a necessary condition, not a sufficient one: it performs if the right levers are tuned. Of the ten knobs, three decide almost everything in a typical on-premise deployment, scaling Workers by queue length (1), retention + TTL (9), and system log hygiene (10); the rest are refinements that show up when the load bites. And above them all lives knob 0, which is not a Langfuse one: sampling in the Collector, which decides how much reaches the pipeline before any internal setting matters. Maximising Langfuse on the 4×H100 cluster is not about squeezing its peak throughput: it is about putting it off the GPUs, feeding it already-sampled traffic, sizing the Worker by the queue, and configuring retention on day one, so that the tool that came to tell the story does not end up being the chapter about the incident.

References

Langfuse, Scaling Langfuse Deployments (doc oficial de sizing y todos los env vars de este post): https://langfuse.com/self-hosting/configuration/scaling.
Langfuse, Self-host Langfuse y Configuration via Environment Variables: https://langfuse.com/self-hosting · https://langfuse.com/self-hosting/configuration.
Langfuse, ClickHouse (self-hosted): https://langfuse.com/self-hosting/deployment/infrastructure/clickhouse.
Langfuse, From Zero to Scale: Langfuse’s Infrastructure Evolution (el porqué del rediseño v2→v3): https://langfuse.com/blog/2024-12-langfuse-v3-infrastructure-evolution.
Langfuse, Migrate v2 to v3 (self-hosted): https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v2-to-v3.
ClickHouse, Langfuse and ClickHouse: A new data stack for modern LLM applications: https://clickhouse.com/blog/langfuse-and-clickhouse-a-new-data-stack-for-modern-llm-applications.
Langfuse, issue #13468 — ClickhouseWriter drops rows after max flush attempts with no DLQ (la ventana de pérdida documentada): https://github.com/langfuse/langfuse/issues/13468.
ClickHouse, TTL for tables and columns: https://clickhouse.com/docs/guides/developer/ttl.
OpenTelemetry, Semantic Conventions for Generative AI (gen_ai.*): https://opentelemetry.io/docs/specs/semconv/gen-ai/.

Batch sizing in vLLM: the two-hour grid search worth weeks of hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

max-num-seqs and max-num-batched-tokens are the two dials that control how much work vLLM processes on each scheduler iteration. Their default values are calibrated to be safe on any hardware, not to maximise throughput on yours. A systematic grid search over 25 configurations, runnable in two hours, identifies the combination that, for your specific workload and hardware, can double throughput without changing a single line of the model or adding a GPU. The OTel metrics that confirm you found the optimum are vllm:num_waiting_seqs, vllm:num_preemptions_total and vllm:time_per_output_token_seconds.

The analogy

An industrial kitchen with one chef and ten stoves. If the maître d’ only sends one order at a time, the chef works at 10% of capacity. If he sends a hundred simultaneous orders but there are ingredients for only twenty, the chef spends half his time waiting for restocking. The optimum is the point where every stove is lit and the resupply never runs dry.

max-num-seqs is how many orders the chef can have in preparation at once. max-num-batched-tokens is how many ingredients he can process in a single movement of the wok. Getting either of them wrong leaves stoves empty.

The problem: the defaults are not for your hardware

In vLLM V1 (≥ 0.6), the defaults are:

max-num-seqs = 1024 (V1) / 256 (V0)
max-num-batched-tokens = 8192

These values guarantee that vLLM starts on any GPU without OOM. They do not guarantee optimal throughput. The reason: the optimal point depends on three variables vLLM does not know at start-up:

The length distribution of your real workload — a RAG system with 2K-token prompts needs a different budget from a chat with 50-token messages.
VRAM available for KV cache — determined by the model, the quantisation and --gpu-memory-utilization.
Real expected concurrency — how many simultaneous users arrive at the 95th percentile.

The interaction between these three factors makes it impossible for a universal default to be optimal in specific cases.

The arithmetic that matters

On each iteration, vLLM’s scheduler decides which tokens to process. The total budget available per step is max-num-batched-tokens. That budget is split between:

Decode tokens: 1 for each active request in the generation phase. With 64 requests in decode, 64 tokens of budget are consumed.
Prefill tokens (in chunks): the rest of the budget goes to processing new prompts.

$$\text{prefill tokens per step} = \text{max num batched tokens} - \text{num requests decode}$$

If max-num-batched-tokens = 8192 and you have 512 requests in decode, each step can only process 8192 - 512 = 7680 prefill tokens. With 2000-token prompts, that is ~3.8 new prompts per iteration.

The problem appears when max-num-seqs is very high relative to the available KV cache. Each active request in decode occupies KV cache blocks. If the blocks run out, vLLM performs preemption: it pauses a request, frees its KV cache and re-queues it. Each preemption costs additional latency for the paused request and complexity for the scheduler.

$$\text{KV budget} = \frac{\text{free VRAM} \times \text{gpu memory utilization}}{\text{bytes per token} \times \text{max model len}}$$

For a Qwen2.5-14B on an RTX 4090 with Q4_K_M (9 GB of model, 15 GB free):

$$\text{KV budget} = \frac{15 \times 0.92 \times 10^9}{40\,000} \approx 345\,000 \text{ tokens}$$

With max-model-len = 8192, the maximum number of simultaneous requests with a full context is:

$$\text{real max seqs} = \frac{345\,000}{8192} \approx 42 \text{ requests}$$

Setting max-num-seqs = 1024 with those numbers guarantees constant preemptions. The optimum is at 40-50.

The grid search: methodology

Step 1: measure the real workload

Before hunting for the optimum, you need to know your traffic’s percentiles. From Langfuse or the vLLM logs:

# Extract the length distribution from Langfuse
import langfuse
client = langfuse.Langfuse()

traces = client.fetch_traces(limit=1000).data
prompt_lens = [t.input_tokens for t in traces if t.input_tokens]
output_lens = [t.output_tokens for t in traces if t.output_tokens]

import numpy as np
print(f"Prompt p50={np.percentile(prompt_lens,50):.0f} p95={np.percentile(prompt_lens,95):.0f} p99={np.percentile(prompt_lens,99):.0f}")
print(f"Output p50={np.percentile(output_lens,50):.0f} p95={np.percentile(output_lens,95):.0f} p99={np.percentile(output_lens,99):.0f}")

Step 2: work out the KV budget

Run once with --dry-run, or read vLLM’s start-up log:

INFO: # GPU blocks: 4521, # CPU blocks: 512

Each block is 16 tokens. 4521 × 16 = 72,336 tokens of total KV budget.

Step 3: the grid

With the KV budget known and the p95 of prompt/output length:

# grid_search_batch.py
import subprocess, json, time

MODEL = "Qwen/Qwen2.5-14B-Instruct-AWQ"
PROMPT_LEN = 512 # p50 of your workload
OUTPUT_LEN = 256
CONCURRENCY = 32 # simultaneous users expected at peak

seqs_values = [32, 64, 128, 256, 512]
tokens_values = [4096, 8192, 16384, 32768, 65536]

results = []
for seqs in seqs_values:
 for tokens in tokens_values:
 cmd = [
 "python", "-m", "vllm.entrypoints.benchmark_throughput",
 "--model", MODEL,
 "--max-num-seqs", str(seqs),
 "--max-num-batched-tokens", str(tokens),
 "--num-prompts", "200",
 "--input-len", str(PROMPT_LEN),
 "--output-len", str(OUTPUT_LEN),
 ]
 out = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
 # Parse throughput from the output
 for line in out.stdout.splitlines():
 if "Throughput" in line:
 tps = float(line.split(":")[1].strip().split()[0])
 results.append({"seqs": seqs, "tokens": tokens, "tps": tps})
 print(f"seqs={seqs} tokens={tokens} → {tps:.1f} tok/s")

# Save for analysis
with open("grid_results.json", "w") as f:
 json.dump(results, f, indent=2)

25 configurations × ~5 min = ~2 hours. Real execution time, not waiting time.

Step 4: read the surface

The result is a 5×5 throughput matrix. The typical shape:

max-num-batched-tokens → 4K 8K 16K 32K 64K
max-num-seqs ↓
32 180 310 380 390 385 ← max-num-seqs too low
64 185 350 480 510 508 ← optimal point for this workload
128 182 340 450 480 475
256 178 320 400 410 402 ← KV cache runs out, preemptions
512 170 290 360 370 368 ← high preemptions

The optimum in this example: max-num-seqs=64, max-num-batched-tokens=32768. Above that, preemptions cancel out the concurrency gain.

Confirmation with OTel

Once the optimal configuration is deployed, three Prometheus metrics confirm it is well calibrated:

# 1. Queued requests — should stay close to 0
# If it grows steadily: max-num-seqs too low or max-num-batched-tokens insufficient
vllm:num_waiting_seqs

# 2. Preemptions — should be 0 or very occasional (<1/min)
# If it grows: max-num-seqs too high for the available KV cache
rate(vllm:num_preemptions_total[5m]) * 60

# 3. ITL (inter-token latency) — should be stable, with no spikes
# Bimodality = badly calibrated batch size (some requests outside the CUDA graph bucket)
histogram_quantile(0.99, rate(vllm:time_per_output_token_seconds_bucket[5m]))

The optimal configuration produces:

num_waiting_seqs ≈ 0 under normal operation
num_preemptions_total stable (not growing)
time_per_output_token unimodal

If num_waiting_seqs is high with gpu_cache_usage_perc low: raise max-num-batched-tokens to process prefills faster. If num_preemptions_total grows: lower max-num-seqs or enable FP8 KV cache to free up blocks.

Reference configurations by profile

Based on the grid search for mid-range hardware (generic 4×H100, 14B-70B model):

Profile	Prompt p50	Output p50	max-num-seqs	max-num-batched-tokens
Conversational chat	150 tok	300 tok	256	16384
Enterprise RAG	1500 tok	200 tok	64	32768
Coding (completion)	800 tok	500 tok	128	32768
Summarisation	2500 tok	400 tok	32	65536
Batch processing	4000 tok	800 tok	16	65536

None of these is universal. They are starting points for the grid search on your real hardware and workload.

When not to touch the defaults

If your system sits below 50% KV cache utilisation (vllm:gpu_cache_usage_perc < 0.50) under real demand and with no num_waiting_seqs, the defaults are enough for your current load. The grid search contributes more when you are close to maximum capacity or when you want to extract the full performance of fixed hardware.

References

FP8 end to end: enable it, measure quality and decide with data

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

FP8 is the configuration change with the highest impact per unit of effort available on H100 and Ada Lovelace hardware. On an H100 it enables native FP8 tensor cores: +40-60% decode throughput and ×2 the VRAM available for the KV cache. On an RTX 4090 and an L40 the compute benefit is smaller, but the ×2 VRAM is real and translates directly into twice the concurrency. The risk is quality degradation, which on well-calibrated modern models is <0.5% on standard benchmarks but can be larger on formal reasoning. The correct workflow is not to switch it on and pray: it is to switch it on in staging, run the eval suite, correlate quality with throughput in OTel, and decide with data.

The analogy

A photographer who works with 35 mm negatives and moves to digital. Digital photos take up less space and are processed faster. But a low-resolution photo of a landscape can be indistinguishable from the high-resolution one to the human eye, while a low-resolution photo of text loses letters. Exactly the same trade-off applies to FP8: for tasks where numerical imprecision is averaged out over thousands of activations (conversation, summarisation, RAG), it is practically invisible. For tasks where a single wrong multiplication propagates into an incorrect answer (formal mathematics, critical code), it can be decisive.

The three layers of FP8 in vLLM

FP8 is not a single flag: it is three independent layers that are enabled separately and have different benefits.

Layer 1 — Model weights (--quantization fp8): The model weights are stored and computed in FP8 E4M3. The models have to be pre-quantised (available on HuggingFace with a -FP8 or -fp8 suffix) or quantised at load time with calibration. The benefit: the model takes up half the VRAM and the weight matmuls are 2× faster on an H100.

# Pre-quantised model (recommended for production)
vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8

# Or on-the-fly quantisation (no extra files, somewhat slower on the first tokens)
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --quantization fp8 \
 --kv-cache-dtype auto

Layer 2 — KV cache (--kv-cache-dtype fp8): The K and V tensors of the KV cache are stored in FP8 instead of BF16. It halves the size of the KV cache, doubling the number of tokens that fit in VRAM. It does not affect the model weights.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # dynamic calibration, mandatory to minimise degradation

Layer 3 — Activations (automatic on an H100): On Hopper GPUs, vLLM automatically enables FP8 for the intermediate activations when both previous layers are active. No extra flag is required.

Full production configuration:

vllm serve neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 \
 --quantization fp8 \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --gpu-memory-utilization 0.92 \
 --max-model-len 16384

The measurable impact by hardware

H100 SXM (Hopper, native FP8 tensor cores)

Metric	BF16 baseline	FP8 enabled	Delta
Decode throughput (tok/s, 70B, batch 32)	~1,800	~2,700	+50%
Model VRAM (70B)	140 GB	70 GB	−50%
KV cache VRAM available (on 4×H100)	180 GB	250 GB	+39%
Maximum concurrency (ctx 8K)	~22,500 tok	~31,250 tok	+39%

This is equivalent to an extra replica for free in terms of KV cache capacity.

RTX 4090 (Ada Lovelace, FP8 in CUDA but without dedicated tensor cores)

Metric	BF16/Q4 baseline	FP8 KV cache added	Delta
Decode throughput (tok/s, 14B Q4)	~45	~47	+4%
KV cache VRAM available	15 GB	15 GB (same model)	—
Total cache tokens (ctx 8K)	~46,000	~92,000	+100%
Maximum concurrency (ctx 8K)	~5 users	~11 users	+120%

On Ada, the compute benefit is smaller (the FP8 tensor cores do not have the same width as on Hopper), but the ×2 in KV cache capacity is entirely real and translates into twice the possible concurrent users.

The correct workflow: enable, measure, decide

Enabling FP8 directly in production without validating quality is inadequate. The correct workflow has four steps.

Step 1: baseline in staging

Before enabling FP8, record the quality metrics of the current BF16 model. The most reproducible way is to run an eval suite over a fixed dataset and save the results:

# Install lm-evaluation-harness
pip install lm-eval

# BF16 baseline
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Meta-Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/baseline_bf16.json

Step 2: enable FP8 and run the same eval suite

# FP8
lm_eval --model vllm \
 --model_args pretrained=neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8,quantization=fp8,kv_cache_dtype=fp8,calculate_kv_scales=true \
 --tasks mmlu,hellaswag,gsm8k \
 --num_fewshot 5 \
 --output_path ./results/fp8_full.json

Step 3: compute the degradation

# compare_eval.py
import json

with open("results/baseline_bf16.json") as f:
 baseline = json.load(f)
with open("results/fp8_full.json") as f:
 fp8 = json.load(f)

tasks = ["mmlu", "hellaswag", "gsm8k"]
print(f"{'Task':<15} {'BF16':>8} {'FP8':>8} {'Delta':>8} {'OK?':>6}")
print("-" * 50)
for task in tasks:
 b = baseline["results"][task]["acc,none"]
 f = fp8["results"][task]["acc,none"]
 delta = (f - b) / b * 100
 ok = "✓" if abs(delta) < 1.0 else "✗ REVIEW"
 print(f"{task:<15} {b:>8.3f} {f:>8.3f} {delta:>+7.1f}% {ok:>6}")

Decision thresholds documented in MLPerf Inference 2025:

< 0.5% degradation: enable in production without restrictions.
0.5% – 1.5%: enable with active quality monitoring via LLM-as-judge.
> 1.5%: investigate before enabling — a possible calibration problem or an incompatible model.

Step 4: domain eval with LLM-as-judge

Academic benchmarks measure what they measure. Your use case may be different. Adding 200 representative samples from your domain, evaluated by an LLM judge, closes the gap:

# domain_eval.py
from langfuse import Langfuse
from openai import OpenAI

client = Langfuse()
judge = OpenAI(base_url="http://judge-llm:8000/v1", api_key="token")

# Load the 200 curated production samples (prompt + expected answer)
samples = load_domain_samples("eval_dataset_200.json")

scores_bf16, scores_fp8 = [], []
for sample in samples:
 for model_type, endpoint in [("bf16", "http://staging-bf16:8000"), ("fp8", "http://staging-fp8:8000")]:
 response = call_model(endpoint, sample["prompt"])
 score = judge.chat.completions.create(
 model="Qwen/Qwen2.5-72B-Instruct",
 messages=[{
 "role": "user",
 "content": f"Rate this answer from 1 to 5 on accuracy and completeness.\nQuestion: {sample['prompt']}\nExpected answer: {sample['expected']}\nModel answer: {response}\n\nReply with a number from 1 to 5 only."
 }]
 ).choices[0].message.content.strip()

 if model_type == "bf16":
 scores_bf16.append(int(score))
 else:
 scores_fp8.append(int(score))

import numpy as np
print(f"Mean BF16 score: {np.mean(scores_bf16):.2f}")
print(f"Mean FP8 score: {np.mean(scores_fp8):.2f}")
print(f"Degradation: {(np.mean(scores_fp8)-np.mean(scores_bf16))/np.mean(scores_bf16)*100:.1f}%")

OTel + Langfuse correlation: the dashboard that decides

The moment of decision rests on a single dashboard with two signals on the same time axis:

Signal 1 — Throughput (Prometheus):

rate(vllm:generation_tokens_total[5m])

Signal 2 — Mean quality (Langfuse → Prometheus via exporter):

# If you have configured Langfuse with scores exported via OTel
langfuse_score_value{name="llm_judge_domain"}

The expected pattern after enabling FP8: throughput rises by 40-60% and quality stays within ±0.1 points. If quality drops by more than 0.3 points and stays low, there is a real problem.

# Alert: quality drops more than 0.2 points, sustained, after the change
ALERT FP8CalidadDegradada
 IF avg_over_time(langfuse_score_value{name="llm_judge_domain"}[30m])
 < (avg_over_time(langfuse_score_value{name="llm_judge_domain"}[1d] offset 2h) - 0.2)
 FOR 15m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Possible quality degradation after the FP8 configuration change" }

When NOT to enable FP8

FP8 is not always the right answer. The cases where degradation exceeds the acceptable threshold:

Formal mathematical reasoning: GSM8K and MATH are the benchmarks most sensitive to FP8. If your use case is solving mathematical problems or precise financial calculation, measure specifically on these benchmarks before enabling it.

Critical code with tests: numerical precision affects the probability of tokens at key positions in a function. The risk is not that the code “looks” bad, but that it passes superficial tests while having subtle bugs.

Very long contexts without --calculate-kv-scales: without dynamic scale calibration, the accumulated numerical error in the KV cache grows with the context. With --calculate-kv-scales active, the impact is minimal up to 32K tokens.

Small models (<7B): the FP8 conversion overhead can exceed the throughput benefit. The break-even point is around 7B parameters.

References

Instrumenting vLLM with OTel: measuring what the optimisations really do

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM exposes two independent observability signals: Prometheus metrics (pull, aggregated) and OTel traces (push, per request). To measure whether chunked prefill, prefix caching, speculative decoding, FP8 KV cache and concurrency are really working, you need both. The metrics tell you what is happening in the system; the traces tell you why one specific request was slow. This article configures the complete pipeline and maps each optimisation to its diagnostic metric.

The analogy

A Formula 1 driver and his telemetry engineers. The driver feels that the car “goes odd” in turn 3, but without the sensor data he does not know whether it is the rear tyre, the differential or the fuel. The engineers see exactly what happened in that turn, temperature per sensor, lateral load per millisecond.

vLLM without OTel is the driver on his own: you notice that the TTFT “looks high” but you do not know whether it is a long prefill, a prefix cache miss, or a KV cache preemption. With OTel you have the full picture: the metrics are the race summary (aggregated), the traces are the lap-by-lap telemetry (per-request).

Architecture of the two signals

vLLM deliberately separates its two observability channels:

 ┌─────────────────────────────┐
│ vLLM │
│ │
requests ────────────►│ SchedulerStats │
│ │ │
│ ├─► Prometheus /metrics │◄── scrape (pull)
│ │ (aggregated, ~15s) │
│ │ │
│ └─► OTLP exporter │──► push (per request)
│ (spans, immediate) │
└─────────────────────────────┘
│ OTLP gRPC/HTTP
▼
┌─────────────────────┐
│ OTel Collector │
│ │
│ receivers: │
│ otlp (traces) │
│ prometheus │
│ exporters: │
│ langfuse │
│ prometheus remote│
│ loki (logs) │
└─────────────────────┘

Prometheus pull exposes metrics with the vllm: prefix at :8000/metrics. They are histograms, gauges and counters updated on every scheduler iteration. Good for dashboards and alerts about the system as a whole.

OTLP push sends one span per request, immediately on completion. It contains attributes of the specific request: prompt tokens, generated tokens, TTFT, model. Good for debugging anomalous requests and for Langfuse.

Installation and basic configuration

# vLLM with OTel support
pip install "vllm[otel]"
# Installs: opentelemetry-sdk, opentelemetry-api,
# opentelemetry-exporter-otlp, opentelemetry-semantic-conventions-ai

# Start vLLM with OTel enabled
export OTEL_SERVICE_NAME="vllm-production"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://otel-collector:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_TRACES_INSECURE="true" # on an internal network without TLS

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --otlp-traces-endpoint http://otel-collector:4317 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5

The Prometheus metrics need no extra configuration: they are always at :8000/metrics.

OTel Collector: minimum configuration

# otel-collector-config.yaml
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

 prometheus:
 config:
 scrape_configs:
 - job_name: vllm
 scrape_interval: 15s
 static_configs:
 - targets: ["vllm:8000"]

processors:
 batch:
 timeout: 5s
 resource:
 attributes:
 - key: deployment.environment
 value: "production"
 action: upsert

exporters:
 otlphttp/langfuse:
 endpoint: "https://cloud.langfuse.com/api/public/otel"
 headers:
 Authorization: "Basic <base64(pk:sk)>"

 prometheusremotewrite:
 endpoint: "http://prometheus:9090/api/v1/write"

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch, resource]
 exporters: [otlphttp/langfuse]
 metrics:
 receivers: [prometheus]
 processors: [batch]
 exporters: [prometheusremotewrite]

The five metrics that matter

Every optimisation has a primary diagnostic signal. If the metric does not move as expected after enabling the flag, there is a configuration or load problem.

1. Chunked prefill → `vllm:time_to_first_token_seconds`

Chunked prefill should reduce the variance of the TTFT, not necessarily the median. Its main goal is for the high percentiles (p99) to come down even if the p50 rises slightly.

# TTFT p50 and p99 — expect p99 to drop with chunked prefill enabled
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

Sign that it works: the p99/p50 ratio approaches 1. Without chunked prefill, one request’s long prefill blocks all the others and the p99 rises disproportionately.

Sign of a problem: p50 and p99 both rise. A --max-num-batched-tokens that is too low makes the chunks so small that the prefill takes many steps to complete even though the other requests are not blocked. Raise the budget.

It is also useful to look at the llm.usage.prompt_tokens attribute per span in the OTel traces: requests with many prompt tokens should have a proportional TTFT, not a blocking one.

2. Prefix caching → `vllm:gpu_prefix_cache_hit_rate`

# Prefix cache hit rate on the GPU (0.0–1.0)
vllm:gpu_prefix_cache_hit_rate

# Evolution over a 5-minute window
rate(vllm:gpu_prefix_cache_hit_rate[5m])

Sign that it works: a sustained hit rate > 0.5 in workloads with a shared system prompt. With a hit rate of 0.8, 80% of requests skip the prefix prefill; the TTFT of those requests falls to the cost of the variable suffix alone.

Sign of a problem: a hit rate close to zero despite system prompts that “look” identical. The usual causes:

# ❌ This breaks the prefix caching hash:
system_prompt = f"Today is {datetime.now()}. You are an assistant..."
# ^^^ a different timestamp on every request

# ✅ The system prompt has to be identical byte for byte:
system_prompt = "You are an assistant specialising in infrastructure..."

Any variation in the system prompt, timestamps, session IDs, interpolated prompt versions, produces a different hash and a cache miss. The OTel traces do not expose the hit/miss per request directly in the current implementation; use them to correlate a high llm.usage.prompt_tokens with a high TTFT in the same request.

3. Speculative decoding → `vllm:spec_decode_draft_acceptance_rate`

# Acceptance rate of the draft model (0.0–1.0)
vllm:spec_decode_draft_acceptance_rate

# Estimated effective speedup (with k=5 proposed tokens)
# speedup ≈ (1 + α·k) / (1 + overhead_draft)
# Simplified: if α=0.75 and k=5 → speedup ≈ 1 + 0.75×5×(1 - cost_ratio)

Sign that it works: a sustained acceptance rate > 0.70. Below 0.60, the draft model’s overhead outweighs the gain from the accepted tokens and speculative decoding is counterproductive.

Sign of a problem: an acceptance rate < 0.50. The usual causes:

A drafter from a different family than the verifier (for instance, Mistral 0.5B as the draft for Qwen 7B).
A high generation temperature (>0.9): the higher the temperature, the more the draft’s distribution diverges from the verifier’s.
A very large batch: at high concurrency, the draft can fall outside the domain of the current requests.

# Alert: inefficient speculative decoding
ALERT SpecDecodeInefficient
 IF vllm:spec_decode_draft_acceptance_rate < 0.60
 FOR 5m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Draft acceptance rate low: disable spec decode or change the drafter" }

In the OTel traces, the complete span of the request includes the total decode time. Without an acceptance rate per span, the way to detect that spec decode is working is to compare the total decode time divided by the tokens generated: if it is significantly lower than the baseline without spec decode, it is helping.

4. FP8 KV cache and concurrency → `vllm:gpu_cache_usage_perc` + `vllm:num_preemptions_total`

These two metrics are the two sides of KV cache management:

# KV cache utilisation (0.0–1.0)
# With FP8 enabled, the same hardware supports more requests before saturating
vllm:gpu_cache_usage_perc

# Cumulative preemptions (counter)
# Rises when vLLM cannot fit more requests and pauses one
rate(vllm:num_preemptions_total[5m])

Sign that FP8 works: with --kv-cache-dtype fp8 enabled, gpu_cache_usage_perc should saturate at concurrency levels around 2× higher than the BF16 baseline before num_preemptions_total starts to grow.

Sign of a problem: num_preemptions_total growing at rates > 1/minute with gpu_cache_usage_perc below 0.90. It indicates that max-num-seqs is too high for the available KV cache: requests enter the system but there are no free blocks to assign to them. Lower max-num-seqs or reduce max-model-len.

# Alert: KV cache saturated with preemptions
ALERT KVCacheSaturated
 IF rate(vllm:num_preemptions_total[2m]) > 0.5
 AND vllm:gpu_cache_usage_perc > 0.85
 FOR 3m
 LABELS { severity = "critical" }
 ANNOTATIONS { summary = "KV cache saturated: lower max-num-seqs or max-model-len" }

The impact of FP8 on capacity can be quantified:

$$\Delta\text{capacity} = \frac{\text{max tokens FP8}}{\text{max tokens BF16}} \approx 2\times$$

Measure before and after enabling --kv-cache-dtype fp8: the level of gpu_cache_usage_perc for a given concurrency should fall by half.

5. Effective concurrency → `vllm:num_running_seqs` + `vllm:num_waiting_seqs`

# Requests active in the engine (decode + prefill in progress)
vllm:num_running_seqs

# Requests queued waiting for a slot
vllm:num_waiting_seqs

# Waiting ratio: if > 0.2 sustained, there is a concurrency bottleneck
vllm:num_waiting_seqs / (vllm:num_running_seqs + vllm:num_waiting_seqs)

Healthy sign: num_running_seqs stable close to the configured --max-num-seqs value, num_waiting_seqs low (< 10% of running).

Sign of a problem: a high num_waiting_seqs with a low gpu_cache_usage_perc. It indicates that the scheduler is not filling the available slots because max-num-batched-tokens is too low: the token budget per step does not allow the pending prefills to be processed fast enough. Raise max-num-batched-tokens.

Reference dashboard: the 5 metrics in Grafana

{
 "panels": [
 {
 "title": "TTFT p50 / p99 (chunked prefill)",
 "targets": [
 {"expr": "histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p50"},
 {"expr": "histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p99"}
 ]
 },
 {
 "title": "Prefix cache hit rate",
 "targets": [{"expr": "vllm:gpu_prefix_cache_hit_rate", "legendFormat": "GPU hit rate"}]
 },
 {
 "title": "Spec decode acceptance rate",
 "targets": [{"expr": "vllm:spec_decode_draft_acceptance_rate", "legendFormat": "acceptance rate"}]
 },
 {
 "title": "KV cache usage + preemptions",
 "targets": [
 {"expr": "vllm:gpu_cache_usage_perc", "legendFormat": "cache usage"},
 {"expr": "rate(vllm:num_preemptions_total[2m]) * 60", "legendFormat": "preemptions/min"}
 ]
 },
 {
 "title": "Effective concurrency",
 "targets": [
 {"expr": "vllm:num_running_seqs", "legendFormat": "running"},
 {"expr": "vllm:num_waiting_seqs", "legendFormat": "waiting"}
 ]
 }
 ]
}

Connecting traces to Langfuse

vLLM’s OTel traces are GenAI semconv compatible spans. Langfuse accepts them directly via OTLP:

# In the OTel Collector (already configured above)
# The otlphttp/langfuse exporter sends traces to Langfuse Cloud or self-hosted

# For self-hosted Langfuse (ENS/sovereign):
exporters:
 otlphttp/langfuse:
 endpoint: "http://langfuse-internal:3000/api/public/otel"
 headers:
 Authorization: "Basic <base64(pk_xxx:sk_xxx)>"

In Langfuse, each vLLM request appears as a trace with:

gen_ai.system: the model served
gen_ai.usage.input_tokens: prompt tokens
gen_ai.usage.output_tokens: generated tokens
Span duration: end-to-end latency

What does not appear directly in the span: the speculative decoding acceptance rate, the prefix cache hit/miss, or the number of preemptions. That data is only in Prometheus. The correct workflow is:

Langfuse identifies an anomalous request by latency.
Prometheus/Grafana shows whether in that interval there were high preemptions, low spec decode, or a prefix cache miss.
The two are correlated by timestamp.

Quick diagnostic matrix

Observable symptom	Prometheus metric	Probable cause	Action
TTFT p99 very high	`ttft p99/p50 >> 2`	Long blocking prefills	Raise `--max-num-batched-tokens`
TTFT p50 high, p99 the same	`ttft p50 > 500ms`	Prefix cache not working	Check the system prompt hash
Slow decode with no improvement	`spec_decode_acceptance < 0.60`	Incompatible drafter	Change the drafter or disable it
Sporadic OOM / crash	`gpu_cache_usage_perc = 1.0` + preemptions	KV cache full	Lower `max-num-seqs` or enable FP8
High queue with free cache	`waiting >> 0` + `cache < 0.70`	Low token budget	Raise `--max-num-batched-tokens`

Implications for sovereign on-premise inference

In an ENS deployment where you cannot use Langfuse Cloud or DataDog, the complete self-hosted stack is:

# docker-compose.yml (or equivalent K8s manifests)
services:
 otel-collector:
 image: otel/opentelemetry-collector-contrib:latest
 volumes: [./otel-config.yaml:/etc/otel/config.yaml]

 langfuse:
 image: langfuse/langfuse:latest
 environment:
 DATABASE_URL: postgres://...

 prometheus:
 image: prom/prometheus:latest

 grafana:
 image: grafana/grafana:latest

The whole pipeline runs on-premise. The traces never leave the perimeter. ENS compliance does not depend on which observability you choose: it depends on the inference data not going out to third parties. With a local stack, both conditions are met.

References

One big replica or many small ones: the decision that defines your platform

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

With 4 GPUs available you have two basic options: one vLLM instance using all 4 (TP=4) or two independent instances using 2 each (TP=2 × 2 replicas). The first gives lower latency per individual request. The second gives higher aggregate throughput at high concurrency, better fault tolerance and finer-grained scaling. The crossover point, where the second overtakes the first, is typically between 16 and 64 concurrent requests for 70B models, much earlier than most people assume. The metric that decides it: goodput, the tokens generated within the latency SLO divided by the total.

The analogy

Two ways to organise a translation service: one senior translator with access to four specialised dictionaries at once (able to resolve any complex query in 30 seconds), or two junior translators with two dictionaries each (they take 45 seconds per complex query, but they can handle two at the same time).

For a client who arrives alone and expects a fast answer, the senior wins. For a queue of twenty clients arriving at once, the two juniors process twice as many queries per hour even though each one takes longer. The question is not who is better, but what kind of traffic you have.

The two architectures in vLLM

Architecture A: TP=4, one replica

# A single instance uses all 4 GPUs via tensor parallelism
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 4 \
 --gpu-memory-utilization 0.92 \
 --port 8000

GPU-0 ─┐
GPU-1 ─┤─ vLLM instance 0 ──► port 8000
GPU-2 ─┤ (TP=4, the model is
GPU-3 ─┘ split across 4 GPUs)

Every attention and FFN operation is split across 4 GPUs. They require an all-reduce after each layer (on NVLink: ~50-200 µs; on PCIe: ~2-8 ms). The complete model is available in the aggregate VRAM.

Architecture B: TP=2 × 2 replicas

# Two independent instances, each with 2 GPUs
# Instance 0 on GPU 0-1
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 2 --port 8000

# Instance 1 on GPU 2-3
CUDA_VISIBLE_DEVICES=2,3 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
 --tensor-parallel-size 2 --port 8001

GPU-0 ─┐ ┌─► port 8000
GPU-1 ─┘─ vLLM instance 0 ─┘
← load balancer
GPU-2 ─┐ ┌─► port 8001
GPU-3 ─┘─ vLLM instance 1 ─┘

Each instance holds half the model. Requests are distributed across instances. No communication between instances (they are completely independent).

Why TP=4 has higher individual latency than TP=2

Tensor parallelism splits every transformer layer. After computing its fraction, each GPU needs to synchronise with the others via all-reduce before moving on to the next layer. The cost of that synchronisation:

$$\text{overhead TP} = n\_layers \times 2 \times \text{allreduce latency}$$

For Llama 3 70B (80 layers) on 4×H100 NVLink:

$$\text{overhead TP4} = 80 \times 2 \times 100\,\mu s = 16\,ms$$

On PCIe (without direct NVLink between GPUs):

$$\text{overhead TP4 PCIe} = 80 \times 2 \times 3\,ms = 480\,ms$$

That overhead is added to every decode step. With TP=2:

$$\text{overhead TP2} = 80 \times 2 \times 60\,\mu s = 9.6\,ms \text{ (NVLink)}$$

The difference between TP=2 and TP=4 on NVLink is ~6 ms per decode step, which matters for TPOT (inter-token latency) in streaming applications.

On PCIe without direct NVLink, TP=4 can be 400 ms slower per step than TP=2. For an output of 200 tokens, that is 80 extra seconds. In that scenario TP=4 over PCIe should never be used unless the model does not fit in 2 GPUs.

The crossover point: when TP=2×2 beats TP=4×1

For a 70B model on 4×H100 SXM (NVLink), aggregate throughput in tokens per second:

Concurrency | TP=4 × 1 instance | TP=2 × 2 instances | Winner
──────────────┼──────────────────────┼───────────────────────┼────────
1 | 200 tok/s | 170 tok/s | TP=4 (latency)
4 | 650 tok/s | 620 tok/s | TP=4 (slight)
16 | 1,800 tok/s | 2,100 tok/s | TP=2×2
32 | 2,400 tok/s | 3,600 tok/s | TP=2×2 (+50%)
64 | 2,800 tok/s | 5,200 tok/s | TP=2×2 (+86%)
128 | 2,900 tok/s | 5,800 tok/s | TP=2×2 (+100%)

Why they diverge at high concurrency: with TP=4, the scheduler of a single instance manages every request but the KV cache is shared. With TP=2×2, each instance has its own scheduler and KV cache: less contention, more real parallelism.

The crossover point on NVLink sits around 16-32 simultaneous requests for 70B. For smaller models (14B, 7B) the crossover happens earlier, because the TP communication overhead weighs more in relative terms.

The three implications nobody mentions

1. Fault tolerance

With TP=4 × 1 replica: if one GPU fails, the whole instance goes down. The service drops to 0% until the GPU recovers or the pod restarts on another node.

With TP=2 × 2 replicas: if one GPU fails, one instance goes down. The service keeps running at 50% capacity. For ENS/NIS2, where availability is a contractual requirement, this difference is decisive.

2. Autoscaling granularity

With KEDA or HPA based on vllm:num_waiting_seqs, autoscaling has to provision in multiples of the deploy unit:

TP=4 × 1: each new node requires 4 GPUs. The minimum scaling granularity is 4 GPUs.
TP=2 × 2: each new pod requires 2 GPUs. The minimum granularity is 2 GPUs, which is finer and more cost-efficient.

3. Quality degradation under load

TP=4 with many concurrent requests starts to suffer preemptions when the KV cache fills up. TP=2×2 spreads that pressure across two independent KV cache pools, so the probability of preemption is lower under the same total load.

Measuring the crossover point with OTel

Goodput is the right metric for comparing the two architectures. Not raw throughput (which ignores the SLO), but the tokens generated within the agreed TPOT SLO:

# Goodput: tokens generated with TPOT inside the SLO (e.g. <50ms/token)
# For TP=4:
rate(vllm:generation_tokens_total{instance="tp4"}[5m])
 * (1 - histogram_quantile(0.95, rate(vllm:time_per_output_token_seconds_bucket{instance="tp4"}[5m])) > 0.050)

# For TP=2×2 (sum of the two instances):
sum(rate(vllm:generation_tokens_total{instance=~"tp2-.*"}[5m]))
 * (1 - histogram_quantile(0.95, sum(rate(vllm:time_per_output_token_seconds_bucket{instance=~"tp2-.*"}[5m]))) > 0.050)

A direct comparison on the same dashboard, with synthetic traffic at different concurrency levels, determines the exact crossover point for your hardware and model.

The decision by workload profile

Profile	Recommended architecture	Reason
Single-user chatbot / low concurrency (<10 simultaneous)	TP=4 × 1	Lower p50 latency, better streaming experience
Enterprise API (20-100 concurrent)	TP=2 × 2	Higher goodput, fault tolerance, finer autoscaling
Batch processing (throughput > latency)	TP=2 × 2 (or more replicas)	Maximum throughput always lies in replicas
Very large model (>80B, does not fit in 2 GPUs)	TP=4 × 1	No structural alternative
ENS / contractual availability	TP=2 × 2 minimum	Losing one GPU is not catastrophic

Kubernetes configuration for both architectures

# Parallel deployments for an A/B test or different topologies

# TP=2 instances (2 replicas per deployment)
apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-tp2
spec:
 replicas: 2
 template:
 spec:
 containers:
 - name: vllm
 args: ["serve", "meta-llama/Meta-Llama-3.1-70B-Instruct",
 "--tensor-parallel-size", "2",
 "--gpu-memory-utilization", "0.92"]
 resources:
 limits:
 nvidia.com/gpu: "2" # 2 GPUs per pod
---
# Service with load balancing across the 2 replicas
apiVersion: v1
kind: Service
metadata:
 name: vllm-tp2
spec:
 selector:
 app: vllm-tp2
 ports:
 - port: 8000
 sessionAffinity: ClientIP  # for prefix cache awareness
 sessionAffinityConfig:
 clientIP:
 timeoutSeconds: 10800 # 3 hours of affinity per session

The sessionAffinity: ClientIP setting in the Kubernetes Service is the simplest way to implement session-affinity routing: requests from the same client always go to the same replica, maximising the prefix cache hit rate on the conversation history.

References

Prefix cache: hit rate engineering to go from 15% to 75%

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM’s prefix cache stores the KV cache blocks of shared prefixes and reuses them in later requests. A hit avoids recomputing that prefix: TTFT drops to the cost of the variable suffix only. In enterprise workloads with fixed system prompts (RAG, domain chatbots, assistants with long instructions) the hit rate should be >70%. In practice it is 10-20% for entirely avoidable reasons. This article identifies them, fixes them, and gives the OTel queries to confirm the result.

The analogy

A simultaneous conference interpreter who has to translate the speeches of twenty speakers. All of them open with the same two-page protocol preamble: the conference declaration, the rules of conduct, the day’s programme. An interpreter with no memory rereads the two pages for each speaker before starting on their specific speech. An interpreter with good notes reads them once, files them, and when the second speaker starts goes straight to the speech.

The prefix cache is that file. The prefix hash is the reference that lets you skip to the new part. But if the preamble changes by even one word, because someone drops in the day’s date, the interpreter has to reread everything from the beginning.

How the prefix cache hash works

vLLM splits the KV cache into blocks of 16 tokens. Each block has a hash computed over its exact content. When a new request arrives, vLLM checks whether any initial block of the prompt is already in cache by comparing hashes.

The hash is computed over the byte-by-byte content of the tokens. Any difference, a space, a different character, one extra token, produces a completely different hash. There is no partial matching inside a block.

Direct consequence: if your system prompt has 512 tokens and token number 3 changes between requests (because you interpolate a date, an ID, a version number), no block hits even though 99% of the text is identical.

Block 0 (tokens 0-15): hash = a3f7... ← in cache?
Block 1 (tokens 16-31): hash = 9d2c... ← in cache?
...
Block 31 (tokens 496-511): hash = 7e1a... ← in cache?

If block 0 does not hit (because its content changed), blocks 1-31 are not even checked although they are identical. The prefix cache is sequential.

Audit: why your real hit rate is low

Before changing anything, you need to know what is breaking the hash. The most direct method: pull the last 1000 production prompts and work out what fraction of the prefix varies.

# audit_prefix_cache.py
import langfuse, hashlib, collections
from transformers import AutoTokenizer

client = langfuse.Langfuse()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")

traces = client.fetch_traces(limit=1000).data
prompts = [t.input for t in traces if t.input]

# Tokenise and extract the first 512 tokens (the typical system prompt)
prefixes = []
for prompt in prompts:
 tokens = tokenizer.encode(prompt, add_special_tokens=False)
 prefix_tokens = tuple(tokens[:512])
 prefixes.append(prefix_tokens)

# How many unique prefixes are there?
unique = len(set(prefixes))
total = len(prefixes)
print(f"Unique prefixes: {unique}/{total} ({unique/total*100:.1f}%)")
print(f"Theoretical hit rate if they were all equal: {(1 - unique/total)*100:.1f}%")

# Find which token differs between the most common prefix and the rest
from collections import Counter
most_common_prefix = Counter(prefixes).most_common(1)[0][0]

divergence_positions = []
for prefix in prefixes:
 if prefix == most_common_prefix:
 continue
 for i, (a, b) in enumerate(zip(most_common_prefix, prefix)):
 if a != b:
 divergence_positions.append(i)
 break

if divergence_positions:
 pos = Counter(divergence_positions).most_common(1)[0][0]
 token_text = tokenizer.decode([most_common_prefix[pos]])
 print(f"\nMost frequent divergence at position {pos}: '{token_text}'")
 print("→ The token at that position varies between requests")

The most common culprits, in order of frequency:

1. Timestamps and dates:

# ❌ Breaks the hash on every request
system = f"Current date: {datetime.now().strftime('%Y-%m-%d %H:%M')}. You are an assistant..."

# ✅ Take the date out of the system prompt
system = "You are an assistant specialising in cloud infrastructure."
# Pass the date as part of the user message if it is needed

2. Session and user IDs:

# ❌
system = f"User ID: {user_id}. Preferences: {user_prefs}. You are an assistant..."

# ✅ Separate the static from the contextual
system = "You are a specialised assistant." # always the same
# Add the user context as the first message of the history

3. Interpolated prompt versions:

# ❌
system = f"[v{PROMPT_VERSION}] You are an assistant..." # changes with every deploy

# ✅ Do not version in the text, version in the prompt name in Langfuse
system = "You are an assistant..."

4. Dynamic few-shots:

# ❌ Examples retrieved at random from a pool
examples = random.sample(example_pool, k=3)
system = f"Examples:\n{format_examples(examples)}\n\nYou are an assistant..."

# ✅ Fixed few-shots always in the same order
FIXED_EXAMPLES = [example_pool[0], example_pool[1], example_pool[2]]
system = f"Examples:\n{format_examples(FIXED_EXAMPLES)}\n\nYou are an assistant..."

Template engineering: the structure that maximises hits

The principle is simple: everything static goes first, everything dynamic goes after. The prefix cache is sequential. Once a block fails to hit, the rest are not looked up either.

OPTIMAL STRUCTURE for maximising prefix cache:
┌──────────────────────────────────────────────┐
│ STATIC BLOCK (tokens 0-511) │ ← hit rate ~100%
│ Invariant system prompt │
│ Fixed instructions │
│ Few-shots always in the same order │
├──────────────────────────────────────────────┤
│ SEMI-STATIC BLOCK (tokens 512-1023) │ ← hit rate ~60-80%
│ RAG documents for this session │
│ Conversation history so far │
├──────────────────────────────────────────────┤
│ DYNAMIC BLOCK (tokens 1024+) │ ← hit rate ~0% (expected)
│ Current user message │
│ Context specific to this request │
└──────────────────────────────────────────────┘

For RAG specifically: if the retrieved documents are the same across a set of similar queries (very frequent in RAG over fixed corporate documents), ordering them always in the same order (by ID, by a fixed score, not by a variable score) multiplies the hit rate of the semi-static block.

Prefix-aware routing: the next level

With a single vLLM instance, the prefix cache works automatically. The problem shows up with multiple replicas: the load balancer distributes requests round-robin, and the prefix cached on replica A is no use at all when the request lands on replica B.

The solution is prefix-aware routing: send requests with the same prefix to the same node.

With Ray Serve (native integration):

# ray_serve_prefix_router.py
from ray import serve
from ray.serve.llm import LLMConfig, build_llm_deployment

@serve.deployment
class PrefixAwareRouter:
 def __init__(self, replicas):
 self.replicas = replicas # list of vLLM handles

 async def __call__(self, request):
 body = await request.json()
 messages = body.get("messages", [])

 # Compute the hash of the system prompt (static prefix)
 system_content = ""
 for msg in messages:
 if msg["role"] == "system":
 system_content = msg["content"]
 break

 prefix_hash = hash(system_content)
 # Deterministic routing: same hash → same node
 replica_idx = prefix_hash % len(self.replicas)
 return await self.replicas[replica_idx].remote(request)

With an L7 gateway (Nginx/Traefik):

# nginx.conf — routing by X-Prefix-Hash header
upstream vllm_backends {
 hash $http_x_prefix_hash consistent;
 server vllm-0:8000;
 server vllm-1:8000;
 server vllm-2:8000;
 server vllm-3:8000;
}

The client computes the hash of the static prefix and includes it as a header:

import hashlib, requests

def llm_request(messages, base_url):
 system_msg = next((m["content"] for m in messages if m["role"] == "system"), "")
 prefix_hash = hashlib.sha256(system_msg.encode()).hexdigest()[:16]

 return requests.post(
 f"{base_url}/v1/chat/completions",
 json={"messages": messages, "model": "my-model"},
 headers={"X-Prefix-Hash": prefix_hash}
 )

Measuring the impact with OTel

# Current hit rate (0.0 to 1.0) — target > 0.70 with enterprise workloads
vllm:gpu_prefix_cache_hit_rate

# TTFT by percentile — should fall when the hit rate rises
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse correlation between hit rate and TTFT is the proof that the cache is working. If the hit rate goes from 15% to 70% and the p50 TTFT does not change, there is a configuration problem: the cache may be disabled, or the routing may not be sending requests to the right node.

Correlation query in Grafana (two-axis panel):

# Left Y axis: hit rate
vllm:gpu_prefix_cache_hit_rate

# Right Y axis: TTFT p50 (inverted)
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse slope should be visible: when the hit rate drops (a spike of requests with new prompts), TTFT rises. When the hit rate stabilises (users repeating the same flow), TTFT falls.

The impact in numbers

For a system with 100 req/min, a 512-token system prompt and a before/after hit rate:

Metric	Hit rate 15%	Hit rate 75%	Difference
Prefill tokens per minute	5,100	12,800 — 50% cached → 6,400 effective	−37% load
TTFT p50 (512 prompt + 100 suffix)	~820 ms	~180 ms (suffix only)	−78%
Prefill capacity freed	—	+1,200 tok/min	available for more requests

The 75% hit rate in this example is equivalent to being able to serve 37% more requests on the same hardware, because the prefill work of 3 out of every 4 requests is already done.

When the prefix cache does not help

The prefix cache is ineffective in workloads where every request has a completely unique prompt: translating a different document every time, code analysis with always-different context, creative generation with no system prompt. In these cases the hit rate structurally cannot exceed 5-10% and the template engineering effort does not pay off.

The signal: if your p99 input length is greater than the p50, you have high prompt variance and the prefix cache contributes little. If the p50 and the p99 are similar (consistent prompts), the prefix cache is the cheapest lever available.

References

LLM tracing with OpenTelemetry GenAI: the aircraft black box the field stabilised in 2026

Wed, 27 May 2026 11:00:00 +0200

This post complements the one on Prompt versioning with Langfuse and MLflow, where prompt_id@version appeared as a span attribute without the whole pipeline being explained, and the one on Evals for LLMs, which distinguishes tracing from eval. Here we go inside tracing itself.

TL;DR

OpenTelemetry GenAI, the gen_ai.* and mcp.* semantic conventions, have been stable since late 2025, and as of May 2026 they are the substrate on which the whole ecosystem (Langfuse, Phoenix, LangSmith, Braintrust, Arize, Honeycomb, Datadog, New Relic) builds its LLM observability. That means your team instruments once, against OTel, and picks a backend afterwards; not the other way round. The canonical pipeline is application → OTel SDK (OpenLLMetry or openinference) → OTel Collector → backend(s), with two-layer sampling (head-based at 1-5 % to keep volume viable, tail-based at 100 % over errors and high latencies). This post takes apart the exact attributes you have to fill in on each span (gen_ai.system, request.model, usage.input_tokens, and so on), the operational difference between the three usual SDKs, and the anatomy of a real chat-with-RAG-and-tool trace with all its child spans.

You are here: OBSERVE

The analogy: the aircraft black box

An aircraft black box does three things at once:

It records everything without anyone having to ask for it explicitly. Every flight parameter, every cockpit conversation, every action by the pilots is logged with a precise timestamp and a standardised format.
It uses a format everybody knows. Any air accident investigator anywhere in the world can read the black box of any manufacturer, because the schema is agreed between all of them. There is no “Boeing black box” and “Airbus black box”: there is one single black box.
It survives the event: it is designed on the assumption that the aircraft may go down. It does not depend on the aircraft itself to be stored or to be read.

LLM tracing with OTel GenAI does exactly those three things:

It records everything automatically. Every model call, every agent step, every tool call, every RAG retrieval ends up as a span, a unit of work with a start, an end, attributes and a parent/child relationship with other spans.
It uses a common format. The attributes gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.response.finish_reasons and the rest are agreed by the OpenTelemetry GenAI working group and every backend respects them.
It survives the backend. Spans do not travel straight to Langfuse: they travel to an intermediate OTel Collector that handles buffering, sampling and fan-out to 1, 2 or N backends. If Langfuse goes down, the Collector holds the spans.

The analogy is not decorative: it is the only way to understand why OTel GenAI beat the ecosystem of proprietary SDKs that every vendor shipped in 2023-2024. The standardised black box beats the proprietary recorder, every time.

Why OTel GenAI exists (and why it won)

The state of things at the end of 2024 was broken: every vendor shipped its own SDK with its own attribute names. LangSmith called it model_name, OpenLLMetry called it llm.model, Helicone called it model. If you wanted to change observability provider, you re-instrumented the whole application. For serious teams, that was unacceptable.

OpenTelemetry already had a framework to solve this: the semantic conventions. A working group (the OpenTelemetry GenAI SIG) proposed a schema. In 2025 it reached “Stable” for the attributes of the basic modality (chat completions + tool calls), and in 2026 it also covers multimodal, embeddings and reasoning modalities.

What changes in practice:

Before (proprietary SDKs): you move from Langfuse to Phoenix → you re-instrument, re-deploy everything, lose the history.
Now (OTel): you move from Langfuse to Phoenix → you edit the exporter in otel-collector-config.yaml, reload the Collector, and the new spans go to the new destination without touching a line of application code.

The canonical `gen_ai.*` attributes

As of May 2026 the stable attributes, that is, the ones that are not going to change, are those in the table. There are more in “experimental” state (multimodal, reasoning) and others in “deprecated” (kept for compatibility).

Attribute	Meaning	Example
`gen_ai.system`	Provider family	`openai`, `anthropic`, `vllm`, `huggingface`
`gen_ai.request.model`	Model requested	`gpt-4o`, `meta-llama/Llama-3.1-8B-Instruct`
`gen_ai.request.temperature`	Sampling temp	`0.7`
`gen_ai.request.max_tokens`	Output cap	`1024`
`gen_ai.request.top_p`	Nucleus sampling	`0.95`
`gen_ai.response.model`	Model that answered (may differ)	`gpt-4o-2024-08-06`
`gen_ai.response.id`	ID of the provider’s response	`chatcmpl-9xY...`
`gen_ai.response.finish_reasons`	Finish reasons	`["stop"]`, `["length"]`, `["tool_calls"]`
`gen_ai.usage.input_tokens`	Input tokens	`1247`
`gen_ai.usage.output_tokens`	Output tokens	`412`
`gen_ai.operation.name`	Operation type	`chat`, `text_completion`, `embeddings`
`gen_ai.prompt.id`	ID of the versioned prompt	`customer_support_v3`
`gen_ai.prompt.version`	Specific version	`14`
`gen_ai.prompt.label`	Semantic label	`production`, `staging`, `canary`
`gen_ai.conversation.id`	Multiturn conversation ID	`session_abc123`

And for tool calls (a function invoked by the model), the gen_ai.tool.* attributes:

Attribute	Meaning
`gen_ai.tool.call.id`	ID of the specific tool call
`gen_ai.tool.name`	Name of the tool
`gen_ai.tool.type`	`function`, `mcp`, and so on
`gen_ai.tool.description`	Short description

And for tool calls over MCP (Model Context Protocol), the mcp.* attributes add a specific layer (mcp.server.name, mcp.method, mcp.session.id and others) that coexists with the gen_ai.* ones.

The practical rule: if your instrumentation fills in these attributes on every span, the Langfuse dashboard, the Tempo graph and the Prometheus metrics come out “for free”, because all three know how to read the convention.

The canonical pipeline: SDK → Collector → backends

Three layers, each with its own responsibility:

1 · The SDK

Three dominant options in 2026:

OpenLLMetry (Traceloop) — the most widespread. It automatically instruments the OpenAI Python SDK, the Anthropic SDK, LangChain, LlamaIndex, Haystack, Cohere, Mistral, vLLM and almost any library in the ecosystem. A single line (Traceloop.init()) instruments everything. Apache 2.0 licence. It maintains its own fork with extensions that are not yet in OTel core, but it exports in line with the convention.

openinference (Arize) — a direct competitor, also Apache 2.0. Closer to Phoenix (same vendor) but it exports standard OTel. Better instrumentation for LangChain/LlamaIndex/DSPy in some versions; worse for vLLM directly.

Langfuse SDK — proprietary in form (the API is Langfuse’s), but underneath it emits OTel spans. The natural choice if Langfuse is the main backend. It has the best support for “session” and “user” linking, concepts that are not yet in OTel core but that Langfuse maps to gen_ai.conversation.id for compatibility.

Practical recommendation for May 2026: OpenLLMetry if you want to stay backend-agnostic (you will change provider); the Langfuse SDK if Langfuse is already your bet (you save yourself a mapping). Both produce valid spans that any Collector consumes.

2 · The OTel Collector

The Collector is the most important piece of the pipeline and the least talked about. Three responsibilities:

Buffering: if Langfuse or Tempo have a problem, the Collector holds the spans in memory (or on disk with persistence enabled) and re-emits them when the backend returns.

Sampling: it applies the two sampling layers (next section) without the application knowing anything about it.

Routing: with the exporters config it points at one or several backends at once. The usual practice is to send to Langfuse + Tempo + Prometheus simultaneously:

Langfuse for the LLM-first UI (prompts, evals, user sessions).
Tempo (or Jaeger) for the full distributed context (an LLM span inside an HTTP request that has touched 12 microservices).
Prometheus for aggregated metrics (P95 latency per model, tokens per second, error rate).

A representative production fragment of otel-collector-config.yaml:

receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

processors:
 batch:
 timeout: 1s
 send_batch_size: 1024
 tail_sampling:
 decision_wait: 10s
 policies:
 - name: errors
 type: status_code
 status_code: { status_codes: [ERROR] }
 - name: slow_traces
 type: latency
 latency: { threshold_ms: 5000 }
 - name: head_5pct
 type: probabilistic
 probabilistic: { sampling_percentage: 5 }

exporters:
 otlphttp/langfuse:
 endpoint: https://langfuse.internal:3000/api/public/otel/v1
 headers: { Authorization: "Basic ${LANGFUSE_AUTH}" }
 otlp/tempo:
 endpoint: tempo:4317
 tls: { insecure: true }
 prometheus:
 endpoint: 0.0.0.0:8889

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [tail_sampling, batch]
 exporters: [otlphttp/langfuse, otlp/tempo]
 metrics:
 receivers: [otlp]
 processors: [batch]
 exporters: [prometheus]

3 · The backends

Langfuse is the dominant option for OSS LLM observability (MIT licence, complete suite: tracing + evals + prompts + datasets). Its model assumes OTel trace_id and span_id and draws them in the UI with the LLM context expanded (formatted messages, counted tokens, costs calculated from a model→price mapping).

Phoenix (Arize, ELv2 licence) has a more eval-first focus. Useful if your team comes from classic NLP and wants UMAP embedding visualisation by default.

LangSmith (LangChain) is proprietary, not OSS. Excellent integration with LangChain/LangGraph; less relevant if you do not use that stack.

Tempo / Jaeger show the LLM traces inside the context of the distributed request. It is what you need when a customer reports that “the page takes 8 seconds” and you want to see whether those 8 seconds are the model, the retrieval or the downstream microservice.

Sampling: the two layers that coexist

Without sampling, a system at 10 req/s generates about 26 million spans a month. The backend drowns, the cost explodes and most of the spans are redundant (200 similar requests tell you no more than 20).

The canonical pattern in 2026 is two combined layers:

Head-based sampling (on the way in)

You decide before the span exists whether you are going to trace it or not, based on a probabilistic decision (1-5 % typically) or on a deterministic rule (always trace if the user is premium, if the model is the new one in canary, and so on).

Pros: cheap (the span is not generated at all), deterministic.

Cons: you can miss infrequent errors. If an error happens once in every 1,000 and you trace 1 %, you will see it once in every 100,000, which is no longer debuggable.

Tail-based sampling (on the way out)

The complete span is generated and at the end you decide whether to keep it or throw it away. The decision can be:

Keep 100 % of errors.
Keep 100 % of traces with latency > X ms.
Keep 100 % of traces that triggered a guardrail.
Keep a random sample of the rest (1-5 %).

Pros: it guarantees that you see the errors and the latency outliers. Without it, the most interesting problems are the ones most likely to be left out.

Cons: you need a buffer, because until the trace finishes you cannot decide. The Collector’s tail_sampling processor keeps the spans in memory for 5-30 seconds until it takes the decision.

The practical rule in 2026 is to combine both: head-based at 5 % + tail-based capturing 100 % of errors and latencies > 5s. That gives you around 5 % of baseline traffic plus 100 % of what is interesting.

Anatomy of a real trace

We are going to take apart the trace of a single user question to a typical chat-with-RAG-and-tool system. The question is “what is the balance of my main account?”.

Seven spans, one single trace. The information you can exploit:

Latency broken down: 38 ms guardrail + 218 ms retrieval + 2104 ms LLM + 60 ms tool + 28 ms guardrail = 2,448 ms (the rest is orchestration overhead). The bottleneck is the LLM (74 % of the time). If the user complains about slowness, you know where to look.
Tokens and cost: 1,247 input + 412 output. With Llama 3.1 70B on-prem at about 12 W/token of equivalent energy cost, roughly 3.5 c€ per response. Multiplied by volume, that gives you the inference bill.
prompt.id+version travelling along: if in a month’s time you notice that quality has dropped, you filter by prompt.version and see whether it coincides with a change to the prompt.
Guardrail decisions traced: if a guardrail blocked the response, it stays in the trace with gen_ai.guardrail.decision=block and a reason. ENS audit satisfied.
Correlated tool calls: tool.get_balance is a child span of gen_ai.chat. If the tool failed, you see the error directly in its span; it does not turn up only in separate logs.

Implications for on-premise hardware

Tracing is not free, but its overhead, properly sized, is negligible.

Cost per span

At the SDK level, an LLM span adds around 50-200 µs of overhead (creation, attributes, serialisation). For an LLM call lasting 2 seconds, that is < 0.01 % of the latency.

At the network level, a typical compressed OTLP span weighs around 1-3 KB. For a system at 100 req/s with 7 spans per request, that is 700 spans/s × 2 KB ≈ 1.4 MB/s to the Collector. Trivial.

Storage

This is where the money goes. Without sampling, a month at 100 req/s is about 18 GB of spans. With head-based at 5 % + tail-based on errors, about 1-2 GB a month. Manageable in any self-hosted ClickHouse or Loki.

On an RTX 4090 (24 GB) + Collector and Langfuse on-prem

For a demo service or a small single-tenant: Langfuse + Postgres + Clickhouse + Collector on the same host as the model. It uses around 8 GB of RAM and < 1 % of CPU continuously. The 4090 serves the model, the rest lives on the CPU. Minimum viable setup for a startup or a small team.

On a generic 4×H100 SXM cluster + central observability

A dedicated pod for the OTel Collector (a DaemonSet on each node + a central Gateway). Langfuse + ClickHouse + Tempo + Prometheus + Grafana in an observability namespace separate from the serving namespace. This is the canonical architecture of any serious ENS/NIS2 production deployment: observability does not share resources with the model.

What we have not covered (upcoming articles)

LLM metrics (Prometheus): the canonical counters and histograms — gen_ai_client_request_duration_seconds, gen_ai_client_input_tokens_total and the rest. How Grafana dashboards are built on that base.
Guardrails as spans: how to model Llama Guard 4, NeMo Guardrails and LLM Guard as child spans of the LLM span, and what attributes they carry. Covered in the post on guardrails — attributes gen_ai.guardrail.line, gen_ai.guardrail.detector, gen_ai.guardrail.category, gen_ai.guardrail.score, gen_ai.guardrail.action.
Distributed tracing with MCP — traceparent propagation from the LLM client to the MCP server, mcp.* attributes, known problems.
eBPF for automatic tracing without an SDK — Tetragon and Hubble extracting LLM traces without explicit instrumentation, for cases where the code cannot be modified.

References

OpenTelemetry GenAI Semantic Conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/.
OpenLLMetry (Traceloop): https://github.com/traceloop/openllmetry.
openinference (Arize): https://github.com/Arize-ai/openinference.
Langfuse OTel integration: https://langfuse.com/docs/opentelemetry/get-started.
OTel Collector tail-sampling processor: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor.
Phoenix (Arize) docs: https://docs.arize.com/phoenix.
Model Context Protocol observability (mcp.* semconv): https://modelcontextprotocol.io/docs/observability.

MCP from the inside and its deep observability: the LSP of AI agents and how to see everything with OpenTelemetry

Wed, 20 May 2026 06:00:00 +0200

TL;DR

Model Context Protocol (MCP) is the standard Anthropic published at the end of 2024 and which by 2026 has become the dominant protocol for connecting AI agents to external tools and data. Its value, the reason the whole industry adopted it in under 18 months, is that it solves a combinatorial problem: before MCP, integrating M AI apps with N tools required M×N ad-hoc integrations; with MCP, M + N. It is the same move the Language Server Protocol made in 2016 for code editors. The architecture is three well-defined roles, Host (the AI app), Client (the connection, one per server) and Server (the piece that exposes capabilities); the primitives are six, three on the server side (Tools, Resources, Prompts) and three on the client side (Sampling, Roots, Elicitation); the protocol is JSON-RPC over two transports, stdio for local processes and Streamable HTTP for remote. The operational challenge appears when there are 10-20 MCP servers running simultaneously, each with several tools, connected to an agent that chains multistep calls: observing what happens, where things fail, how much each tool costs, which tenant invokes what becomes critical. The ecosystem’s answer in 2026: the new OpenTelemetry GenAI semantic conventions for MCP (already stable), trace context propagation via params._meta (because JSON-RPC does not bring it natively), FastMCP with built-in OTel instrumentation, MCP Gateways as a centralised layer (Traefik Hub, MintMCP, OpenObserve), and MCP Inspector for interactive debugging. This article walks the architecture from the outside in, puts each concept in its exact place, and goes down to the detail of observability: traces, RED metrics, real use cases and pitfalls.

This is the third post in the post-tracing series. Previous posts: Evals and Guardrails. Here we go down to the protocol that connects agents to tools, and how to see it in production.

The master analogy (in three versions)

MCP is a communication protocol. Like any protocol, it is best understood with the right analogy. I am going to give you three, because each one lights up a different facet and the combination leaves you understanding it better than any technical definition.

Version 1 — The USB-C of AI apps (the official one)

This is the analogy Anthropic adopted when presenting it. Before USB-C, every electronic device had its own connector. Your phone had microUSB or Lightning, your laptop a proprietary power port, your headphones a 3.5mm jack, your external disk USB-A at one end and mini-USB at the other. The result: three boxes full of specific cables that got lost, none of them good for two things, and buying a new device meant buying new accessories.

USB-C changed that. A single physical connector that many protocols cross: data (USB 3, USB 4, Thunderbolt), video (DisplayPort), power (Power Delivery), audio. You connect anything to anything and it works; the protocols negotiate above.

MCP plays the same role for AI apps. Before MCP, every application that wanted to integrate tools with an LLM, Claude Desktop, Cursor, Continue, in-house custom agents, invented its own way of doing it. Every tool vendor had to write N different integrations, one per app. The result: massive fragmentation, a lot of duplicated code, integrations that broke when an app changed its internal API.

With MCP, the connector is one: any app that speaks MCP can use any MCP tool. Just as your USB-C talks to printers, monitors and disks without the printer “knowing” whether the cable is connected to a Mac or to a Linux box.

Version 2 — The LSP of code editors (the most technically precise)

This is my favourite because the analogy is structurally identical, not just metaphorical.

Until 2016, if you wanted your code editor to support a new language, Rust, Go, TypeScript, somebody had to write a specific plugin for your particular editor. VSCode had its Rust plugin, IntelliJ a different one, Vim another, Emacs another. Every decent feature (go-to-definition, autocompletion, refactoring) was an implementation duplicated N times. M editors × N languages = M·N integrations.

In 2016 Microsoft proposed the Language Server Protocol (LSP): each language implements a single “language server” (a process that understands that language); each editor implements a single LSP client; when you work with Rust code in VSCode, VSCode launches rust-analyzer as a subprocess and talks LSP to it over stdio. Any LSP editor + any LSP server = it works. M + N.

MCP is literally this pattern, transferred from “editor + language server” to “AI app + tool provider”. And it shares even the technical detail: both pass JSON-RPC over stdio (among other transports). When Anthropic designed MCP, they looked at LSP. Anyone coming from the world of editors and IDEs will find MCP familiar.

Version 3 — The operating system driver (the operational one)

Finally, an analogy that helps you understand what a specific MCP server does.

An operating system does not know directly how to talk to your particular HP LaserJet printer. What it knows is a generic interface: “print document”, “query status”, “cancel job”. The printer driver is the piece that translates that generic interface into the proprietary commands of your specific printer.

An MCP server does exactly the same:

Your AI agent knows a generic interface: invoke a tool with a defined schema, read a resource by URI, ask for a prompt template by name.
The MCP server is the driver: it translates those generic operations into the concrete APIs of the underlying system, your PostgreSQL database, your filesystem, your GitHub API, your Stripe.

This leaves the AI agent free from knowing how it authenticates with GitHub, what exact SQL PostgreSQL uses, what endpoints Stripe has. It speaks MCP; the server takes care of the details.

With the three analogies combined: MCP is the layer between the LLM and the world, a standard USB-C implemented like LSP in JSON-RPC, with each server acting as a driver for one concrete underlying system.

What concrete problem MCP solves

Before going down to the architecture, it is worth pinning down the specific problem MCP solves, because without that many design decisions look arbitrary.

The problem is the quadratic cost of integrations.

Imagine you have M applications that use LLMs (Claude Desktop, Cursor, Continue, ChatGPT Desktop, your own custom agent, and so on) and N external tools those LLMs could use (filesystem, GitHub, Slack, PostgreSQL, Jira, Notion, and so on). Without a standard:

Each (application, tool) pair requires a specific integration.
Every time the application changes its internal API, N integrations have to be updated.
Every time the tool changes its API, M have to be updated.
For your new tool to be adopted, you have to write M integrations.
For your new application to support the ecosystem, you have to write N.

The real result in 2023-2024: massive fragmentation. OpenAI’s function calling was not compatible with Anthropic’s tool use; every framework (LangChain, LlamaIndex, dspy) had its own wrapper; Claude Desktop plugins did not work in Cursor; and so on.

MCP breaks the quadratic. Each application implements the protocol once; each tool implements the protocol once; any pair works. M + N.

It is exactly what happened with USB-C, with LSP, with SQL (before there were proprietary APIs per database), with POSIX (before there were proprietary APIs per operating system). The pattern repeats because it always solves the same kind of problem.

The architecture: three roles, clearly placed

We are going to pin down where each thing lives, because mixing up the roles is the number one source of confusion in MCP.

Three roles. Let us pin down what each one does and where it physically lives.

Host: the AI application

The Host is the application the user opens. Claude Desktop, Cursor, Continue, ChatGPT Desktop, a custom agent your team builds, a VSCode extension. What the user perceives as “the product”.

The Host is responsible for:

Deciding which MCP servers to connect (configured by the user in a file or via the UI).
Launching or connecting to each MCP server.
Creating one MCP Client per server (it is 1:1, they are not shared).
Embedding the LLM (or calling it via API) that takes the decisions about which tools to use.
Mediating the user’s authorisation for sensitive actions (showing the human “the agent wants to run tool X, do you allow it?”).

Important: the LLM lives inside the Host, not in the servers. The servers are dumb; they execute operations when asked. The reasoning (“should I call this tool now?”) lives in the host’s LLM.

Client: the connection, one per server

An MCP Client is a specific connection between the Host and a Server. If your Host has 5 MCP servers configured, it has 5 clients, not one shared. Each client:

Maintains its socket or stdio pipe with the server.
Negotiates capabilities in the initial handshake (which protocol version, which primitives both support).
Serialises JSON-RPC requests to the server and deserialises responses.
Is the point where the Host invokes operations on the server.

The 1:1 client-server separation matters because it lets each server have its own session state, its specific permissions and its independent authenticated context. There is no multiplexing in the client.

Server: the piece that exposes capabilities

The MCP Server is the piece that implements the tool-provider side of the protocol. It receives JSON-RPC from the client, processes it, executes the action against the underlying system and returns a response.

There are two flavours physically:

Local server: it starts as a subprocess of the Host and communicates over stdio. Its lifecycle is the Host’s (when you close Claude Desktop, the local servers die). Typical model: your Host launches node filesystem-mcp-server.js as a child.
Remote server: it runs as an independent service, reachable over HTTP. Multi-tenant, authenticated, scalable. Typical model: a company publishes https://mcp.acme.com/v1 and many hosts connect to it.

This difference has enormous consequences for observability (we will come back to it shortly).

Summary of where each thing lives

Component	Lives in	How many there are	Speaks what with whom
Host	User’s machine	1 (the open app)	UI with the user; launches clients
LLM	Embedded in the Host (or cloud API)	1 (the main one)	Reasons; asks for tools
Client	Host	1 per server	JSON-RPC with its server
Local server	Subprocess of the Host	1 per local integration	stdio with its client
Remote server	External service	1 per service	HTTP/SSE with its clients
Underlying system	External	Depends	API/DB/FS, not MCP

If you get confused in a discussion, come back to this table. The number one source of errors in MCP is saying “the server” when you mean “the host”.

The two layers of the protocol

MCP separates the data layer and the transport layer. This separation is what lets the protocol work over local stdio and over remote HTTP without changing anything in the primitives.

Data Layer: JSON-RPC with MCP extensions

The data layer defines the vocabulary of the messages. It is JSON-RPC 2.0. Each message is a JSON with jsonrpc: "2.0", a method (eg tools/call, resources/read), params, and an id to correlate request with response.

On top of JSON-RPC, MCP adds:

Lifecycle: the initial handshake (initialize, initialized) that negotiates capabilities.
The primitives (next section): tools/*, resources/*, prompts/*, sampling/* and so on.
Notifications: messages without a response (eg notifications/cancelled to abort a tool in progress).
Meta-information: the params._meta field carries cross-cutting metadata by convention (trace context, request IDs).

Transport Layer: how the messages move

The transport layer defines how the JSON-RPC messages travel. Two official transports:

stdio: the client launches the server as a subprocess and they communicate over its stdin/stdout/stderr with JSON-RPC. One message per line, separated by a newline. No network, no TLS handshake, no auth (trust is inherited from the operating system itself: if you launch the subprocess, you trust it). Minimal latency (~100 μs round-trip), maximum bandwidth (memcpy, not a socket).

Use case: local servers that live on the same machine as the host. Most of the MCP servers you see in public directories are stdio.

Streamable HTTP: the client sends a POST to an HTTP endpoint on the server; the server responds with JSON, optionally opening a Server-Sent Events stream to send asynchronous notifications or long responses. Auth by bearer token, API key or custom headers.

Introduced in the November 2025 spec, it replaces the pure SSE transport of earlier versions, which had bidirectionality limitations. Use case: remote servers serving many simultaneous clients, with authentication and multi-tenancy.

Important: the primitives are the same on both transports. A tools/call is identical over stdio and over HTTP. The transport is accidental, not fundamental.

The six primitives: placed in the architecture

Here is the meat. There are six primitives in MCP. They are often confused because several look like they do similar things. The key classification: three live on the server side (the server exposes, the client consumes) and three on the client side (the client exposes, the server consumes).

Server-side: what the server gives the host

Tools are actions the server exposes. Each tool has a schema (typed parameters, description) and an implementation. When the host’s LLM decides to invoke a tool, the client sends tools/call to the server, which executes it and returns a result.

Example: the github-mcp server exposes create_issue(repo, title, body). The host’s LLM decides “I am going to create an issue”, calls this tool, github-mcp talks to the GitHub API, returns the issue ID to the LLM.
Architectural place: the server exposes them, the LLM consumes them.

Resources are contextual data the server exposes, addressable by URI. They are not actions; they are content reads. A resource has a URI (file:///path/to/doc.md, postgres://table/users), metadata and an endpoint to read content.

Example: the filesystem-mcp server exposes the files in the authorised directories as resources. The LLM asks for resources/read with the URI file:///docs/api.md and gets the text.
Architectural place: the server exposes them, the host reads them (and optionally passes them to the LLM as context).

Key difference between Tools and Resources: Tools are verbs (they execute, they modify state, they have side effects); Resources are nouns (they exist, they are read, they are idempotent). If you have something that is “search for text in files” → probably a Tool (an action). If it is “this specific file” → a Resource. The distinction matters for auditing and permissions: tools require more control.

Prompts are parameterised prompt templates the server exposes. The user or the host can invoke them to inject a conversational pattern into the model.

Example: a code-review-mcp server exposes a prompt review_diff(diff_text, style="strict") that returns a complete, well-written prompt asking the LLM to review code.
Architectural place: the server exposes them, the user or the host invokes them, the LLM receives them as input.

Prompts are the least used of the three primitives; many servers do not even implement them. But they let a team publish good prompts as a reusable library, separate from the agent.

Client-side: what the host gives the server

This is where MCP differs from protocols like HTTP REST: the server can also ask things of the host, it is not a one-way street. Three primitives travel in that direction.

Sampling: the server asks the host to run a generation with its LLM. That is, the server borrows the host’s LLM in order to reason.

Example: the search-mcp server receives a query from the agent, searches its corpus, finds 50 results and needs to summarise them before returning. Instead of having its own LLM, it sends a sampling/createMessage to the client; the host passes this to its LLM, runs the generation with the user’s permissions, and returns the summary to the server.
Architectural place: the server asks for it, the host (with its LLM and the user’s authorisation) fulfils it.
Why it matters: the user controls which model is used, what cost is paid, what permissions apply. The server does not need its own OpenAI API key.

Roots: the host tells the server where to look. Roots are URIs (directories, repositories, namespaces) the host authorises the server to explore.

Example: your Claude Desktop starts filesystem-mcp with roots [file:///Users/me/projects]. The server knows it must only operate inside that folder, not in /etc/passwd.
Architectural place: the host declares them in the handshake, the server respects them.

Elicitation: the server asks the host for additional information from the human user via a structured UI.

Example: the stripe-mcp server is about to process a refund of €5000. Before executing, it sends an elicitation/createMessage to the client; the host shows the user “Confirm this €5000 refund” with a button; when the user confirms, it returns OK to the server, which then proceeds.
Architectural place: the server asks, the host shows the user, the user decides, the answer goes back to the server.
It is the key primitive for human-in-the-loop on sensitive actions.

Visualising the flow of the six primitives

 HOST SERVER
│ │
Server-side ─────┼─────────────────────────────────────┤
│ │
tools/list ──────┼─────── asks what tools exist ──────▶│
│◀────── returns the list ────────────│
│ │
tools/call ──────┼────────── run this tool ───────────▶│
│◀────── result ──────────────────────│
│ │
resources/read ──┼────────── read this URI ───────────▶│
│◀────── content ─────────────────────│
│ │
prompts/get ─────┼─────── give me this prompt ────────▶│
│◀────── compiled prompt ─────────────│
│ │
Client-side ─────┼─────────────────────────────────────┤
│ │
sampling ────────│◀────── I need a generation ─────────│
│── use my LLM ───┐ │
│── returns ──────▼──────────────────▶│
│ │
roots ───────────┼───── declared in the handshake ────▶│
│ │
elicitation ─────│◀────── ask the user X ──────────────│
│── shows UI ────┐ │
│── confirms ────▼───────────────────▶│

JSON-RPC in action: a concrete example

To make the theory concrete, here is a real MCP conversation between a client and the filesystem-mcp server:

// 1. Initial handshake (client → server)
{
"jsonrpc": "2.0", "id": 1, "method": "initialize",
"params": {
"protocolVersion": "2026-03-01",
"capabilities": {
"sampling": {}, // this client supports sampling
"roots": { "listChanged": true }
},
"clientInfo": { "name": "ClaudeDesktop", "version": "1.2.0" }
}
}
// 2. The server responds with its capabilities
{
"jsonrpc": "2.0", "id": 1, "result": {
"protocolVersion": "2026-03-01",
"capabilities": {
"tools": { "listChanged": true },
"resources": { "subscribe": true, "listChanged": true },
"prompts": {}
},
"serverInfo": { "name": "filesystem-mcp", "version": "0.5.2" }
}
}
// 3. The client asks for the list of tools
{
"jsonrpc": "2.0", "id": 2, "method": "tools/list"
}
// 4. The server returns its tools with a schema
{
"jsonrpc": "2.0", "id": 2, "result": {
"tools": [
{
"name": "read_file",
"description": "Read a file from the filesystem",
"inputSchema": {
"type": "object",
"properties": { "path": { "type": "string" } },
"required": ["path"]
}
},
{ "name": "write_file", "description": "...", "inputSchema": {} },
{ "name": "list_directory", "description": "...", "inputSchema": {} }
]
}
}
// 5. The LLM decides to call read_file; the client sends tools/call
{
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
"params": {
"name": "read_file",
"arguments": { "path": "/Users/me/projects/notes.md" },
"_meta": { // ← the extension where trace context will go
"traceparent": "00-abc123...-def456-01"
}
}
}
// 6. The server returns the file content
{
"jsonrpc": "2.0", "id": 3, "result": {
"content": [
{ "type": "text", "text": "# My notes\n\n..." }
]
}
}

The important thing to note: params._meta. That is the bag where MCP conventionally passes cross-cutting metadata, including trace context. We will come back to it shortly.

The observability problem: why traditional tracing is not enough

So much for theory. Let us go down to the operational problem: in a 2026 production cluster, a typical agent has 5-15 MCP servers connected simultaneously, each with 5-20 tools, and each conversation with the agent can generate dozens of chained tool calls. Without observability, debugging incidents is impossible.

Why generic tracing (Hubble, OTel without MCP conventions) is not enough:

Stdio is not visible on the network. Local servers talk over OS pipes. Your Hubble or your Datadog APM see nothing; there are no packets to capture. AgentSight (seen in the previous post in the eBPF series) with stdiocap captures it but gives you the raw JSON-RPC, without semantic context (which tool it is, which resource, which prompt).

Generic HTTP does not understand MCP either. If you trace the HTTP to a remote MCP server without MCP conventions, you see a POST to /v1 with an opaque JSON-RPC body. You lose “which tool was invoked”, “what arguments”, “was it elicitation or sampling”. RED metrics per endpoint are no use to you; you need RED per tool.

JSON-RPC does not propagate trace context natively. Unlike HTTP (the W3C traceparent header) or gRPC (metadata), JSON-RPC has no standard field for trace context. If you do not propagate it, every call to the server starts a new trace disconnected from the agent’s trace.

Multistep multi-server is very hard to follow. A single user conversation can translate into: 1) a call to github-mcp get_pr; 2) a call to filesystem-mcp read_file for several files; 3) a call to the main LLM with all the context; 4) a call to postgres-mcp query; 5) a call to slack-mcp send_message. Without propagated trace context, those are five disconnected traces. With propagation, it is one tree.

The solution: OpenTelemetry semantic conventions for MCP, already stable in 2026.

OpenTelemetry semantic conventions for MCP

The GenAI MCP semantic conventions are the set of standardised attributes for MCP-related spans and metrics. They were published as part of the GenAI subgroup of the OpenTelemetry SIG and are the first part of the GenAI semantic conventions to reach stable.

Why MCP-specific semantic conventions

Before they existed, teams instrumented MCP with the generic RPC semantic conventions (the ones you would use for gRPC or XML-RPC). It half worked. The MCP-specific conventions add:

Attributes to identify which primitive was executed (mcp.method.name = "tools/call").
Attributes to identify which specific tool/resource/prompt was touched (mcp.tool.name, mcp.resource.uri, mcp.prompt.name).
Attributes for the bidirectional flow (sampling/elicitation requests from the server to the client).
Attributes for the handshake (mcp.protocol.version, mcp.client.name, mcp.server.name).
Standardised RED metrics per tool (mcp.tool.call.duration, mcp.tool.call.errors).

The canonical attributes

The attributes any MCP-aware instrumentation should emit:

Attribute	Meaning	Example
`mcp.method.name`	JSON-RPC method	`"tools/call"`
`mcp.tool.name`	Name of the tool	`"read_file"`
`mcp.resource.uri`	URI of the resource	`"file:///docs/api.md"`
`mcp.prompt.name`	Name of the prompt	`"code_review"`
`mcp.session.id`	MCP session ID	`"sess-abc123"`
`mcp.protocol.version`	Protocol version	`"2026-03-01"`
`mcp.client.name`	Client identity	`"ClaudeDesktop/1.2.0"`
`mcp.server.name`	Server identity	`"filesystem-mcp/0.5.2"`
`mcp.transport`	Transport used	`"stdio"` or `"http"`
`mcp.error.code`	JSON-RPC error code	`-32602` (Invalid params)
`gen_ai.usage.input_tokens`	Tokens consumed (if sampling)	`1240`
`gen_ai.usage.output_tokens`	Tokens generated (if sampling)	`512`

The last two come from the generic GenAI semantic conventions and apply when the MCP call involves sampling (the server using the client’s LLM).

RED metrics per tool

Beyond spans, the semantic conventions define three core metrics:

mcp.tool.call.duration (histogram): the latency of each invocation.
mcp.tool.call.count (counter): the total number of invocations.
mcp.tool.call.errors (counter): errors per tool.

Labelled with mcp.tool.name, mcp.server.name, mcp.client.name. Pivotable in Grafana to answer “which tool is the slowest”, “which tool fails most”, “which client loads which server hardest”.

Trace context propagation: the `params._meta` trick

JSON-RPC has no headers like HTTP, so MCP cannot use the W3C traceparent header directly. The solution the ecosystem has converged on: propagate trace context in params._meta.

When the MCP client sends a tools/call, its OTel instrumentation does:

import json
from opentelemetry.propagate import inject

carrier = {}
inject(carrier) # fills it with traceparent/tracestate from the active span

params = {
 "name": "read_file",
 "arguments": {"path": "/notes.md"},
 "_meta": carrier, # ← propagates trace context
}

When the server receives it, it does the symmetrical thing:

from opentelemetry.propagate import extract

ctx = extract(request.params.get("_meta", {}))
with tracer.start_as_current_span("tools/call", context=ctx):
 # this span is a child of the client's
 return execute_tool(request.params)

The result: the server’s span is a child of the client’s span in the trace tree. When you look at the trace in Tempo or Phoenix, you see the whole chain: user → host → client → server → execution → response → client → host → response to the user.

This requires both ends to instrument consistently. If the server does not extract the context, you see disconnected spans but at least you have traceability on the client side.

Instrumentation patterns

There are three routes to instrumenting MCP, in increasing order of effort:

1. FastMCP with built-in OpenTelemetry

FastMCP is one of the most used Python frameworks for building MCP servers. It ships built-in OpenTelemetry instrumentation: every tool, resource template and prompt operation generates spans automatically with the correct MCP conventions.

from fastmcp import FastMCP
from opentelemetry.sdk.trace.export import OTLPSpanExporter

mcp = FastMCP("my-server", otel_endpoint="https://otel-collector:4318")

@mcp.tool()
def search_docs(query: str) -> str:
 """Search the corpus for matching documents."""
 # this automatically generates a span with
 # mcp.tool.name=search_docs, mcp.method.name=tools/call, etc.
 return run_search(query)

Zero instrumentation code. Spans with the correct conventions. It is the recommended pattern if you are starting an MCP server in Python from scratch.

2. Manual OpenTelemetry SDK

For servers that already exist or are in other languages (TypeScript, Go), the option is to instrument manually with the standard OTel SDK and emit the conventional MCP attributes:

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

async def handle_tools_call(req: JSONRPCRequest):
 ctx = extract_trace_context(req)
 with tracer.start_as_current_span("mcp.tools.call", context=ctx) as span:
 span.set_attribute("mcp.method.name", "tools/call")
 span.set_attribute("mcp.tool.name", req.params["name"])
 span.set_attribute("mcp.server.name", "filesystem-mcp")
 try:
 result = await execute_tool(req.params)
 return result
 except Exception as e:
 span.set_attribute("mcp.error.code", -32603)
 span.record_exception(e)
 raise

More boilerplate but it works with any existing server.

3. MCP Inspector for interactive debugging

MCP Inspector (official) is a tool for interactive protocol-level debugging. It launches a local proxy (port 6277) between your client and the server, and opens a web UI (port 6274) where you see every JSON-RPC message back and forth in real time.

It is not production observability, it is development and debugging. But it is irreplaceable during the bring-up of a new server: you see exactly which requests arrive, which responses are returned, which errors occur. It saves hours of ad-hoc logging.

MCP Gateways: the centralised piece for enterprise

When your organisation has many agents connecting to many MCP servers, managing the connection matrix becomes operationally serious. The natural question, “can there be a proxy in front of all the MCP servers that centralises auth, rate limiting, logging and observability?”, already has an answer: MCP Gateways.

An MCP Gateway is a proxy that:

Accepts MCP connections from the hosts/agents.
Routes them to the corresponding backend MCP servers.
Applies centralised authentication and authorisation (which agent can call which tool).
Applies rate limiting per agent, per tool, per tenant.
Observes: it emits OTel metrics for every operation passing through.
Propagates the identity of the agent to the backend server (with several models: token forwarding, token exchange, impersonation).

The options that have established themselves in 2026:

Traefik Hub MCP Gateway — from the Traefik team. Declarative configuration, native integration with Traefik’s Kubernetes/Helm ecosystem.
MintMCP — a gateway focused on observability and multi-tenancy. SaaS and self-host.
OpenObserve MCP Gateway — integrated with the OpenObserve observability platform.

For small deployments (one team, few agents) a Gateway can be overkill. For enterprise (dozens of agents, dozens of servers, regulated compliance), it is practically mandatory.

Real use cases for MCP observability

Let us land this with five cases where properly instrumented MCP observability delivers immediate value:

1. Audit per tool, per tenant, per agent

Question: “who ran the delete_repo tool last month?”. Without MCP observability, impossible. With OTel conventions + identity propagation: a query in your trace backend filtering by mcp.tool.name="delete_repo", grouping by mcp.client.name or by the user_id propagated in _meta. Compliance happy.

2. Cost per tool and per tenant

Question: “how much does each tool cost?”. If the tools invoke external APIs (Stripe, OpenAI sampling) or consume significant resources (a GPU for an inference tool), knowing their aggregated cost matters. With mcp.tool.call.duration + gen_ai.usage.* aggregated by tool and tenant, you build cost accountability dashboards without instrumenting anything extra.

3. Debugging multistep chains that fail

Question: “the agent failed to complete this task, where did it go wrong?”. The propagated trace connects: the user’s span → the LLM’s span with its CoT → the spans of each tool invoked → the final LLM span. If the chain broke at the third tool, in Tempo you see the red span with the specific error message. Reproducing the failure is trivial.

4. Tool latency and degradation

Question: “which tool is degrading?”. RED metrics per tool in Grafana show p95/p99 latency over time. When a tool starts climbing from 200ms to 800ms (because the underlying service is collapsing), you see it before the users complain.

5. Detecting loops and agentic anomalies

Question: “is any agent stuck in a loop?”. If an agent calls tools/call read_file 80 times in 30 seconds for the same path, something is clearly wrong. An alert on mcp.tool.call.count grouped by (session_id, tool_name) detects this. Combined with loop detection at the reasoning level, it closes the circle.

Operational pitfalls

Lack of identity propagation

Your Gateway authenticates the agent, but passes requests to the backend without propagating identity. The result: the backend’s logs say “service-account” for everything, and it is impossible to audit who invoked what. Choose a propagation strategy early: token forwarding (simple, exposes tokens to the backend), token exchange (more secure), or impersonation with cross-logging.

Stdio servers that do not show up in your APM

This is the field’s number 1 pitfall. Your Cursor agent uses filesystem-mcp over stdio; you see nothing in Datadog because there is no network traffic. Solution: instrument the stdio server with an OTel SDK that exports over OTLP to your collector (via gRPC or HTTP; the OTel collector can receive even though the server talks stdio to its client). Or use AgentSight stdiocap to capture the raw JSON-RPC and process it offline.

Multiple protocol versions in production

Different clients use different MCP versions simultaneously. Your metrics dashboard mixes apples and oranges. ALWAYS label with mcp.protocol.version and filter/group by it.

`_meta` lost when passing through a proxy

Your Gateway accepts the client’s request, rewrites it for the backend, and forgets to copy params._meta. The result: the trace is broken at the Gateway, two disconnected traces. Make sure your Gateway preserves or re-injects trace context at each hop.

Trace volume with chatty servers

Some MCP servers emit many small operations (filesystem listings, partial reads). Without sampling, they fill your backend with useless traces. Apply tail-based sampling that keeps complete sessions or keeps only traces with errors or high latency.

Cardinality in metrics

mcp.tool.call.duration with mcp.session.id as a label blows up cardinality. Do not include unique per-session IDs in labels; keep cardinality under control with labels that take few discrete values (tool name, server name, client name, error code).

Confusing client and server spans

When you look at the tree, distinguish: the client sees the total latency from its perspective (network included); the server sees only its own work. If you look only at the server span to debug the latency perceived by the user, you miss the RTT. Use both.

What we have not covered

Experimental WebSocket MCP transport: an alternative to Streamable HTTP, not yet standard.
MCP servers in cloud-native deployments with sidecars: an emerging pattern of deploying MCP servers as pod sidecars.
MCP federation: composing several servers as one (similar to GraphQL federation).
eBPF + MCP: how AgentSight’s stdiocap and Cilium’s hooks complement native instrumentation.
MCP testing and contract tests: how to validate that your server complies with the spec.

References

Specification and concepts:

Model Context Protocol — sitio oficial — entrada canónica.
MCP architecture overview.
Transports — MCP docs.
MCP Inspector (GitHub) — debugging interactivo.

OpenTelemetry GenAI MCP:

Semantic conventions for Model Context Protocol — OpenTelemetry — referencia normativa.
Adding OpenTelemetry Trace Support to MCP (Discussion #269) — historia de la propuesta.
How to Instrument MCP Servers with OpenTelemetry (OneUptime).
How to trace MCP server tool calls with OpenTelemetry and Elastic APM.
MCP Observability with OpenTelemetry (SigNoz).
Distributed tracing for agentic workflows (Red Hat Developer).
OpenTelemetry for AI Agents in MCP Workflows (MintMCP).

Frameworks and gateways:

FastMCP OpenTelemetry — instrumentación built-in.
Traefik Hub MCP Gateway — gateway de Traefik.
MintMCP — gateway con foco en observabilidad.
OpenObserve MCP Gateway guide.
What is an MCP Gateway (DEV Community).
OpenTelemetry MCP Server (Traceloop) — el patrón inverso: usar MCP para que agentes consulten traces OTel.

Cross-references:

Post anterior: Guardrails y safety.
AgentSight y el nuevo tracing de LLMs — donde se introdujo stdiocap para capturar stdio de servidores MCP locales.
Evals: la capa después del tracing.

AgentSight and the new LLM tracing: zero-instrumentation with eBPF against Langfuse, LangSmith, Phoenix and company

Tue, 19 May 2026 18:00:00 +0200

TL;DR

Observing an LLM agent in production in 2026 splits into two approaches with opposite philosophies. The instrumented one, dominant until 2025, lives in tools such as Langfuse, LangSmith, Arize Phoenix, Helicone, OpenLLMetry/Traceloop or Pydantic Logfire: you install an SDK, decorate your calls, emit spans with the OpenTelemetry GenAI convention (gen_ai.request.model, gen_ai.usage.input_tokens, and so on) and export them to a backend. Enormous depth when you control the code; zero visibility when the agent is an opaque binary you run without instrumenting. The zero-instrumentation one, which AgentSight popularised in the second half of 2025, turns the perspective 180º: it puts eBPF hooks on the uprobes of the SSL/TLS libraries and captures the plaintext of every HTTPS request before encryption, without touching the app’s code, with less than 3% overhead and the guarantee of being tamper-proof (the agent cannot falsify what is seen in the kernel). Combined with BPF capture of stdio for local MCP servers, AgentSight gives you complete observability of any agent, including closed binaries such as Claude Code, Gemini CLI or Cursor, on a Kubernetes cluster. The two families are not enemies: the 2026 reference stack combines both (instrumented for your own apps with LangChain, eBPF for opaque binaries and tamper-proof compliance) over OpenTelemetry GenAI semantic conventions as the common vocabulary the ecosystem is stabilising this year.

This is the fourth and last post of the eBPF series. Part 1: eBPF from zero to Cilium. Part 2: Tetragon: runtime security. Part 3: Hubble: network observability. Here we close the circle with the semantic dimension: what an AI agent does, not only what network it opens or what syscalls it emits.

The analogy: traditional APM vs network sniffer

Anyone who has operated enterprise applications knows the two tribes of monitoring. The APM tribe (New Relic, AppDynamics, Datadog APM): you install an agent or an SDK in each application, mark spans, collect traces with enormous depth inside each process, down to lines of code, SQL queries, Java methods. The wire-level tribe (network sniffers, SolarWinds NPM-style tools, NetFlow): it does not touch the application; it observes the cable, sees protocols, latencies, retransmissions, identifies problems the app does not know it has.

Each one sees different things and both are useful. Anyone who has lived through a serious incident where APM said “all green” while users suffered knows that wire-level would have caught the problem (a saturated middlebox, a badly configured MTU, a TCP timeout). Anyone who has tried to debug a memory leak with sniffers knows it was impossible without APM.

LLM agent observability in 2026 is exactly at this point. The APM-style has been in place for a couple of years: Langfuse, LangSmith, Phoenix, OpenLLMetry. Enormous depth, requires instrumenting the app. The wire-level with eBPF has just arrived: AgentSight is the first project to take it to production. Less depth inside the agent, but it sees any agent without touching anything and it is tamper-proof. Both are useful. The industry is in full coexistence.

Why observing LLM agents is different

Before getting into tools, it is worth pausing on what makes LLM agents specific as subjects of observability:

Non-determinism. The same input can produce different outputs. Reproducing an incident requires capturing exactly the conversation, the model, the parameters and, ideally, the seed. An aggregate “p95 latency” metric falls short; what you need is replay of the individual trace.

A chain of external invocations. A typical agent calls LLM → tools (tool calling) → MCP servers → other APIs → back to the LLM. A chat session can generate dozens of chained calls that have to be correlated by trace_id to understand the decision.

Cost linear in tokens. Every call is paid for in tokens. Without tracing input/output tokens per request, you cannot assign cost to a tenant or a team, nor detect loops that eat your budget in an hour.

Semantic risk. Prompt injection (a user input containing instructions to manipulate the model), jailbreaks, secret leakage through tool calls. It is a class of problem that does not appear in traditional applications, and observability must see it.

Opaque binaries. In 2026, many teams deploy third-party agents such as Claude Code, the Cursor agent, Aider, Gemini CLI or Codex CLI as internal tools. They are not your own applications; they are closed binaries that call the vendor’s API. Instrumenting them is impossible. Observing them requires something else.

Multi-agent and orchestration. More and more architectures have agents that invoke other agents (planner → executor → critic). Observability must understand the topology, not just the individual span.

With these five points in mind, the tools we are about to see differ mainly in which parts of the problem they cover well and which parts they leave blind.

The instrumented approach: how it works

The model is direct and familiar:

Your code calls the LLM or the tools using an official library: openai, anthropic, langchain, llama_index, dspy.
You install a tracer SDK (Langfuse, LangSmith, OpenLLMetry, Logfire) that wraps or monkey-patches those libraries.
Each call emits an OpenTelemetry span with standardised attributes: model used, input/output tokens, latency, parameters, messages, tool invoked, result.
The spans are exported over OTLP to a backend that shows them as a tree of traces.

# Typical example with OpenLLMetry + any SDK
from traceloop.sdk import Traceloop
from openai import OpenAI

Traceloop.init(app_name="my-agent", api_endpoint="https://otel-collector:4318")

client = OpenAI()
# this call automatically emits a span with
# gen_ai.request.model, gen_ai.usage.input_tokens, etc.
resp = client.chat.completions.create(
 model="gpt-4.1",
 messages=[{"role": "user", "content": "..."}]
)

What you see afterwards: a dashboard with each conversation as a trace, each call as a span, the complete prompts and completions (if you opt in), the computed cost, latencies per span, errors flagged.

OpenTelemetry GenAI semantic conventions: the common vocabulary

The fragmentation of the field is being mitigated with OpenTelemetry GenAI Semantic Conventions. It is the CNCF’s effort to get all tools to emit spans with the same attribute names:

gen_ai.system — the provider (openai, anthropic, vertex_ai, and so on).
gen_ai.request.model — the model requested (gpt-4.1, claude-3-5-sonnet).
gen_ai.response.model — the model actually used (sometimes it differs, e.g. fallbacks).
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — counters.
gen_ai.request.temperature, gen_ai.request.top_p, and so on — parameters.
gen_ai.response.finish_reasons — why it ended (stop, length, content_filter).
gen_ai.operation.name — the type of operation (chat, embedding, completion).

In early 2026, the client spans moved from experimental to stable. The rest (server spans, multi-agent events) is still in development. The operational meaning: if your SDK emits these attributes, any backend that understands OTel GenAI can consume them. Moving from Langfuse to Phoenix to Helicone does not imply re-instrumenting, only changing the exporter.

The SIG is actively developing conventions for multi-agent systems: agent teams, tasks, actions, memory, artifact tracking. This is what is missing for complex agent architectures to have a common vocabulary. In 2026 it is experimental; stabilisation is expected at the end of the year or the start of 2027.

Instrumented tools: the 2026 landscape

Tool	Licence	Self-host	Focus	Where it shines
Langfuse	MIT	Yes	LLM observability + evals + prompt mgmt	Best OSS balance, complete suite
LangSmith	Commercial	No	Native LangChain/LangGraph	If you use LangChain, zero-config integration
Arize Phoenix	ELv2 (OSS)	Yes	OTel-native, strong RAG	Vector DBs, retrieval, embeddings
Helicone	Commercial + OSS lite	Yes (lite)	Simple proxy	Setup in minutes, OpenAI-only
OpenLLMetry / Traceloop	Apache 2.0	Yes	OTel SDK for LLMs	Vendor-neutral, exports to any OTel backend
Pydantic Logfire	Commercial	No	App + LLM unified	If you use Pydantic AI, native integration
Weights & Biases Weave	Commercial	Limited	Experimentation + production	If you already use W&B for training
Laminar / Braintrust	Commercial	No / Yes	Evals + tracing	More recent, focused on evaluation

Deep dive: Langfuse

It is worth pausing on Langfuse because in 2026 it is the default choice among the open-source options and the one most teams have adopted this year. It is a YC W23 project, MIT licence, and keeps a sustained release cadence with serious architectural changes between versions.

Four declared pillars: observability (tracing), evaluations, prompt management, playground/datasets. Each one separately has complete commercial products behind it; Langfuse integrates them into a single platform with a single backend.

SDK v4: OTEL-native, not a replacement

The big recent operational change is the SDK v4, a thin layer over the official OpenTelemetry client. The choice is deliberate: instead of maintaining a client of its own that would fall behind the OTel primitives, Langfuse uses the standard SDK and enriches the spans with LLM-specific attributes and helpers. The consequence: any code already instrumented with vanilla OpenTelemetry (@opentelemetry/sdk-node, opentelemetry-sdk in Python) can export to Langfuse without major changes, and the other way round, if tomorrow you want to migrate from Langfuse to another OTel backend, the spans are portable.

In Python the idiomatic decorator is @observe:

from langfuse import observe, get_client

langfuse = get_client()

@observe()
def buscar_documentos(query: str):
 # any internal call is traced as well
 return vector_store.similarity_search(query)

@observe(as_type="generation")
def llamar_llm(prompt: str):
 # marked as "generation" so it appears with LLM metadata
 return openai_client.chat.completions.create(...)

@observe()
def pipeline_rag(pregunta: str):
 docs = buscar_documentos(pregunta)
 return llamar_llm(build_prompt(pregunta, docs))

The call tree is captured automatically: the trace shows pipeline_rag as the root span, with buscar_documentos and llamar_llm as children, nested. Without writing a single with tracer.start_as_current_span(...) by hand.

In TypeScript the equivalent is modular: you install @langfuse/tracing, @langfuse/otel and @opentelemetry/sdk-node, and you can use TS decorators, context managers or manual spans, the three models interoperate. The consequence: third-party libraries that emit OTel spans (openai, @anthropic-ai/sdk, Vercel AI SDK instrumentations) show up in Langfuse with no extra work.

Self-host architecture: designed for serious production

The Langfuse backend architecture has two explicit decisions that set its self-host deployment apart:

Persistence in S3/Blob Storage first. When a tracing event arrives, it is persisted in object storage before touching the database. Only when the later processing confirms OK is it inserted into Postgres/Clickhouse. If the DB goes down temporarily, the events are not lost; they stay in S3 waiting to be reprocessed. For production where losing the traces of an incident is equivalent to losing evidence, this is load-bearing.
Long migrations as background jobs. Schema upgrades that on other platforms mean a downtime window run in the background on Langfuse while the application keeps serving. Upgrade downtime drops drastically.

The officially supported deployment modes:

Docker Compose: for development and POCs. One command, everything up.
VM: a single node, containers, no orchestration. For small environments.
Kubernetes with Helm: the recommended mode for production. Official chart maintained. Supports external Postgres, external Clickhouse, external S3, HPA.

The external dependencies in a typical production setup: Postgres (metadata, prompts, configuration), Clickhouse (tracing events, high-cardinality queries), S3 or a compatible blob store (pending events), Redis (the queue between components). Yes, that is several pieces; it is what holds up the durability and the scale.

Prompt management as a first-class citizen

What sets Langfuse apart from platforms focused only on tracing is that the prompts live in Langfuse, not in the application repo or in spreadsheets. Each prompt has:

Name and version (v1, v2, v3…). Changing the prompt does not require redeploying the app: the app asks the SDK for the prompt, and the SDK caches it and refreshes when there is a new version.
Typed variables: {{user_input}}, {{context}}. Rendering with validation.
Tags and labels: by environment (production, staging), by team, by experiment.
Client and server cache: the SDK caches locally with a configurable TTL, avoiding a roundtrip to Langfuse on every call.
Linkage with traces: each trace records which exact version of which prompt was used. Investigating “this answer came out wrong” leads to prompt version Y, not to “some version of the prompt at some point”.

from langfuse import get_client

langfuse = get_client()

prompt = langfuse.get_prompt("rag-system-prompt", version=3)
# or by label: langfuse.get_prompt("rag-system-prompt", label="production")

compiled = prompt.compile(context=docs_text, user_input=question)
# 'compiled' is the final string, ready to send to the LLM

For teams iterating on prompts daily, this is what avoids the chaos of “which version of the prompt is actually running in production right now”.

Evaluations: four combinable evaluation models

Langfuse covers the four patterns of answer evaluation:

LLM-as-a-judge: you configure a model (typically GPT-4 or Claude) with a rubric and it evaluates each answer. Result: a numeric score (0-1) and a justification. Applicable to automatic tracing (all answers) or batch (a dataset selection).
User feedback: the app lets the user mark an answer as good/bad. The feedback is associated with the trace and the prompt version, which lets you see which versions have a worse rate.
Manual labeling: a UI where human labelers score answers. Useful for golden datasets and for evaluating the judge.
Custom evaluators through the API/SDK: your own evals (a unit test, a business metric) report a score through the API. They integrate with CI.

Combined, they give regression testing of the prompt: you change from v3 to v4, evaluate the golden dataset with LLM-as-judge, compare; if v4 gets worse in any of the segments, the merge fails.

Integrations

Langfuse does not compete with OpenLLMetry, LangChain or LiteLLM: it integrates them. The ones that are tested and documented:

OpenTelemetry: any OTel instrumentation emits to Langfuse over OTLP.
LangChain and LangGraph: a native callback that captures the whole chain.
LlamaIndex: native callback.
OpenAI SDK (Python and TS): a wrapper that adds tracing automatically.
LiteLLM: integration as a callback, which covers 100+ providers through LiteLLM.
OpenLLMetry / Traceloop: they emit to Langfuse like any OTel backend.
MLflow: through an OTel exporter from MLflow to Langfuse.
Vercel AI SDK: native instrumentation.

The strategy is clear: Langfuse is a backend, not an SDK. Your team chooses how it instruments; Langfuse accepts any path. The operational consequence: moving from Langfuse to another OTel backend tomorrow is viable.

When Langfuse is not the answer

So as not to present it as a silver bullet:

If you only use LangChain and have no resources to self-host: LangSmith will give you a smoother integration (it is the same team).
If your only need is a proxy with cost tracking and no evals: Helicone is simpler.
If you want an integrated commercial vendor solution: Datadog LLM Observability, New Relic AI Monitoring or Dynatrace AI are Enterprise alternatives with 24/7 support.
If your load is pure batch massive inference with no agents: you probably do not need semantic tracing; Prometheus + Grafana with OTel metrics is enough.

For everything else, your own apps with serious tracing, multi-tenant with quotas, teams iterating on prompts daily, RAG with continuous evaluation, Langfuse is the safe bet.

Quick choice summary:

LangChain → LangSmith (zero effort, automatic instrumentation).
Your own multi-framework applications with OSS → Langfuse (MIT, self-host, complete).
RAG with vector stores → Arize Phoenix (better retrieval visibility).
Simple proxy, low budget → Helicone.
Strict vendor neutrality → OpenLLMetry/Traceloop.
Pydantic AI → Logfire (same team).

Strengths and weaknesses of the instrumented model

Strengths:

Enormous depth: nested spans with all the context (chain steps, retrieval, embeddings, tool calls).
Semantic vocabulary: the SDK knows the domain (LLM, vector store, agent).
Maturity: three years of evolution, a rich ecosystem, ready-made dashboards.
Built-in evals: the top platforms combine tracing with evaluation (judge LLM, datasets, regression).

Weaknesses:

Requires control of the code: if you cannot instrument, it does not work.
Trust in the app: if the app reports badly or has a bug, so does the trace. It is not tamper-proof.
Coupling to the SDK: version changes in a library can break the instrumentation.
Uneven coverage: Python SDKs are mature; Go, Rust and JS are younger.

The zero-instrumentation approach: AgentSight

AgentSight is the project from the eunomia-bpf group that champions the opposite approach. Its paper on arxiv (2508.02736), presented at the Workshop on Practical Adoption Challenges of ML for Systems, formalises the proposal. The premise is direct:

Instead of instrumenting the agent, observe it at the system boundary.

And “system boundary” means the kernel boundary: the last point before a piece of data leaves the process towards the network or the filesystem. There, with eBPF, things are seen as they are, without the application being able to cooperate in hiding them.

Architecture: three planes

AgentSight assembles three layers:

Plane 1 — SSL/TLS uprobes. eBPF can attach programs to functions in userspace libraries (uprobes). The target functions are the encryption ones: SSL_write, SSL_read from OpenSSL/BoringSSL, and their equivalents in Rustls. AgentSight puts hooks on them that capture the arguments: the plaintext buffer the app passes to be encrypted, just before TLS processes it. On reception it does the symmetric thing: a hook after SSL_read with the freshly decrypted plaintext. Result: AgentSight sees the complete content of any HTTPS request the app makes without needing a man-in-the-middle, certificates or traffic decryption. The payload is plaintext because it was captured before being encrypted.

This works because uprobes are cheap (around 100 ns per invocation) and because apps use common TLS libraries. The few apps that implement their own TLS (rare in production) escape this hook; for those you need a different kprobe or manual instrumentation.

Plane 2 — Kernel events. In parallel, AgentSight observes relevant syscalls through tracepoints: execve (which processes start), connect/accept (network), read/write with file descriptors (filesystem and stdio), unlink, clone. Any action by the agent that has an effect outside the process passes through here. This covers, among other things, shell commands executed by the agent: if a Claude Code agent decides to run rm -rf to “clean up the project”, the execve is seen even if the LLM API does not report it.

Plane 3 — Correlation engine. The two previous planes produce streams of asynchronous events. AgentSight has a userspace component that correlates them causally cross-process: an outgoing HTTP request with bash -c rm -rf can be correlated with the previous LLM response that suggested it, through PIDs, timings and heuristics. The paper mentions the optional use of a secondary LLM (Anthropic Claude, for instance) that analyses the sequence of events and produces semantic alerts: “the agent responded with a tool call that was not in the whitelist”, “the reasoning chain has gone 47 iterations without converging”.

`stdiocap`: capturing the stdio of local MCP servers

One specific piece that deserves its own mention is stdiocap, a separate BPF tool included in the repo. The Model Context Protocol (MCP), popularised by Anthropic in 2024 and mainstream in 2025-2026, has two transport modes: HTTP/SSE (network) and stdio (between the client and the server it starts as a subprocess). Local MCP servers, the ones that run on the same machine and are started by the client as children over pipes, communicate through stdin/stdout/stderr with JSON-RPC.

stdiocap hooks read/write/dup on the stdin/stdout/stderr file descriptors of a target process and records all the JSON-RPC traffic between the MCP client and server. It is the same idea as the SSL capture but for stdio: you observe the conversation without either the client or the server knowing. Typical use case: seeing which tools of the filesystem-mcp MCP server a Claude Code agent has invoked in the last hour, what arguments it passed, what errors it received. Impossible with classic instrumentation (MCP servers are usually third-party binaries).

Guarantees: tamper-proof, kernel safety, <3% overhead

Three properties make AgentSight interesting for production:

Tamper-proof: the observation happens in the kernel (uprobes, syscalls). A malicious or compromised application cannot falsify what is seen. Compare with instrumentation: if the agent decides not to emit the span for its action, it does not appear in Langfuse. Here it has no choice.
Kernel safety: eBPF formally verifies that programs terminate and respect bounds checks. It cannot crash the kernel. The same as in the rest of the eBPF series.
<3% CPU overhead measured on real agent workloads (per the paper). The number compares favourably with SDK instrumentation, which typically adds 5-10% in intensive applications.

What it detects out of the box

The paper and the documentation highlight three classes of detection:

Real-time prompt injection: the correlation engine can apply rules or a detection model over the plaintext captured by the SSL uprobes. If the prompt contains suspicious patterns, “ignore all previous instructions”, a system prompt embedded in a user input, instructions to exfiltrate data, it raises an alert.
Reasoning loops that burn resources: agents that enter infinite loops calling tools without progressing. Detectable because the causal chain does not converge to a “final answer” and the tokens pile up. The correlation engine flags them.
Bottlenecks in multi-agent setups: when several agents coordinate, AgentSight sees the communication matrix between all of them and can detect agents that block while waiting, deadlocks, excessive fan-out.

The clash and the coexistence

The two families look like competitors, but in reality they see different things and complement each other in production.

What only the instrumented one sees

Internal agent variables that never reach the wire: the intermediate state of a LangChain chain, the values before passing them to a tool, how a prompt is built from a template with internal vars.
Deep semantic spans: retrieval > embed > vector_search > rerank > format_context > prompt_template > llm. AgentSight sees only the final call to the LLM; the path taken to build it is invisible.
Evaluations: scoring of answers, judge LLMs, quality regression. This lives only in instrumented platforms.

What only eBPF sees

Opaque binaries: Claude Code, Cursor, Gemini CLI, third-party agents. You do not have the code; you cannot instrument them. Only eBPF sees them.
System-level actions: the agent decides to run git push --force or kubectl delete. The action is seen in the execve. The agent’s instrumentation may not report it (especially if it was a command the agent generated as output without going through an explicit “tool”).
Tamper-proof audit: for regulatory compliance (HIPAA, SOC2, NIS2), having observation the app cannot dodge has formal value. eBPF provides it.
Local MCP servers over stdio: invisible to classic instrumentation unless each server emits its own spans (rare).

What both see, complementarily

Prompts and completions: the instrumented approach emits them with rich metadata; eBPF captures them off the wire. A perfect cross-check for spotting discrepancies.
Calls to external APIs: APM flags it; eBPF confirms it at kernel level.
Latency: APM per span; eBPF measures RTT at TCP level and network connectivity.

Decision matrix

Case	Instrumented	eBPF (AgentSight)
Your own app with LangChain	Yes, first	Optional
Your own multi-framework app	Yes	Optional
Third-party binary (Claude Code, Cursor)	Does not work	Yes, the only path
Tamper-proof regulatory compliance	Insufficient	Yes, required
Zero-trust multi-tenant	Insufficient	Yes, required
Local MCP servers (stdio)	Difficult	Yes, with stdiocap
Answer quality evaluation	Yes, required	No (out of scope)
Depth of the internal chain	Yes, required	No (a black box for AgentSight)
Reasoning loop detection	Possible with plumbing	Yes, built in
Real-time prompt injection	Possible (post-processed)	Yes, in stream

The natural conclusion: for your own apps, instrumented; for opaque binaries or compliance, eBPF; for everything that matters, both.

2026 reference architecture

Four recipes that cover the bulk of real cases:

Setup A — Your own application with LangChain or similar

Needs: depth, evals, a team comfortable with SDKs.

Langfuse self-host or LangSmith cloud as the backend.
OpenLLMetry SDK or LangSmith SDK instrumenting the code.
OpenTelemetry Collector between the app and the backend for routing flexibility (to Langfuse + Tempo + Loki, for instance).
Hubble for the network layer in the cluster (inter-pod latency, drop attribution).

Setup B — Productionising an opaque binary (Claude Code, Gemini CLI)

Needs: observe without touching, audit, control cost.

AgentSight deployed as a DaemonSet over the cluster (or standalone on the node).
Grafana with dashboards fed by AgentSight’s metrics.
An OTLP exporter from AgentSight to an OTel backend (Tempo, Jaeger). The spans will use the GenAI semantic conventions once they are fully standardised.
Tetragon optionally for policy over what the agent is allowed to run (Sigkill if it tries rm -rf or similar).

Setup C — Zero-trust multi-tenant platform

Needs: agents from different clients running on the same cluster, mandatory audit, nobody trusts anybody.

AgentSight as the tamper-proof audit layer. Compliance requires it.
Langfuse multi-tenant for the clients that do instrument.
Tetragon with TracingPolicyNamespaced per tenant (different policies per namespace).
Hubble with persistent flow logs for forensics.
Cilium NetworkPolicy to isolate tenants from each other on the network.

Setup D — A local MCP server on a workstation

Needs: seeing what an agent does with a stdio MCP server.

AgentSight stdiocap pointed at the PID of the client or the server.
Complete JSON-RPC capture to a file or to an OTLP endpoint.
Visualisation: Grafana, or simply jq over the log.

A real use case: if you are integrating an MCP server of your own and want to see what tool calls a Claude Code or Cursor agent makes to your server, stdiocap is the cleanest way. You need to modify neither client nor server.

Operational traps

Sensitive data in prompts (instrumented)

By default, Langfuse, LangSmith and similar tools capture the complete content of prompts and completions. If your app processes PII, secrets or medical data, that goes to your observability backend. Configuring redaction or content opt-out before going to production is mandatory. OTel GenAI has specific flags (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false) to avoid it.

Sensitive data in prompts (AgentSight)

The same problem, worse: AgentSight captures literally what goes to the wire, in plaintext. If the agent talked to api.openai.com with a prompt containing sensitive data, AgentSight has that plaintext. You have to encrypt or redact before storing.

Pinned certificates or non-standard TLS

Some high-security apps do certificate pinning or use unconventional TLS implementations (Go’s crypto/tls, custom BoringSSL). In those cases, the uprobes on libssl do not cover them. AgentSight detects when it cannot observe and reports the gap; you still have to add specific hooks for the alternative SDK.

Token volume and storage

An application with medium traffic can generate millions of tokens a day. If you store them all in Langfuse or Phoenix with long retention, the database grows fast. Strategies: aggressive sampling, short retention for normal sessions and long retention only for errors/anomalies, content redaction and keeping only metadata.

Tracing with sampling and consistency

To cut cost, many installations sample: only 1 in every N traces is persisted. Be careful with inconsistent sampling: a trace can carry several spans across multiple services, and if the sampling decision is taken per span, you end up with incomplete traces. OTel has head sampling (in the SDK, at the start), which is consistent, and tail sampling (in the collector, at the end), which allows finer rules. For LLM, tail sampling is ideal: sample everything, discard only the “normal” traces and keep the ones with errors, high latency or high cost.

Multi-agent and trace propagation

When agent A calls agent B, you have to propagate the trace context (W3C Trace Context headers) so it shows up as a single tree. If you do not, you see two disconnected traces. Modern platforms do it automatically with inject/extract, but if your transport between agents is custom (over Redis pub/sub, over a DB), you have to propagate by hand.

The cost of uprobes on critical libraries

Hooking libssl adds around 100 ns per invocation. On extreme TLS traffic loads (tens of thousands of connections/s per core), that adds up. AgentSight keeps it under 3% on typical agent workloads (which are chatty but not networking-intensive). If your use were sniffing all the HTTPS on the node, it could hurt more.

What we have not covered (next series)

Evals: the next layer after tracing. Phoenix, Langfuse, LangSmith and company offer answer evaluation (judge LLM, datasets, regression). It is a world of its own.
Guardrails and safety: NeMo Guardrails, Llama Guard, Llama Prompt Guard, evaluators specific to prompt injection and jailbreaks.
Deep MCP server observability: how the OpenTelemetry GenAI conventions are being extended to MCP servers for trace-aware tools.
eBPF + on-device inference: when the LLM runs locally through vLLM or llama.cpp, the uprobes can see the output token queue BEFORE it goes to the client. New territory.
Statistical analysis of agent flows: detecting drift, outliers, patterns that indicate degradation.

Closing the eBPF series

This series of four articles has travelled eBPF from first principles to the 2026 frontier:

eBPF from zero to Cilium — what eBPF is, networking hooks, how Cilium skips the TCP/IP stack, BGP Control Plane v2.
Tetragon: runtime security — observability and enforcement of processes in the kernel.
Hubble: network observability — L3-L7 flow logs and the frontier with AI agents.
This one — AgentSight, LLM tracing, instrumented vs zero-instrumentation.

If you have got this far you have the map to sit down with a platform, security or AI team in 2026 and recognise what each piece does, what problem it solves and where to start. That whole stack, Cilium for CNI and BGP, Tetragon for runtime security, Hubble for network observability, AgentSight for AI agents, sharing eBPF as a common substrate, Cloud Native governance and OpenTelemetry vocabulary. It is the clean architecture the industry asked for a decade ago and that finally exists.

References

AgentSight:

AgentSight GitHub (eunomia-bpf) — the project.
AgentSight: System-Level Observability for AI Agents Using eBPF (arxiv 2508.02736) — the formal paper.
AgentSight ACM workshop publication.
AgentSight blog post (eunomia.dev) — accessible description.

OpenTelemetry GenAI semantic conventions:

Instrumented platforms:

Langfuse — MIT, self-host + cloud.
LangSmith — LangChain team.
Arize Phoenix — OSS, OTel-native.
Helicone — simple proxy.
OpenLLMetry (Traceloop) — Apache 2.0, OTel SDK.
Pydantic Logfire — AI observability.

2026 comparisons:

Cross-references from the series:

eBPF from zero to Cilium.
Tetragon: runtime security.
Hubble: network observability.
LLM inference series: KV cache, vLLM on Kubernetes, PagedAttention, LLM Operators on K8s.

Opentelemetry on lo0 — Tech Blog

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

TL;DR

You are here: OBSERVE, day 2

The analogy: changing the track gauge without closing the line

The three-rung ladder

The three combinations that refuse to start

What breaks in each mode

The three steps, in order

The ClickHouse migrations backfill nothing

The five-step chain that does backfill history

The failure that does not block

Where the ten minutes come from

The forty-eight-hour TTL

What can and cannot be undone

How this looks in code

Migration checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

TL;DR

You are here: OBSERVE, day 2

The analogy: the file that gets photocopied whole at every signature

Which agentic platform is the most used

How LangGraph gets into Langfuse

The observation type is decided by the node name

What a turn costs, measured

What this means for ClickHouse

Interrupts: two traces where there should be one

Thread, session, user, and what is only read at the root

The filter that drops your own spans

The model and the cost with a gateway in the middle

How this looks in code

Agent instrumentation checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (1 of 10): the data model changed underneath, and what really goes into a trace

TL;DR

You are here: OBSERVE, day 2

The analogy: the archive that changed system

The data model, as it is in the code

The tables

The ten observation types

The limits, with their source

The two instrumentation routes

What gets lost over OTLP, and it is one specific thing

The precedence chain, which is the map you need to have

Metadata, and how it ends up in the table

Sessions, users and cardinality

Scores: five types and three sources

Masking, and what it does not cover

Volume control

What not to put into a trace

Version 3 and the calendar date

How this looks in code

Instrumentation checklist

Traps and things that are not what they look like

The series: the ten articles

See also

Sources

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

TL;DR

You are here: the seam between DEPLOY and OBSERVE

The analogy: the toll booth that issues the ticket

Three routes, and one with an expiry date

The configuration that works today

What is lost when moving to OTel v2

Cost: why the trace says zero

Registering the price

How that number reaches Langfuse, and who wins

The tokens of a streaming response

Correlation: from the client to the engine

What does work

What does not work without effort

The vLLM side, and why the Collector stops being optional

The four serial queues

Knob 7 — The `FINAL` modifier for OTel-only projects