Self-Hosting on lo0 — Tech Blog

Langfuse v4, day 2 (5 of 10): the worker's thirty-nine queues, and the three that decide whether you lose data

Mon, 14 Sep 2026 08:30:00 +0200

Fifth article in a series of ten about operating Langfuse v4 in production. The previous one covered migrating from version 3. This one goes inside the worker that is already running. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

Thirty-nine queues and thirty-two switches. The catalogue is in packages/shared/src/server/queues.ts:406. The QUEUE_CONSUMER_*_IS_ENABLED variables are read once at boot, so turning a queue off is a restart, not a hot change.

Sharding spreads nothing without Redis Cluster. The producer only applies the hash if REDIS_CLUSTER_ENABLED === "true", and that variable ships "false". Raising LANGFUSE_INGESTION_QUEUE_SHARD_COUNT on an installation with single-node Redis starts N workers and leaves N−1 with no work.

The dead-letter retrier ships disabled. QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED is the only one of the thirty-two whose default is "false".

And even switched on, it does not cover ingestion. Its list holds five queues and none is ingestion-queue. An event that exhausts its six attempts sits in the failed set.

The ClickHouse writer drops rows. Three attempts with a flat hundred-millisecond delay and then out. The code comment is explicit: // TODO - Add to a dead letter queue in Redis rather than dropping. The metric to watch is called langfuse.queue.clickhouse_writer.rows_dropped.

Turn a queue’s consumer off and you stop seeing its depth. The metrics emitter only polls queues with a worker registered in that container. The queue keeps growing and the dashboard goes flat.

Project deletion runs one job every ten minutes. By limiter, deliberately, because each one scours ClickHouse.

There are two optional health probes and one of them can cause the very restart loop it is meant to fix if configured with less than sixty seconds of initial delay.

You are here: OBSERVE, day 2

The previous article left the installation migrated. This is the map of the process that does the work, and it comes before the capacity article because the queues are where saturation shows up before ClickHouse notices.

The analogy: the sorting hub

A postal sorting hub has separate belts for parcels, registered mail, returns and internal pouches. Each belt has its speed and its staff, and some are throttled on purpose because the destination cannot take more.

The job of whoever runs it is not to watch the fastest belt, it is to know which one jams first, which can stop for an afternoon without anyone noticing and which cannot stop for ten minutes. And above all, where the bin for items nobody could sort is, because every hub has one and in many of them nobody looks at it.

The Langfuse worker has thirty-nine belts. This article is the plan, with the speeds and the location of the bin.

The map of the thirty-nine

Grouped by what they do, with what stops happening if the group stops:

Ingestion (4). ingestion-queue, secondary-ingestion-queue, otel-ingestion-queue, secondary-otel-ingestion-queue. If they stop, the web keeps accepting events and uploading them to object storage, and keeps enqueueing. Nothing reaches ClickHouse. The SDK sees no error at all. This is the group that cannot stop.

Propagation (1). event-propagation-queue, the dual-write one from the previous article, with a global concurrency of one across the cluster.

Evaluation (8). trace-upsert, dataset-run-item-upsert-queue, create-eval-queue, evaluation-execution-queue and its secondary, llm-as-a-judge-execution-queue, code-eval-execution-queue, experiment-create-queue. If they stop, evaluations are neither created nor executed. Traces keep arriving. This is the group that can wait.

Deletion and retention (7). trace-delete, score-delete, dataset-delete-queue, project-delete, data-retention-queue and its processing queue, batch-action-queue. If they stop, neither UI-requested deletions nor per-project retention run. That is not an operational annoyance, it is a breach if you have a retention commitment in writing.

Exports (3). batch-export-queue, core-data-s3-export-queue, metering-data-postgres-export-queue. Downloads a user asked for stay pending.

Outbound integrations (8). The PostHog, Mixpanel and blob storage pairs, plus webhook-queue, entity-change-queue and notification-queue. In each pair, the bare queue is the scheduler that fans out per project and the processing one does the work.

Cloud billing (4). They only register with a Stripe key. On a self-hosted installation they never start.

Maintenance (4). dead-letter-retry-queue, monitor-queue, in-app-agent-run-queue and v4-legacy-api-usage-queue, which ingests nothing: it is a cron that scans ClickHouse’s system.query_log to find which projects still use the old API.

The switches number thirty-two, all QUEUE_CONSUMER_*_IS_ENABLED and all "true" except the dead-letter retrier. Four queues have no switch of their own: the model-as-judge one rides inside the evaluation switch, the two Cloud data export queues have their own variable, and the processing queues share a switch with their scheduler.

The mechanism is a top-level if around each worker registration. A false switch means the worker is never instantiated, not that the queue stops receiving work: producers keep enqueueing regardless. Turning a queue off is not consuming it, not not filling it.

The sharding that does not shard

Nine queues support sharding: the four ingestion ones, three evaluation ones, model-as-judge and trace-upsert. Each has its own shard-count variable, and they all default to 1.

Raising them looks like the obvious first move to scale ingestion. It is not, because of this (packages/shared/src/server/redis/ingestionQueue.ts:48):

 const shardIndex =
 IngestionQueue.getShardIndexFromShardName(shardName) ??
 (env.REDIS_CLUSTER_ENABLED === "true" && shardingKey
 ? getShardIndex(shardingKey, env.LANGFUSE_INGESTION_QUEUE_SHARD_COUNT)
 : 0);

The : 0 at the end is the whole story. The producer only computes a shard if Redis is in cluster mode, and REDIS_CLUSTER_ENABLED ships "false". On an installation with single-node Redis, which is most self-hosted installations, raising the shard count to four starts four workers, three of which stare at empty queues, while all traffic keeps coming through ingestion-queue.

The sharding key, when it does apply, is projectId-eventBodyId run through SHA-256, taking the first eight hex characters and a modulo. Shard names are ingestion-queue for zero and ingestion-queue-N for the rest.

Without Redis Cluster, the lever that does work is concurrency, which is a different thing.

Concurrencies, and the ones set to one on purpose

The defaults, worth knowing because several sit at one for good reasons:

Queue	Variable	Default
`ingestion-queue` (per shard)	`LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	20
`secondary-ingestion-queue`	`LANGFUSE_INGESTION_SECONDARY_QUEUE_PROCESSING_CONCURRENCY`	5
`otel-ingestion-queue`	`LANGFUSE_OTEL_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	5
`trace-upsert`	`LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY`	25
`monitor-queue`	`LANGFUSE_MONITOR_QUEUE_PROCESSING_CONCURRENCY`	10
evaluation and model-as-judge	several	5
`create-eval-queue`	`LANGFUSE_EVAL_CREATOR_WORKER_CONCURRENCY`	2
`trace-delete`, `score-delete`, `dataset-delete-queue`, `project-delete`	several	1
`event-propagation-queue`	fixed in code	1 cluster-wide

Deletions sit at one and carry a rate limiter on top. The project one is the most aggressive (worker/src/app.ts:234):

 limiter: {
 // Process at most `max` delete jobs per LANGFUSE_CLICKHOUSE_PROJECT_DELETION_CONCURRENCY_DURATION_MS (default 10 min)
 max: env.LANGFUSE_PROJECT_DELETE_CONCURRENCY,

One project deletion every ten minutes. If a customer asks you to delete forty projects, that is six hours and forty minutes on the clock, and rightly so, because each of those deletions is a broom sweeping through ClickHouse.

The ingestion queues, by contrast, have no limiter at all. There the brake is concurrency and the ClickHouse writer.

One BullMQ detail that recurs across queues and explains a good number of other people’s stalls (worker/src/app.ts:261):

 // The default lockDuration is 30s and the lockRenewTime 1/2 of that.
 // We set it to 60s to reduce the number of lock renewals and also be less sensitive to high CPU wait times.
 lockDuration: 60000, // 60 seconds
 stalledInterval: 120000, // 120 seconds
 maxStalledCount: 3,

An event’s path, from API to ClickHouse

It is worth following it end to end once, because there are three places it can be lost.

The web groups the batch’s events by body id, uploads one JSON file per group to object storage and aborts if that upload fails: no blob, no enqueue. Then it enqueues one job per body id on ingestion-queue.

The worker picks up the job, optionally checks a recently-processed cache in Redis (off by default, five-minute lifetime), decides whether to redirect the project to the secondary queue, and downloads the file from storage. There is a comment here worth a capacity chart on its own:

Process files in batches. If a user has 5k events, this will likely take 100 seconds.

Then it merges and writes. And writing is not writing: the service does not talk to ClickHouse, it puts the row in a buffer whose defaults are 1,000 rows, 1,000 milliseconds and 3 attempts.

That buffer is the third place things get lost, and the most serious one in the system (worker/src/services/ClickhouseWriter/index.ts:535):

 // Re-add the records to the queue with incremented attempts
 let droppedCount = 0;
 queueItems.forEach((item) => {
 if (item.attempts < this.maxAttempts) {
 entityQueue.push({ ...item, attempts: item.attempts + 1 });
 } else {
 // TODO - Add to a dead letter queue in Redis rather than dropping
 recordIncrement("langfuse.queue.clickhouse_writer.error");
 droppedCount++;
 }
 });

Once the three attempts are spent, the row is dropped. And no queue retry saves it, because the BullMQ job already completed successfully: as far as the queue is concerned, that event was processed. The internal retries are not properly exponential either: the delay is flat at a hundred milliseconds, so all three attempts happen within three hundred milliseconds and a ClickHouse that is unwell for half a second takes the batch with it.

The runbook consequence is direct. langfuse.queue.clickhouse_writer.rows_dropped has to be on a dashboard and it has to have an alert at zero. Any non-zero value is silent data loss.

Retries, and the bin nobody looks at

Each queue defines its own policy. The ones that matter:

Queue	Attempts	Backoff
`ingestion-queue`	6	exponential, 5 s
`secondary-ingestion-queue`	5	exponential, 5 s
`evaluation-execution-queue`	10	exponential, 1 s
`code-eval-execution-queue`	3	exponential, 30 s
`trace-upsert`	2	exponential, 5 s, with a 30 s initial delay
`trace-delete`, `score-delete`	2	exponential, 30 s
`monitor-queue`, `in-app-agent-run-queue`	1	no backoff, deliberately

The two single-attempt ones are a design decision, and the comment explains it well: “Postgres owns correctness (claim CAS + reconcile-on-read); BullMQ is delivery-only, so never redeliver on its own.”

And now the bin. There is no dedicated dead letter queue. What exists is BullMQ’s native failed set per queue, with its own retention, plus a queue that retries from there. That queue has two problems for whoever operates it:

First, it ships disabled. It is the only one of the thirty-two defaulting to "false".

Second, even switched on, its scope is hand-written and has five entries (worker/src/services/dlq/dlqRetryService.ts:9):

 private static retryQueues = [
 QueueName.ProjectDelete,
 QueueName.TraceDelete,
 QueueName.ScoreDelete,
 QueueName.BatchActionQueue,
 QueueName.DataRetentionProcessingQueue,
 ] as const;

ingestion-queue is not there. Ingestion events that exhaust their six attempts sit in the failed set, up to a hundred thousand of them, and nobody ever retries them. To get them out there is an admin route, POST /api/admin/bullmq with action retry, which the file itself describes as the one the managed service uses for exactly this.

The metrics, with their exact names

Each queue’s metric name is built at runtime from the queue name, turning dashes into underscores and stripping the _queue suffix. So ingestion-queue produces langfuse.queue.ingestion, and secondary-ingestion-queue produces langfuse.queue.secondary_ingestion. Sharded queues always add a shard tag.

What comes out of that:

langfuse.queue.<queue>.rate with type in request, completed, failed, error and stalled.
langfuse.queue.<queue>.time_distribution with type in wait and processing.
langfuse.queue.<queue>.depth with type in waiting, failed and active, plus an aggregate series with shard: "all".
langfuse.queue.<queue>.dlq_oldest_age, the age of the oldest failed job, which emits zero when the set empties so the dashboard resets.
From the writer: langfuse.queue.clickhouse_writer.rows_dropped, .error, .processing_time and ingestion_clickhouse_insert_queue_length.

Two things to know about those metrics. The waiting depth includes paused jobs, to match BullMQ’s counter. And the big trap (worker/src/features/queue-metrics-runner/index.ts:70):

 // Only poll queues that have registered workers. This avoids calling
 // getInstance() on queues this worker doesn't consume, which would
 // create unnecessary Redis connections and can trigger side effects

Only queues that container consumes emit metrics. If you split roles and turn a queue’s consumer off across every container, that queue stops appearing on dashboards while it keeps growing in Redis. It is the perfect way not to find out.

The metric that saturates first is still the same one: langfuse.queue.ingestion.depth with type: "waiting". The emission interval defaults to one second.

The two probes, and the one that can restart you in a loop

The worker exposes /api/health and /api/ready, identical but for one thing: the second returns 500 after SIGTERM, which is how draining works. Both always check Postgres with a SELECT 1 and Redis with a ping capped at two seconds, hardcoded.

Then there are two optional checks enabled by query parameter:

?failIfEventPropagationStuck=true looks at the propagator’s heartbeat and returns 503 if it has not been refreshed for more than 35 minutes. A missing heartbeat does not count as stuck, deliberately, so a freshly booted container is not restarted in a loop. And the condition that must be respected:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

?failIfQueueConsumptionStuck=true is an in-memory signal of that container, with no Redis involved: it is marked on the active and completed events of any worker. It returns 503 if that container has gone 60 minutes without picking up or finishing a single job. It exists for a very specific failure mode:

After Redis lock loss BullMQ workers can wedge permanently: the process stays alive and connectivity checks pass, but no queue picks up jobs ever again.

The one-hour threshold leans on the default crons keeping a healthy worker busy at least once an hour. In multi-replica deployments each scheduler tick lands on a single replica, so the comment itself warns about raising the threshold if a replica can legitimately sit idle.

Draining, by the way, has no cap. The order is close the HTTP server, stop the periodic runners, close the BullMQ workers and then flush the ClickHouse writer, which on shutdown empties the whole queue rather than one batch. The only thing bounding that drain is the Kubernetes grace period, so it had better be generous.

The crons

Queue	Pattern	What it does
`event-propagation-queue`	`* * * * *`	propagates one staging partition
`dead-letter-retry-queue`	`0 /10 * * *`	retries failed jobs from five queues
`v4-legacy-api-usage-queue`	`/15 * * *`	scans `system.query_log` for old-API usage
`blobstorage-integration-queue`	`/20 * * *`	fans exports out per project
`cloud-usage-metering-queue`	`5 * * * *`	Cloud billing
`posthog-integration-queue`, `mixpanel-integration-queue`	`30 * * * *`	hourly fan-out
`metering-data-postgres-export-queue`	`30 2 * * *`	daily export
`core-data-s3-export-queue`, `data-retention-queue`	`15 3 * * *`	daily export and retention

And a warning the scheduler file itself writes in capitals: cron registration is fire-and-forget from the queue constructors, and a failure is only logged. A transient Redis failure at boot can leave a cron unscheduled until the next restart, without the container ever being marked unhealthy.

Splitting roles, which is what all this is for

With thirty-two switches, what you can build is one worker per family. The split that makes sense on an installation with agentic load:

# Pool 1: ingestion. This is the one that scales with traffic.
QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_OTEL_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED=true # note: global concurrency 1
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=20
# everything else false

# Pool 2: evaluations. Scales with the number of evaluators, not with traffic.
QUEUE_CONSUMER_EVAL_EXECUTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_CREATE_EVAL_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_TRACE_UPSERT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EXPERIMENT_CREATE_QUEUE_IS_ENABLED=true

# Pool 3: maintenance. One replica is enough.
QUEUE_CONSUMER_TRACE_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_PROJECT_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DATA_RETENTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED=true # switch it on

Two warnings about this split. Each worker registration opens its own Redis connection, so the connection count grows with queues times shards times replicas, and that is checked against Redis maxclients before multiplying replicas. And the moment you split roles you lose the metrics of the queues that container does not consume, so some replica needs to consume every queue or you go blind on that one.

Checklist

There is an alert on langfuse.queue.clickhouse_writer.rows_dropped, threshold zero.
There is an alert on the depth of langfuse.queue.ingestion with type: "waiting".
There is an alert on dlq_oldest_age for the queues that matter, and somebody knows how to use POST /api/admin/bullmq.
The dead-letter retrier is on in some replica, knowing it does not cover ingestion.
Nobody raised the shard count without Redis Cluster.
The optional health probes are configured with initialDelaySeconds of at least 60 seconds.
The Kubernetes termination grace period allows the ClickHouse writer to flush.
Every queue is consumed by at least one replica, so its metrics are not lost.
Redis maxclients copes with queues times shards times replicas.

Traps

Sharding without Redis Cluster does not spread load. It starts the workers and leaves them idle.

The dead-letter retrier ships disabled and does not cover ingestion. Two independent facts, both uncomfortable.

The ClickHouse writer drops rows after three attempts. With a TODO in the code admitting it and three hundred milliseconds of total headroom.

Turning a consumer off also turns its metrics off. The queue keeps growing, the dashboard stays flat.

A cron can end up unscheduled while the container stays healthy. A transient Redis failure at boot, and it is gone until the next restart.

The PostHog and blob storage integrations disable themselves when an error is classified as a customer-config fault, and if the notification also fails they stay off and silent.

A misconfigured propagation probe restarts in a loop. Less than sixty seconds of initial delay and you get the cycle.

Draining has no cap of its own. The Kubernetes grace period bounds it and nothing else.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues (this article): the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/server/queues.ts, packages/shared/src/server/redis/, worker/src/env.ts, worker/src/app.ts, worker/src/queues/, worker/src/services/ClickhouseWriter/, worker/src/services/dlq/, worker/src/features/health/, worker/src/features/queue-metrics-runner/, worker/src/utils/shutdown.ts.
Langfuse self-hosting documentation, checked on 14 September 2026.
BullMQ: going to production, checked on 14 September 2026.

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

Mon, 14 Sep 2026 08:00:00 +0200

Fourth article in a series of ten about operating Langfuse v4 in production. The previous three looked at the client side: what goes into a trace, what an agent turn costs and two well-known agents instrumented. This is the first one on the server side. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

The default write mode is the destination, not the origin. LANGFUSE_MIGRATION_V4_WRITE_MODE ships as events_only (packages/shared/src/env.ts:337). That is right for a new installation and wrong for a migration: in that mode the traces and observations tables stop being written.

The worker refuses to start on three combinations. legacy with direct OTel writes, events_only without the preview opt-in, and events_only with OTel behaviour set to dual. The code comment names them: combinations that would silently lose data (worker/src/env.ts:722).

In events_only twenty-four REST routes return 404. Traces, observations, sessions, v1 and v2 scores, metrics, spans, generations and the dataset-run ones. Not a bug: a deliberate short circuit so stale data from tables nobody writes is never served.

History is backfilled by a chain of five background migrations, not by the ClickHouse migrations. Not one of the ten new ClickHouse migrations copies a single old row.

A failed background migration does not block the next one. The manager has no dependency model and skips them. The chain protects itself with predecessor guards that the first two steps do not declare, so a failure of step 1 does not stop step 2 from starting.

The old API’s ten-minute delay has an exact explanation. Three-minute partitions in the staging table, a lock four minutes after creation, and a per-minute cron that only touches partitions older than ten minutes, with a global concurrency of one.

That staging table has a forty-eight-hour TTL. If the propagator stops over a long weekend, those partitions disappear and the hole in events_full is permanent. No code detects it and none repairs it.

The point of no return is in Postgres, not in ClickHouse. A Prisma migration runs DROP TABLE on traces, observations and scores, and Prisma Migrate has no down migrations.

You are here: OBSERVE, day 2

The first three articles measured what an agent produces. This one starts on operating the backend that receives it, and it comes before the capacity article because you cannot size what has not been migrated yet.

The analogy: changing the track gauge without closing the line

Changing the gauge of a line in service is not done in one go. The third rail is laid, rolling stock of both gauges runs for a while, and only when every train is of the new gauge is the old rail lifted.

The Langfuse migration has exactly that shape. dual mode is the third rail: the old tables and the new ones are written at the same time. And as with the track, the danger is not the day of the change, it is at both ends. Laying the new rail and lifting the old one on the same day strands trains. And once the old rail is up, going back is not a matter of will, it is a matter of the material no longer being there.

The three-rung ladder

Everything turns on one variable with three values, defined three times over in the three packages that have to agree, with the same enum and the same default (packages/shared/src/env.ts:337, worker/src/env.ts:586, web/src/env.mjs:610):

LANGFUSE_MIGRATION_V4_WRITE_MODE: z
 .enum(["legacy", "dual", "events_only"])
 .default("events_only"),

The two helpers that read it say what each rung does better than any explanation (worker/src/env.ts:705):

export const v4WritesToEventsTable = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "legacy";

export const v4WritesToLegacyTables = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "events_only";

What gets written in each mode:

Table	`legacy`	`dual`	`events_only`
`traces`, `observations`	yes	yes	no
`scores`	yes	yes	yes
`dataset_run_items_rmt`	yes	yes	no
`observations_batch_staging`	no	yes	no
`events_full`	no	yes	yes
`events_core`	no	yes (through the materialised view)	yes

The middle rung is the only one where both coexist, and therefore the only place you can step back from without losing anything. The code itself offers it as the remedy when something breaks in events_only (web/src/pages/api/public/ingestion.ts:296):

As a temporary migration bridge, set LANGFUSE_MIGRATION_V4_WRITE_MODE=dual on both the web and worker services and redeploy.

The both in that sentence matters. All three packages read the variable and the code comment insists three separate times on keeping them in sync. Deploying the worker with one value and the web with another is the first way to break this.

The three combinations that refuse to start

The worker validates its configuration at boot and throws in three cases, under a comment that leaves no doubt (worker/src/env.ts:718):

 // Hard errors: combinations that would silently lose data.
 if (mode === "legacy" && otel === "direct") {
 throw new Error(
 "Invalid V4 config: LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct " +
 "requires LANGFUSE_MIGRATION_V4_WRITE_MODE in {dual, events_only}. " +
 "Direct OTel writes target events_full, which is not read in legacy mode.",
 );
 }

The second one bites in a real migration: events_only also requires LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true, because otherwise the web reads would target the legacy tables that mode no longer writes. The third forbids events_only with OTel behaviour set to dual.

A failed boot is the good news. A failed boot shows up in thirty seconds. The combinations it does not validate take days to surface.

What breaks in each mode

This is the table to have in front of you before moving the variable, because the two ends break different things.

In legacy the event tables are not written, so whatever depends on them disappears. Monitor routes return 404 through a dedicated middleware (web/src/server/api/trpc.ts:413). Metrics v2 is unavailable. Dashboard widgets stay at version 1 (DashboardService.ts:39). The v4 preview is forced off and cannot be switched on.

In events_only the legacy tables are not written, so twenty-four REST routes short-circuit with a 404 (web/src/features/public-api/server/createAuthedProjectAPIRoute.ts:319):

 // Short-circuit routes that read from legacy traces/observations tables
 // when the deployment is in events_only mode — those tables are no longer
 // populated, so the response would be stale or empty.

The list covers traces, traces/{id}, observations, observations/{id}, sessions, sessions/{id}, scores and scores/{id} in v1 and v2, metrics, metrics/daily, spans, generations, events, dataset-run-items and the two datasets/{name}/runs routes. If you have your own integrations against those routes, they break on cutover day, and returning 404 rather than stale data is the right call even though it stings.

One detail that is usually reported wrongly: /api/public/ingestion is never shut down. The deprecation notice says so itself (deprecations.ts:104):

This route is never removed: scores (and sdk-log) stay accepted.

In events_only that route stays alive and keeps accepting two event types (web/src/pages/api/public/ingestion.ts:274):

const EVENTS_ONLY_ALLOWED_TYPES = new Set<string>([
 eventTypes.SCORE_CREATE,
 eventTypes.SDK_LOG,
]);

The rest are rejected per event, not per batch: the batch comes back as a 207 with 400s inside. An old client sending traces and scores mixed together will see the scores go through and the rest fail, which is exactly the kind of partial failure that takes a long time to notice.

The three steps, in order

Given all of the above, the order of a no-window migration is this:

Finish the v3 background migrations before upgrading. This comes before everything else and there are no shortcuts. v4 deletes the rows of seven old migrations, and the comment explains why (prisma/migrations/20260723124010_drop_dataset_run_items_table/migration.sql:7): their worker scripts are deleted alongside, so an unfinished row would leave the manager with no script to resolve. If one was halfway, that data is lost and the row disappears quietly.
Upgrade to v4 with LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy, apply the ClickHouse and Prisma migrations. Here the installation behaves like v3 and you verify everything still stands.
Move to dual. From then on both table families are written and the propagator starts. Let it run for as long as it takes and launch the historic backfill.
Once history is complete and verified, move to events_only with the preview opt-in enabled, and only then lift the old rail.

All the work is between 3 and 4, and it can take weeks. There is no rush: dual is a stable state.

The ClickHouse migrations backfill nothing

There are 96 files in packages/shared/clickhouse/migrations/canonical/, 48 up and 48 down, which have to be materialised before running (pnpm ch:migrations:materialize) and cleaned afterwards, including after a failed migration. The runner is golang-migrate.

The ten v4 ones are 0039 through 0048. The first three create events_full, events_core and the materialised view that feeds the second from the first. The three drop migrations remove event_log, project_environments and dataset_run_items. The ones in between add columns and indexes.

What matters for planning is what they do not do. None copies an old row. The materialised view only sees rows inserted into events_full after it exists. The indexes in migrations 0043 and 0047 are added without materialising historical parts, and 0047 says so in writing:

Skip indexes are added without materializing historical parts to keep this migration metadata-only.

Translated into operations: during and after the migration, searches over older data do full scans. Not a bug, the price of a migration that does not block.

And one cluster trap: migrations 0042, 0043, 0047 and 0048 carry alter_sync = 2, meaning every ALTER waits for all replicas. One slow or down replica hangs the whole migration. On a single-replica installation you never notice; on three, you do.

The five-step chain that does backfill history

The backfill is done by five background migrations running in the worker, registered in Postgres with names that carry the order inside them:

20260701_v4_step_1_create_root_spans_from_traces: creates virtual root spans in events_full from existing traces, with the identifier concat('t-', t.id). It skips traces referenced by a dataset run item, which belong to step 4. Chunked by monthly partition.
20260701_v4_step_2_rewrite_observations_to_pid_tid_sorting: copies observations into a scratch table with a different key order. Chunked by monthly partition.
20260701_v4_step_3_backfill_events_full_from_observations: the bulk of it. Chunked by ClickHouse part, not by partition, and the comment explains why that is the only safe granularity:

One todo per part keeps each INSERT bounded to a single ClickHouse part (single-digit GBs in most cases), which is the only granularity that’s safe to assume self-hoster hardware can chew through without OOM or memory-limit failures.
20260701_v4_step_4_backfill_events_full_from_dataset_run_items: walks the dataset run items by cursor and writes the root and its descendants. Batches of 200, a cap of 50,000 descendants per item, a 90-day window.
20260701_v4_step_5_drop_pid_tid_sorting_tables: drops step 2’s scratch tables.

There are only two switches for the five. The first four share LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL, which ships true, and the fifth has its own, LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES, which ships false on purpose so the scratch tables stay around until the operator confirms the backfill went well.

One migration at a time across the whole cluster. The manager takes a lock in Postgres, refreshes it with a heartbeat every 15 seconds and treats a lock older than 60 seconds as expired, all inside a serialisable transaction. Adding worker replicas does not speed the backfill up by one minute. That is the first thing that surprises anyone used to scaling out to go faster.

They are resumable, and rather better than this usually is. Progress lives in a JSONB field on the row, and on resume the worker re-attaches to the queries the previous worker left running, by query id, instead of relaunching them:

In-flight queries keep running server-side; the next run re-attaches to them via the persisted queryIds.

On how long they take, the code says nothing. Not the files, not the folder’s README, not the migrations. The only time threshold anywhere is a design one: “A good threshold is something that takes more than 5 minutes to run”. Any figure in hours or days you read elsewhere, including the one I repeated myself before checking, is not in the code.

The failure that does not block

This is the finding that changes a runbook the most. When a background migration fails, the manager marks it with failedAt and moves on. And the predicate it uses to pick the next one excludes exactly the rows carrying failedAt, so a failed migration is skipped and does not block the ones after it.

The code acknowledges the gap and explains the patch (worker/src/backgroundMigrations/utils/backfillBase.ts:107):

The BackgroundMigrationManager has no dependency model — it runs every eligible row in name order and marks each finished/failed independently, so a failed upstream step does NOT by itself stop a downstream step from running on partial data. Each downstream step calls this from its validate() […] Because the next step in turn guards on its own predecessor, a single failure transitively halts the rest of the chain.

The guard exists, but only in steps 3 and 4. Steps 1 and 2 declare no predecessor. A failure of step 1 does not stop step 2 from starting; what saves the situation is that step 3 does check step 2.

Practical consequence: the only reliable signal that the backfill is going well is to look at the background_migrations table in Postgres and check that all five rows have finishedAt and none has failedAt. And to restart a failed one you clear failedAt by hand and relaunch with --retry-failed, as the error message itself says.

Two concrete failure modes that deserve a place in the runbook:

Step 3 breaks if ClickHouse merges a part midway. The message says it all: “Part … no longer active after processing — its rows are in a merged successor part that is not in this run’s todo list. Clear failedAt and re-run with state.chunksLoaded=false”. Recovery is manual, editing the JSONB.
Step 4 stops dead on one giant trace, if it exceeds 50,000 descendants. The three ways out the error offers are raising the cap, skipping the item (and losing that trace entirely) or fixing it by hand in SQL.

And an asymmetry worth knowing before comparing data: the historic backfill does not copy trace metadata down to child spans, deliberately, whereas the live path does concatenate them. History and new data are not identical.

Where the ten minutes come from

The v3 API warns that its data runs about ten minutes behind (deprecations.ts:16):

Data on this API is delayed by about 10 minutes. Real-time is only OpenTelemetry writes plus v2 observations and v2 metrics reads.

That delay is the sum of four things, all visible in the code:

The staging table partitions on three-minute intervals.
A partition is treated as closed four minutes after creation: “We ’lock’ partitions 4min after their creation, i.e. the 15:00:00 partition should stop receiving updates at 15:04:00.”
The propagation cron runs every minute and only looks at partitions older than LANGFUSE_EXPERIMENT_EVENT_PROPAGATION_PARTITION_DELAY_MINUTES, which defaults to 10.
That cron has a global concurrency of one across the cluster and processes a single partition per invocation, so a backlog makes the delay grow rather than recover.

The way out of the delay is not to lower that variable, it is to have clients write over OTLP with direct writes, which is the real-time route. The header that enables it is x-langfuse-ingestion-version: 4, sent automatically by the Python SDK from 4.0.0 and the JavaScript one from 5.0.0, and a value above 4 is rejected with a 400.

The forty-eight-hour TTL

This is the frightening one, and it appears in no migration guide.

The staging table carries TTL toDateTime(s3_first_seen_timestamp) + INTERVAL 48 HOUR with ttl_only_drop_parts = 1. If the propagator stops for more than forty-eight hours, a long bank holiday for instance, those partitions delete themselves. The data is already in the legacy tables, so an installation in dual keeps working, but the hole in events_full is permanent, and no code detects it and none repairs it. The health probe detects stuck, not lost.

The defence is an alert, and one exists for this. The worker health endpoint accepts ?failIfEventPropagationStuck=true and returns 503 if the propagator’s heartbeat has not been refreshed for more than LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES, which defaults to 35. The comment carries a condition to read before putting it in a Kubernetes probe:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

Another detail of the same mechanism: the propagation cursor lives in Redis, not in Postgres. If Redis is flushed, the cursor resets and every live partition of the staging table is reprocessed.

What can and cannot be undone

The ClickHouse down migrations exist, and they are honest about what they do. The one for 0039 warns in capitals:

WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this table may predate this migration […] Rolling back drops that data regardless of which path created the table.

The ones for the dropped tables recreate the schema and nothing else: “Any data held in event_log before the up migration dropped it is NOT restored.”

But the real point of no return is not in ClickHouse, it is in Postgres. There is a Prisma migration called 20260723150000_drop_legacy_tracing_tables that does exactly what it says:

DROP TABLE IF EXISTS "traces";
DROP TABLE IF EXISTS "observations";
DROP TABLE IF EXISTS "scores";

Prisma Migrate has no down migrations. It applies, and that is that.

Summarising the real reversibility: the write mode can go down from events_only to dual whenever you want, and that is the panic button. What does not come back is what was not written while you were up there: the legacy tables receive nothing retroactively. And step 5 of the chain, the one that drops the scratch tables, ships disabled precisely so you can do forensics if something went wrong.

One last note on the ground you are standing on. The comment in the export-source policy drops something worth reading twice: “legacy-writes-disabled, which in practice only surfaces on self-hosted (Cloud does not run events_only)”. The events_only path is the one you will use and not the one the vendor’s managed service uses. It is less battle-tested in production than dual.

How this looks in code

The minimum runbook, in variables:

# Step 2: upgrade to v4 behaving like v3
LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy
LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=dual_write # "direct" will not start here
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=false

# Step 3: third rail. Both services, web and worker, together.
LANGFUSE_MIGRATION_V4_WRITE_MODE=dual
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=true
LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES=35

# Step 4: only once all five background_migrations rows have finishedAt
LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only
LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true # without this it will not start
LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES=true # last of all

And the query that answers the only question that matters for weeks:

SELECT name, started_at, finished_at, failed_at, failed_reason,
 state->>'chunksLoaded' AS chunks
FROM background_migrations
WHERE name LIKE '20260701_v4_%'
ORDER BY name;

Migration checklist

Every v3 background migration is finished before the image is upgraded.
There is a Postgres and a ClickHouse backup from before the first DROP TABLE.
The write mode has the same value in the worker and in the web, and both change together.
The upgrade went to legacy first, and the service was verified unchanged.
There is an alert on ?failIfEventPropagationStuck=true, with initialDelaySeconds of at least 60 seconds.
Somebody looks at the background_migrations table every day while the backfill runs.
The in-house integrations calling the twenty-four routes that will return 404 have been inventoried.
Client SDKs are on Python 4.0.0 or JavaScript 5.0.0 at minimum before moving to events_only.
Step 5 of the chain is enabled last, and only after verifying the backfill.

Traps

The default is the end state. A v3 installation booting v4 without touching the variable lands in events_only and stops writing the old tables from the first second.

A failed background migration does not block the next one. And the first two steps of the chain declare no predecessor.

More worker replicas do not speed up the backfill. One migration at a time across the cluster, by design.

Forty-eight hours of stopped propagator is a permanent hole. No detection and no repair.

New indexes do not cover old data. Historical searches scan until somebody materialises the indexes by hand.

On a cluster, every ALTER waits for all replicas. One down replica hangs the migration.

Classic ingestion rejects per event. A mixed batch returns 207 and half of it is lost without anyone reading the response body.

History and the live path do not produce quite the same thing. Trace metadata does not descend to children in the backfill.

The route you will take is not the one the vendor takes. Cloud does not run events_only.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window (this article): the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/env.ts, worker/src/env.ts, web/src/env.mjs, web/src/features/public-api/server/deprecations.ts, web/src/pages/api/public/ingestion.ts, web/src/features/public-api/server/createAuthedProjectAPIRoute.ts, worker/src/backgroundMigrations/, worker/src/features/eventPropagation/, packages/shared/clickhouse/migrations/canonical/, packages/shared/prisma/migrations/.
Langfuse compatibility and migration, checked on 14 September 2026.
OpenTelemetry integration migration to v4, checked on 14 September 2026.

Langfuse v4, day 2 (3 of 10): two well-known agents instrumented, Open Deep Research and GPT Researcher

Mon, 14 Sep 2026 07:00:00 +0200

Third article in a series of ten about operating Langfuse v4 in production. The first walked through what goes into a trace; the second measured what an agent turn costs over a minimal graph. This one applies the same to two agents people deploy. Code read on 14 September 2026: Open Deep Research at commit 1b7d2e8 and GPT Researcher at 6f99857, with Langfuse 4.15.2 and LangGraph 1.2.11.

TL;DR

One Open Deep Research request with the default configuration is 1,416 observations and 8.1 MB. Measured over a faithful replica of its topology with five concurrent units, six supervisor iterations and ten react calls per researcher, which are its defaults. It all lands in a single trace. And it is a floor, because on the bench the search results are three hundred characters and real ones are much larger.

None of those 1,416 is of type AGENT. Its nine nodes are called clarify_with_user, write_research_brief, research_supervisor, final_report_generation, supervisor, supervisor_tools, researcher, researcher_tools and compress_research. None contains the string the handler looks for.

The parallel fan-out does not break the trace, and the fear that it does is unfounded. Open Deep Research forwards the config to its subgraphs and GPT Researcher passes a fresh one carrying only tags. I tested both patterns: one trace in both cases, because LangChain rebuilds the config from the context variable.

GPT Researcher is two products in one repository. The multi-agent mode is LangGraph and traces like any graph. The standard mode has no graph, and there every model call opens its own trace: eight calls, eight traces, measured.

Its human node does not use interrupt(), it waits on a websocket. The node’s observation stays open while the person thinks, so its duration measures human latency. The 95th percentile of that graph means nothing unless that node is excluded.

Instrumenting both requires no change to their repositories. For Open Deep Research it is enough to import the compiled graph and pass the handler through config. For GPT Researcher’s standard mode you have to wrap with the SDK, because there is nowhere to hook callbacks.

You are here: OBSERVE, day 2

The previous article produced a formula over a toy graph: 4 + 5n observations per turn, with the byte volume growing with the square of the steps. A formula over a toy is good for understanding the mechanism and no good for sizing anything.

This article applies it to two real agents, chosen because they are well known and because they break in different ways. The series grows to ten articles and the migration one moves along one place.

The analogy: two factories, one audit

A traceability audit in a factory that runs the whole process on its own lines is tedious but complete: every station leaves a record, and the auditor reconstructs the part from beginning to end. The volume of paper is enormous and that is the only problem.

The second factory subcontracts half the process. The auditor sees material going into a building and finished product coming out, with times and quantities, and has nothing about what happens inside. There is little paper. The problem is not the paper either.

Open Deep Research is the first factory and the problem is volume. GPT Researcher, in its standard mode, is the second, and the problem is that the trace does not carry what you need to know. Both are fixable, with different tools, and confusing them leads to buying disk when what is missing is instrumentation.

The two agents, and why these

Open Deep Research (langchain-ai/open_deep_research, 12.7 k stars) is LangChain’s reference research agent. All its work goes through LangChain, so callback instrumentation sees it whole.

GPT Researcher (assafelovic/gpt-researcher, 28.4 k stars) is the best-known open source research agent and it is not from LangChain. Its multi-agent mode uses LangGraph; its standard mode, which is what the API and the web interface run, does not.

One is in-house and the other is not, which for a comparison is what matters.

Open Deep Research: the shape of the graph

Three graphs, two of them nested as subgraphs (src/open_deep_research/deep_researcher.py):

The root graph, deep_researcher (line 701), with four nodes: clarify_with_user, write_research_brief, research_supervisor and final_report_generation.
The supervisor subgraph (line 353), with supervisor and supervisor_tools, iterating between them.
The researcher subgraph (line 589), with researcher, researcher_tools and compress_research, which is a classic react loop.

The multiplication lives in supervisor_tools. When the supervisor decides to delegate, that node launches one researcher subgraph per unit and awaits them together (line 305):

research_tasks = [
 researcher_subgraph.ainvoke({
 "researcher_messages": [HumanMessage(content=tool_call["args"]["research_topic"])],
 "research_topic": tool_call["args"]["research_topic"]
 }, config)
 for tool_call in allowed_conduct_research_calls
]

tool_results = await asyncio.gather(*research_tasks)

The detail that decides whether the trace comes out whole is that config at the end: the node receives LangGraph’s configuration and forwards it to the subgraphs, so the callbacks travel and every researcher hangs off the right place.

The three ceilings in the default configuration (src/open_deep_research/configuration.py) are max_concurrent_research_units = 5, max_researcher_iterations = 6 and max_react_tool_calls = 10. They multiply each other.

What one request costs, measured

I replicated that topology with fake models, the same technique as the previous article: three graphs, the same nesting, the same fan-out with asyncio.gather, three-hundred-character search results and two-hundred-character model replies. No network and no GPU, so it reproduces anywhere.

Units	Iterations	React calls	Observations	generation	chain	Attribute KB	Largest observation
1	1	1	26	8	18	56.8 KB	3.7 KB
2	2	3	92	26	66	257.3 KB	4.6 KB
5	6	10	1,416	370	1,046	8,108 KB	11.0 KB

The last row is the repository’s default configuration. One research request, one person pressing a button once, leaves 1,416 observations and eight megabytes in the observability backend. In a single trace, under a single identifier.

Those eight megabytes are a floor, not a ceiling. On the bench each search result takes three hundred characters; a real Tavily result carrying page content takes between ten and a hundred times more, and that text enters the history and gets re-serialised in every later observation of the same researcher, which is the previous article’s quadratic growth acting on much larger text.

Put into capacity arithmetic: a hundred research requests a day, which for a small team is nothing, is 141,600 observations and on the order of a gigabyte a day uncompressed, with toy searches. With real searches, tens of gigabytes. It is another order of magnitude compared with a chat assistant, and it is the reason this article comes before the capacity one.

Zero AGENT observations

Among those 1,416 observations there is not one of type AGENT. The previous article explained why: the handler decides that type by looking for the string agent in the class path or the run name. Open Deep Research’s nine nodes are clarify_with_user, write_research_brief, research_supervisor, final_report_generation, supervisor, supervisor_tools, researcher, researcher_tools and compress_research.

They are good names. They describe what each node does and they leave out a word that adds nothing for whoever reads the code. The result is that LangChain’s reference research agent, seen from Langfuse, is a tree of chains.

It is fixable from outside, without touching the repository, by naming the subgraph invocation or renaming the two nodes that represent a model decision, which are supervisor and researcher. Without that, any dashboard grouping by observation type will not tell decision work apart from plumbing.

Instrumenting it without touching the repository

Open Deep Research ships as a LangGraph Server graph, with its langgraph.json pointing at deep_researcher. There are two ways to get the handler in, depending on how you run it.

If you invoke it from your own code, the compiled graph is importable and the config is enough:

from langfuse import Langfuse, propagate_attributes
from langfuse.langchain import CallbackHandler
from open_deep_research.deep_researcher import deep_researcher

trace_id = Langfuse.create_trace_id(seed=f"{request_id}")
handler = CallbackHandler(trace_context={"trace_id": trace_id})

with propagate_attributes(
 trace_name="deep-research",
 user_id=user,
 session_id=request_id,
 tags=["odr", "research"],
):
 result = await deep_researcher.ainvoke(
 {"messages": [HumanMessage(question)]},
 config={
 "callbacks": [handler],
 "configurable": {
 "thread_id": request_id,
 "max_concurrent_research_units": 3,
 "max_researcher_iterations": 4,
 },
 },
 )

Lowering those two ceilings from 5 and 6 to 3 and 4 is not an observability recommendation, it is a product decision that also divides the trace volume by three. Better taken at the same time.

If you deploy it on LangGraph Server there is no invocation of yours to put the config into, so the handler gets bound at compile time. The pattern is to wrap the exported graph:

# instrumented_graph.py, and point langgraph.json here
from langfuse.langchain import CallbackHandler
from open_deep_research.deep_researcher import deep_researcher

graph = deep_researcher.with_config({"callbacks": [CallbackHandler()]})

That route loses the per-request seeded trace identifier, because the handler is created once. If the agent uses interrupts, that loss matters, for the reason the previous article measured.

GPT Researcher: two products in one repository

Here is the interesting part, and it is a trap you fall into with any agent that has grown in layers.

The repository holds the gpt_researcher package, which is the engine, and the multi_agents package, which is an orchestration on top. Only the second uses LangGraph.

The orchestrator (multi_agents/agents/orchestrator.py:66) has eight nodes: browser, planner, human, researcher, writer, fact_checker, visualizer and publisher, with two conditional loops, one for human review of the plan and one for fact checking. The researcher node launches in parallel one editor subgraph per section (multi_agents/agents/editor.py:134), and that subgraph has its own revision loop between reviewer and reviser.

All of that traces like any graph. And the models are called with self.llm.ainvoke(messages, **kwargs) from the generic provider (gpt_researcher/llm_provider/generic/base.py:365), which is a LangChain chat model with no explicit config, so it picks up the ambient configuration while running inside a node. The generations show up.

The standard mode is another thing. The GPTResearcher class has no graph, no nodes and does not go through LangGraph at any point. It is the mode the API and the web interface use, that is, the one most people run.

The myth of the fresh config

Before going on, something that looks like a bug and is not has to be dismantled, because it shows up in many agents.

When the researcher node launches the section subgraphs, it does it like this (multi_agents/agents/editor.py:74):

final_drafts = [
 chain.ainvoke(self._create_task_input(research_state, query, title),
 config={"tags": ["gpt-researcher"]})
 for query in queries
]
research_results = [result["draft"] for result in await asyncio.gather(*final_drafts)]

That config is fresh and carries only tags. The method does not even receive the node’s configuration, so it could not forward it. The intuitive conclusion is that the subgraphs lose the callbacks and every section ends up in its own trace.

It does not happen. I set both patterns side by side, Open Deep Research’s forwarding the config and GPT Researcher’s passing a fresh one, and both give one trace with one root span. The reason is that LangChain does not take the config you pass as is: it starts from the ambient configuration held in a context variable and overlays the keys you bring. The callbacks are still there, and context variables are copied into the tasks asyncio.gather creates.

What does get lost are the parent’s tags, replaced by the new list. That is a filtering nuisance, not a broken trace.

The standard mode: eight calls, eight traces

With no graph there is no root run, and with no root run every model call is its own tree. Measured with eight standalone calls to a chat model, with the handler passed by hand on each one:


Observations	8
Distinct traces	8
Root spans	8
Types	8 generation

Eight traces of one observation each, with no relation between them. None of them knows it belongs to the same piece of research. And that is the favourable case, because the standard mode offers no public place to put that handler: the **kwargs reach ainvoke from create_chat_completion (gpt_researcher/utils/llm.py:120), but the path crosses several internal layers that are not meant as an extension point.

The route that works without patching the repository is to wrap from outside with the SDK and let the inner calls hang off that:

from langfuse import get_client, observe, propagate_attributes
from gpt_researcher import GPTResearcher

@observe(name="gpt-researcher", as_type="agent")
async def research(question: str, request_id: str, user: str) -> str:
 with propagate_attributes(user_id=user, session_id=request_id, tags=["gptr"]):
 researcher = GPTResearcher(query=question, report_type="research_report")
 await researcher.conduct_research()
 report = await researcher.write_report()

 client = get_client()
 client.update_current_span(
 output={"length": len(report), "sources": len(researcher.get_source_urls())},
 metadata={"gptr_costs": researcher.get_costs()},
 )
 return report

With that there is one trace per piece of research, with an agent observation at the root and the generations hanging inside, in the same tree. Searches and document chunking do not appear, because they do not go through LangChain. If they are needed, they get added with start_as_current_observation(as_type="tool") around your own calls, which is the first of the three ways out of the span filter the previous article described.

Worth noting researcher.get_costs(). GPT Researcher keeps its own cost accounting, and putting it in as metadata is the only thing that lets you reconcile it later with what Langfuse computes. When the two figures disagree, it is almost always the gateway’s model alias.

The human node that stays open

The multi-agent mode has a human node that asks for an opinion on the plan before researching. It does not use LangGraph’s interrupt(). It waits inside the node, reading from the websocket (multi_agents/agents/human.py).

The two approaches have opposite consequences for the trace. With interrupt(), the graph hands control back and the trace splits in two, which is what the previous article measured. Waiting inside the node, the trace is single but that node’s observation stays open for as long as the person takes, minutes or hours.

Neither is wrong. What is wrong is the dashboard that averages durations without knowing which of the two it has in front of it. With the wait inside the node, the 95th percentile of graph duration measures how long a human takes to answer a message, not how long the agent takes. Alert on that metric and the alert fires at lunchtime.

The practical rule is to exclude human-wait nodes by name from any latency metric, and to measure that wait separately, which tends to be an interesting product metric in its own right.

What to look at in the trace once it arrives

With both agents instrumented, and before building any dashboard, there are four checks that are done once and save weeks:

Count the observations of a typical request. If it comes out above a thousand, this platform’s problem is going to be volume and sampling has to be decided before anything else.
Look for more than one root span. More than one means there is work being traced outside the tree, and you want to know which before there are thousands.
Check that generations carry cost. If they come in at zero, it is the model alias, fixed by registering that name with its price.
Sort observations by duration and remove the ones waiting on a person. What stays at the top is where the time really is.

Checklist

The graph is invoked with a trace identifier seeded per request, not a random one.
The agent’s concurrency and iteration ceilings are set deliberately, knowing they multiply trace volume.
Nodes representing a model decision carry agent in the name, or it has been accepted in writing that there will be no observations of that type.
Work that does not go through LangChain is wrapped with the SDK, or it has been decided that it need not be visible.
Human-wait nodes are identified and excluded from latency metrics.
The agent’s own cost accounting, if it has one, goes in as metadata so it can be reconciled.
There is a measured figure of observations per request, taken from your own installation and not from this article.

Traps

A research agent’s defaults are demo values, not production ones. Five units times six iterations times ten react calls multiply, and whoever chose them was thinking about report quality, not about your ClickHouse.

One repository can hold two agents with two different instrumentation stories. Instrumenting the mode the README documents and not the one the API runs is a mistake with no symptom: the traces arrive, just of something else.

Passing a fresh config to a subgraph does not break the trace. It breaks the tags. The tree survives because the callbacks travel through a context variable.

A node waiting on a person contaminates any duration statistic. And not obviously, because the node finishes cleanly and flags no error.

Good node names and good observation classification are conflicting goals. The winner is whoever writes the agent’s code, who is normally not you.

Eight megabytes per request with three-hundred-character searches. With real results, the number your installation produces will not look like this article’s, and it will be larger.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented (this article): Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Open Deep Research source, commit 1b7d2e8, read on 14 September 2026: src/open_deep_research/deep_researcher.py and configuration.py.
GPT Researcher source, commit 6f99857, read on 14 September 2026: multi_agents/agents/orchestrator.py, multi_agents/agents/editor.py, multi_agents/agents/human.py, gpt_researcher/llm_provider/generic/base.py and gpt_researcher/utils/llm.py.
Own measurement of 14 September 2026 over replicas of both topologies, with LangGraph 1.2.11, langchain-core 1.6.3, Langfuse SDK 4.15.2, InMemorySpanExporter and fake models, no network.
GitHub stars for both repositories, checked on 14 September 2026.

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

Sun, 13 Sep 2026 07:00:00 +0200

Second article in a series of ten about operating Langfuse v4 in production. The first one, what really goes into a trace, walked through the data model by reading the server code. This one measures the other side. Verified against Langfuse 4.15.2 (Python SDK), LangGraph 1.2.11 and langchain-core 1.6.3, with figures taken on 13 September 2026 on a bench with no network.

TL;DR

One agent turn is four observations plus five per tool call. Measured with an in-memory exporter over a minimal ReAct graph: zero tools, 4 observations; one, 9; five, 29; ten, 54; forty, 204. The formula holds exactly across the whole range.

The byte volume grows quadratically, not linearly. The same turn goes from 12.2 KB with one tool call to 1.87 MB with forty. 88 % of those bytes is the input field, because every observation re-serialises the full message history as it stands at that moment.

Tool definitions travel attached to every model call. The handler appends them to the input field as messages with role tool. Measured: 364 bytes per tool. With twenty tools that is 7.3 KB on every single step of the loop.

The AGENT observation type is assigned by string match on the name. A node called planificador comes out as CHAIN; the same node called agente_planificador comes out as AGENT. The logic is literally looking for agent in the class path or in the node name (langfuse/langchain/CallbackHandler.py:397).

An interrupt with a resume produces two separate traces. Verified with invoke, with stream and with subgraphs, and also reusing the same handler object. The SDK ships machinery to reattach the trace, but it only fires if the root run finishes with a control-flow exception, and in LangGraph 1.2.11 the interrupt comes back through the return value.

The fix fits in one line and is measured. Passing trace_context={"trace_id": Langfuse.create_trace_id(seed=thread_id)} when building the handler leaves the interrupt and the resume in the same trace, even with different handlers and therefore with different replicas.

The span processor drops by default whatever it does not recognise. It only exports spans from the SDK’s own tracer, spans carrying some gen_ai.* attribute, or spans from a closed list of thirty-five instrumentation prefixes (langfuse/_client/span_filter.py:11). Your own code’s spans do not arrive unless they come through one of those three doors.

You are here: OBSERVE, day 2

The series was planned with eight articles and has grown to ten. The reason is ordering: the next two are about migrating and about sizing ClickHouse, and both need an input figure that nobody publishes, which is how many observations and how many bytes a real unit of work produces. A chat produces a trace with two observations. An agent produces two hundred. Sizing with the chat figure leads to a cluster that falls over the day someone connects the agent.

So before migrating, measure. And to measure there has to be in front what people actually deploy.

The analogy: the file that gets photocopied whole at every signature

A procurement file passes across six desks. At each desk somebody reads what is there, adds a sheet and passes it on.

There are two ways to leave a record of that route. The first is to write in a register which desk touched it, when, and which sheet it added. Six short entries. The second is to photocopy the whole file on the way into each desk and on the way out, and to archive the twelve copies. The register takes one sheet. The copies take, at the sixth desk, six times what they took at the first, and in total they take something on the order of the square of the number of desks.

LangChain’s callback instrumentation does the second thing. It does not record the delta of each step, it records the complete state on the way into and out of each step. With a chat that makes no difference, because the state is short and there is one step. With a forty-step agent, the archive weighs almost two megabytes per file processed.

This is not a defect that needs fixing. It is what lets you open a trace and see exactly what the model saw at step 17 without reconstructing anything. But it drives the sizing, and the number has to be known before signing the contract.

Which agentic platform is the most used

The question has two answers depending on the metric, and both are defensible.

By PyPI downloads over the last thirty days, checked on 13 September 2026:

Package	Downloads / 30 days
`langgraph`	52.3 M
`strands-agents`	35.2 M
`openai-agents`	22.9 M
`crewai`	19.1 M
`google-adk`	14.5 M
`pydantic-ai`	8.0 M
`llama-index`	5.0 M
`agno`	1.9 M
`autogen-agentchat`	0.6 M

By GitHub stars the order flips at the top: CrewAI is at 58.4 k, LangGraph at 38.1 k and the OpenAI Agents SDK at 28.6 k.

PyPI downloads are inflated by continuous integration and by images that reinstall on every build, so they indicate presence in pipelines rather than agents in production. Stars measure interest, not deployment. With both together, LangGraph is the one that shows up in more pipelines and the one with the most worked-through instrumentation path in Langfuse, so it is the one instrumented here. What follows about volume applies equally to any platform instrumented through LangChain callbacks, because the cost comes from the callback model, not from the graph.

How LangGraph gets into Langfuse

The route is worth stating, because the previous article left two open and only one is used here.

langchain-core 1.6.3 does not import OpenTelemetry anywhere. The check is direct against the installed wheel and returns not one file. LangGraph does not emit OTLP spans on its own either. What exists is LangChain’s callback system, and the Langfuse CallbackHandler hooks into it.

That handler does not talk to the classic ingestion API. It creates observations with the v4 SDK, which underneath is OpenTelemetry: LangfuseSpanProcessor sets up an OTLPSpanExporter against {base_url}/api/public/otel/v1/traces (langfuse/_client/span_processor.py:123). So it is OTLP, but generated by the SDK, not by third-party auto-instrumentation.

The difference matters for one concrete reason left hanging in the previous article. Because the data passes through the SDK calls, the classic masking function does apply: input, output and metadata all go through _process_media_and_apply_mask before becoming attributes (langfuse/_client/span.py:534). With auto-instrumentation of the OpenInference or OpenLLMetry kind, the messages travel in span events and masking does not touch them. For an agent handling customer data inside its tools, that difference decides the route.

The trade-off is that a callback handler only sees what goes through LangChain. HTTP calls a tool makes on its own, the database query, the subprocess, none of that shows up. And if you instrument your code with OpenTelemetry so that it does show up, you run into the filter described further down.

The observation type is decided by the node name

The first article insisted that v4 has ten observation types and that using the right one changes what the interface knows how to group. The LangChain handler emits six of those ten: tool, retriever, generation, agent, chain and span. It never emits event, embedding, evaluator or guardrail.

The assignment is direct for four of them: tool callback, type tool; retriever callback, type retriever; model callback, type generation. For chain callbacks, which is what every node of a LangGraph graph is, the criterion is this (langfuse/langchain/CallbackHandler.py:393):

elif callback_type == "chain":
 # Detect if it's an agent by examining class path or name
 if serialized and "id" in serialized:
 class_path = serialized["id"]
 if any("agent" in part.lower() for part in class_path):
 return "agent"

 name = self.get_langchain_run_name(serialized, **kwargs)
 if "agent" in name.lower():
 return "agent"

 return "chain"

Measured over the same graph with the node renamed and everything else identical:

Node name	Tool calls	Observations	Types
`planificador`	10	54	11 generation, 33 chain, 10 tool
`agente_planificador`	10	54	11 generation, 22 chain, 11 agent, 10 tool

Eleven observations change type because of eleven characters in a function name. This has two practical consequences. The first is that any dashboard counting AGENT observations is counting naming conventions. The second is that the interface draws the agent graph when the trace contains any observation of a type other than span, event or generation, and chain qualifies, so the drawing comes out either way. What changes is what you can filter and aggregate afterwards.

The operational recommendation is boring and it works: name the nodes that represent a model decision with a name containing agent, and leave the rest as chains. It is a naming convention promoted to data schema, which is not pretty, but it is what there is and it is stable.

What a turn costs, measured

The setup is a minimal ReAct graph: one node that calls the model, a ToolNode with one tool, a conditional edge with tools_condition and an in-memory checkpointer. The model is langchain-core’s GenericFakeChatModel, which returns a fixed list of messages, so neither network nor GPU is needed and the experiment reproduces on any laptop. The exporter is OpenTelemetry’s InMemorySpanExporter, passed to the client through the span_exporter parameter, so observations are counted with no Langfuse server in front.

Result, with the node named planificador:

Tool calls	Observations	generation	chain	tool	Attribute bytes
0	4	1	3	0	4.2 KB
1	9	2	6	1	12.2 KB
5	29	6	18	5	64.1 KB
10	54	11	33	10	173.5 KB
20	104	21	63	20	541.4 KB
30	154	31	93	30	1,108 KB
40	204	41	123	40	1,874 KB

The observation count is exactly 4 + 5n. The five per loop iteration are: the model node, the model call, the conditional edge, the tools node and the tool. The four fixed ones are the root graph, the first partial pass and the close.

The bytes do not follow that line. From 10 to 20 steps the volume multiplies by 3.1; from 20 to 40, by 3.5. That is the quadratic growth expected when each of the n steps drags along a history whose length is proportional to n.

The breakdown by field confirms it. With forty steps, out of 1,709 KB measured in that run, 1,502 KB sit in langfuse.observation.input and only 56 KB in the output. 88 % of what is ingested is the same history repeated at different stages of progress. The largest single observation in that turn is 22.2 KB, well below the 9.5 MB per OTLP span ceiling, so the problem is not the size limit: it is the aggregate.

And inside the input there is a second multiplier. The handler appends each tool’s definition to the generation’s input field, as messages with role tool (langfuse/langchain/CallbackHandler.py:1190). Measured with three-parameter tools and a one-line description:

Bound tools	Input field bytes of one generation
0	34
1	398
5	1,854
10	3,674
20	7,334

That is 364 bytes per tool, on every generation, always. An agent with twenty tools and forty steps ingests 7.3 KB × 41 generations = 300 KB in tool schemas alone, which never change over the course of the turn.

What this means for ClickHouse

A honest caveat belongs here. I compressed the full payload of those runs with ZSTD level 3, which is what events_full uses, and got ratios between 29× and 86×, improving the longer the turn. That number is inflated by how synthetic the bench is: my tool always returns the same string, so the redundancy is higher than in production.

What does hold is the shape. Repeating the history is exactly the pattern columnar compression absorbs well, so the ClickHouse disk is not where it hurts. Where it hurts is earlier: in the bandwidth towards the OTLP endpoint, in the worker CPU that decompresses and flattens, and in the ingestion queue, all of which see uncompressed bytes. The first article noted that the queue that saturates first is langfuse.queue.ingestion.depth, and this is why it saturates.

Quick sizing arithmetic, with the measured figures and an average ten-step turn:

1,000 turns a day is 54,000 observations and around 170 MB uncompressed a day.
10,000 turns a day is 540,000 observations and 1.7 GB uncompressed a day.
If the average turn goes from ten steps to twenty, observations double and bytes triple.

Sensitivity to the number of steps is the parameter to watch, more than the number of users. A prompt change that makes the agent take three more turns around the loop multiplies the observability bill with nobody touching the infrastructure. Article 6 of the series comes back to this with ClickHouse system tables to measure the real bytes per observation in your own installation, which is the only thing that replaces this estimate.

Interrupts: two traces where there should be one

A production agent with human approval uses interrupt(), hands control back, waits for a person and resumes with Command(resume=...). The operational question is whether the trace survives that pause.

The SDK ships explicit machinery for the case. There is a store of pending trace contexts keyed by thread_id, capped at 1,024 entries with eviction of the oldest (MAX_PENDING_RESUME_TRACE_CONTEXTS), and a function that recognises a Command carrying resume and recovers the stored context.

Measured, it does not fire in this combination of versions:

Case	Same trace?
Same handler object, `invoke`	No
Same handler object, `stream`	No
Interrupt inside a subgraph	No
Fresh handler for the resume	No

The reason is in the code and it is consistent. The pending context is only stored from on_chain_error when the failing run is the root one and the resulting level is DEFAULT, which is what happens with LangGraph’s control-flow exceptions (CallbackHandler.py:866). In LangGraph 1.2.11 the interrupt does not leave the root as an exception: the graph returns normally, with the interrupt in the return value. The root run ends through on_chain_end and the branch that stores the context never runs. The interrupted node’s observation does come out right: level DEFAULT and the Interrupt in the status message, not marked as an error, which is the correct behaviour.

The fix is verified and does not depend on the SDK changing:

from langfuse import Langfuse
from langfuse.langchain import CallbackHandler

trace_id = Langfuse.create_trace_id(seed=thread_id)
handler = CallbackHandler(trace_context={"trace_id": trace_id})

With that, the interrupt and the resume land in the same trace even if the handler is a different object and even if another replica serves the resume, because the identifier derives from the thread_id and not from in-memory state. It is the same technique the documentation recommends for joining several agents into one trace, applied to joining an agent with itself.

With one nuance to decide deliberately. Seeding with the bare thread_id puts every turn of that conversation into a single trace, which grows without bound while the conversation stays alive. If what you want is one trace per turn with the resume attached to the right turn, seed with thread_id plus the turn number, and leave conversation grouping to the session identifier, which is what it is for. Since v4 has no trace table, a huge trace is not a huge row: it is a filter returning many rows, and what suffers is the interface when opening it.

Thread, session, user, and what is only read at the root

The handler recognises five metadata keys: langfuse_session_id, langfuse_user_id, langfuse_trace_name, langfuse_tags and langfuse_prompt. The first four are read only when parent_run_id is None, that is, only on the root run (CallbackHandler.py:587). Putting them in the config used to invoke a subgraph does nothing.

The recommended route, and the one that works without surprises, is the context manager:

from langfuse import propagate_attributes

with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["production", "support-l1"],
):
 graph.invoke(payload, config=cfg)

What LangGraph contributes on its own arrives anyway. In the measured run these show up as observation metadata: langgraph_node, langgraph_step, langgraph_triggers, langgraph_path, langgraph_checkpoint_ns, checkpoint_ns and thread_id, plus ls_provider, ls_model_type and ls_integration. The thread_id is also promoted to trace metadata.

That is fine, but keeping in mind the first article: in events_core what is indexed is metadata_names, that is the key names, not the values. Filtering by langgraph_node = "planificador" is a scan. If you are going to segment by node routinely, the right place is the observation name, which is where the node already appears, or a tag, assuming that tags are not indexed either. The cheap field for slicing by node in v4 is the observation name itself.

The session identifier deserves an explicit decision. LangGraph’s thread_id and Langfuse’s session_id are the same concept and they do not connect on their own. Mapping them one to one is the reasonable thing, and it is also what makes the whole conversation navigable in the interface even when each turn is a separate trace.

The filter that drops your own spans

This is the finding that surprises most people while setting it up. The Langfuse span processor does not export everything that passes through the TracerProvider. It exports what satisfies one of these three conditions (langfuse/_client/span_filter.py:104):

It was created by the Langfuse SDK’s own tracer.
It carries at least one attribute starting with gen_ai.
Its instrumentation scope matches one of thirty-five prefixes in a closed list.

The list covers the expected and a bit more: openinference, litellm, haystack, langsmith, strands-agents, pydantic-ai, autogen-core, vllm and a couple of dozen opentelemetry.instrumentation.* entries.

The consequence for an agent is direct. If you instrument your tools with OpenTelemetry by hand so that the ERP call or the Postgres query shows up inside the agent’s trace, those spans are dropped silently, with no log and no error. There are three ways out, in order of cleanliness:

Create those observations with the Langfuse SDK, with start_as_current_observation(as_type="tool"), which is what the rest of this article assumes.
Put some gen_ai.* attribute on the span, which is abusing the convention to smuggle the span through, so only if the span does describe an interaction with a model.
Send those spans to your normal OpenTelemetry collector instead of Langfuse, and correlate by trace identifier. For a sovereign platform with its own collector, this is the sensible option: Langfuse keeps the model side, the collector keeps the system side, and the trace identifier joins them.

The behaviour being silent is what makes this take an afternoon to find. It belongs in the runbook.

The model and the cost with a gateway in the middle

One detail that affects anyone putting LiteLLM in front, which is the setup described in the operational pair and in GPU multi-tenancy.

The model name the handler puts on the generation comes first from the ls_model_name metadata and, failing that, from the component’s serialisation (CallbackHandler.py:1297). With a ChatOpenAI pointed at the gateway, that name is whatever alias you defined in LiteLLM, for example qwen-32b-internal. Langfuse computes cost from its own model table, which does not know that alias, so the generation arrives with tokens and without cost.

It is fixed by defining the model in Langfuse under that same name with its price per million tokens, which is also the only way for the cost to reflect what your GPU costs rather than what somebody else’s API costs. It is an afternoon of work and it stops the agentic platform’s cost panel from sitting at zero forever.

How this looks in code

The minimum to have in place for a LangGraph agent to produce useful traces in Langfuse v4:

import os
from typing import Annotated, TypedDict

from langfuse import Langfuse, get_client, propagate_attributes
from langfuse.langchain import CallbackHandler
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition

# 1. Client. Masking applies because the data goes through the SDK.
def mask(*, data, **kwargs):
 return redact_pii(data)

Langfuse(
 public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
 secret_key=os.environ["LANGFUSE_SECRET_KEY"],
 host=os.environ["LANGFUSE_HOST"],
 mask=mask,
 environment="prod", # lowercase, dashes, 40 chars, must not start with "langfuse"
 sample_rate=float(os.getenv("LANGFUSE_SAMPLE_RATE", "1.0")),
)

# 2. Graph. The decision node carries "agent" in its name on purpose.
class State(TypedDict):
 messages: Annotated[list, add_messages]

graph = StateGraph(State)
graph.add_node("agent_planner", model_node)
graph.add_node("tools", ToolNode(TOOLS))
graph.add_edge(START, "agent_planner")
graph.add_conditional_edges("agent_planner", tools_condition)
graph.add_edge("tools", "agent_planner")
app = graph.compile(checkpointer=checkpointer)

# 3. Per turn: deterministic trace seeded with thread and turn, session = thread.
def run_turn(thread_id: str, turn: int, text: str, user: str):
 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {
 "callbacks": [handler],
 "configurable": {"thread_id": thread_id},
 }
 with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["agent", "support"],
 ):
 out = app.invoke({"messages": [HumanMessage(text)]}, config=cfg)
 return trace_id, out

# 4. Resume after human approval: same seed, same trace.
def resume_turn(thread_id: str, turn: int, decision):
 from langgraph.types import Command

 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {"callbacks": [handler], "configurable": {"thread_id": thread_id}}
 return app.invoke(Command(resume=decision), config=cfg)

# 5. Score the turn. Scores do not travel over OTLP: they have their own route.
get_client().create_score(
 trace_id=trace_id,
 name="resolved",
 value=1,
 data_type="BOOLEAN",
)

Two notes on what is not in the example. sample_rate is head sampling and per trace, so it samples whole turns, which is what you want: half an agent trace is no use to anybody. And if the process is a web server, the client is built once at startup, not per request, whereas the handler can be per turn with no problem, because with the seeded identifier it no longer holds state worth preserving.

Agent instrumentation checklist

Nodes representing a model decision carry agent in the name. The rest do not.
Every turn has a deterministic trace identifier seeded with thread and turn number.
LangGraph’s thread_id goes as Langfuse’s session_id, always.
Trace attributes are set with propagate_attributes, not as metadata on a subgraph.
The gateway’s model alias is registered in the Langfuse model table, with a price.
Tools doing work outside LangChain are instrumented with the Langfuse SDK, not with bare OpenTelemetry.
There is an alert on ingestion queue depth, and the average number of steps per turn is known.
Someone has measured, on the actual installation, how many observations and how many bytes a real turn produces. The figures in this article come from a synthetic bench.
Sampling is set through an environment variable, so it can be lowered without deploying code.

Traps

Type AGENT does not mean there is an agent. It means somebody wrote agent in the name. And the other way around: a whole graph without that string produces not one agent-type observation.

Putting langfuse_session_id in a subgraph’s metadata does nothing. It is only read on the root run.

An interrupt splits the trace in two. Unless you seed the identifier. And with several replicas, the handler’s in-memory state is never going to save you, because the resume can land on another pod.

Your OpenTelemetry spans do not arrive. The default filter drops them without warning if the scope is not in the list and there are no gen_ai.* attributes.

The number to watch is not users, it is steps per turn. Volume grows with the square of the steps, and the steps are decided by the prompt, not by the infrastructure.

You store megabytes and the interface shows you a thousand characters. The materialised view’s truncation to two hundred characters and the thousand-character read ceiling are still there. You pay for the full ingestion and see a clipping, unless you open the specific observation.

Cost comes out at zero with the gateway in front. Model alias unknown to the price table.

flush_at above the queue size blows up when constructing the client. With an OpenTelemetry ValueError that does not mention Langfuse anywhere.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front (this article): instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Own measurement of 13 September 2026: LangGraph 1.2.11, langchain-core 1.6.3 and Langfuse Python SDK 4.15.2, with OpenTelemetry’s InMemorySpanExporter and GenericFakeChatModel, no network.
Langfuse SDK 4.15.2 source: langfuse/langchain/CallbackHandler.py, langfuse/_client/span_filter.py, langfuse/_client/span_processor.py, langfuse/_client/span.py.
PyPI downloads over the last thirty days via pypistats, checked on 13 September 2026.
GitHub stars for LangGraph, CrewAI and the OpenAI Agents SDK, checked on 13 September 2026.

Langfuse v4, day 2 (1 of 10): the data model changed underneath, and what really goes into a trace

Sat, 12 Sep 2026 07:00:00 +0200

First article in a series of ten about operating Langfuse v4 in production. The architecture of version 3 and its ten performance levers were already covered in Langfuse from the inside; the seam with the gateway, in the operational pair. This series is about day 2. Verified against Langfuse 4.35.0, commit of 11 September 2026.

TL;DR

Version 4 is observations first. There is no trace table. A trace is a set of observations that share an identifier, and the trace attributes live denormalised in every row: name, user, session, tags, version, release and environment (clickhouse/migrations/canonical/0039_create_events_full.up.sql). The whole intuition of “first I create the trace and then I hang things off it” stops describing the storage.

There are ten observation types, not three. Alongside the classic SPAN, EVENT and GENERATION come AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (packages/shared/src/domain/observations.ts:5). Using the right type changes what the interface knows how to group.

The real limits are in the code and they do not match what people assume. The body of a request to the classic ingestion route is capped at 4.5 MB (api/public/ingestion.ts:40). The OTLP body, at 512 MiB, compressed and decompressed, with a 413 when it is exceeded. An OTLP span above 9.5 MB is only written to the log, not rejected. Identifiers accept 800 characters. The trace name, 1,000.

The ingester looks for the user in five places and the session in six. With strict precedence order, starting from Langfuse’s own attributes and ending in the metadata of other frameworks. Among the session ones sits gen_ai.conversation.id, which is the OpenTelemetry standard, so a conformant instrumentation works without translation.

Scores do not travel over OTLP. Zero occurrences of the word across the 3,864 lines of the OTLP ingestion processor. They come in through the classic ingestion route, through the scores endpoint or through the interface. Any design that assumes an evaluator can write its score down the same channel as the trace is wrong.

The classic masking function does not cover what almost everyone thinks it covers. It only acts on data that passes through the SDK calls. The attributes produced by a third-party automatic instrumentation come out unmasked. There is a new function that does act on the batch attributes, but it leaves out span events, which is where the recent conventions put the messages.

Filtering by tags or by status message is a scan. In the listings table, the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the dates, the model and the metadata names. The name, the tags, the version and the status message have no index.

You are here: OBSERVE, day 2

The previous series about this pair covered selection and setup. The six-service architecture explained version 3; tracing with OpenTelemetry explained the pipeline; the pair with the gateway explained where the trace gets lost between the two pieces.

What is left uncovered is the work of the next three years: migrating, sizing ClickHouse, retaining and deleting, taking backups, alerting, getting the data out. This series is about that, and it starts with the question that conditions everything else, which is what gets stored.

The analogy: the archive that changed system

An archive with a folder per case file and documents inside each folder is easy to understand. You open the folder, you look at what is there. The problem shows up when you have to find every document of one kind over the last two years: you have to open every folder.

Large archives solve that the other way around. There is no folder: each document carries, copied into its header, the case reference, the client name and the date. Searching by client is immediate. Reconstructing a case file is grouping by reference. In exchange, each document weighs a little more, and what gets written into that header matters a great deal, because everything you can search for comes from there.

Langfuse v4 made exactly that change. What used to be a folder with attributes of its own now travels repeated in the header of every document. The two operational consequences of this article come from there: which fields are worth filling in and which fields are not worth filling with content.

The data model, as it is in the code

The tables

Three new pieces in ClickHouse:

events_full (migration 0039): the table of record, with the full input and output, compressed with ZSTD level 3 and indexed with full-text indexes over the lowercase versions. ReplacingMergeTree engine ordered by event timestamp and deletion flag, partitioned by month of the start time.
events_core (migration 0040): the listings table, lighter.
events_core_mv (migration 0041): the materialised view that fills the previous one, and which truncates to two hundred characters the input, the output and each metadata value.

That truncation to two hundred characters is the first thing to know. What you see in a listing is not what is stored; it is a clipping. And when reading a full trace a second cap applies on top, LANGFUSE_SERVER_SIDE_IO_CHAR_LIMIT, with a default value of a thousand (packages/shared/src/env.ts:513).

In the events_full table there is a configuration line that says a lot: index_granularity_bytes = '64Mi', with the comment that it avoids very small granules because of large rows. It is the schema itself acknowledging that the rows of this system are bulky.

The ten observation types

SPAN, EVENT, GENERATION, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (domain/observations.ts:5), with their corresponding ingestion events (server/ingestion/types.ts:279).

For a platform with agents, the difference between marking a tool call as a generic SPAN or as a TOOL is the difference between being able to count tool calls per session and having to reconstruct it by hand. It is worth reviewing the instrumentation with this list in front of you.

The level is still DEBUG, DEFAULT, WARNING and ERROR (domain/observations.ts:31), with a free-form status message.

The limits, with their source

Field	Limit	File
Trace, span and observation identifiers	800 characters, no carriage return	`ingestion/types.ts:10`
Trace name	1,000 characters	`ingestion/types.ts:428`
Environment	40 characters, pattern of lowercase, digits, hyphen and underscore	`ingestion/types.ts:226`
Text-type score	500 characters	`domain/scores.ts:44`
Score config name	35 characters	`domain/score-configs.ts:11`
Classic ingestion body	4.5 MB per request	`api/public/ingestion.ts:40`
OTLP body	512 MiB, adjustable	`web/src/env.mjs:583`
OTLP span considered large	9.5 MB, only logged	`packages/shared/src/env.ts:193`
Name, status message, version, release, tags	no declared limit	`ingestion/types.ts:424`
Metadata	no declared limit	`ingestion/types.ts:425`

The last two are the dangerous ones, because they have no schema cap and the real ceiling ends up being the one on the request body. A three-megabyte metadata value passes validation and reaches ClickHouse.

A note on the langfuse prefix in the environment field: it is reserved for the product’s own internal traces and is stripped idempotently from public inputs (ingestion/types.ts:228), including the case of someone trying to sneak it in duplicated. The default value is default.

On rate limits: in the cloud there are per-plan quotas, but in self-hosted they are not applied. The service returns directly that there is no limiting (features/public-api/server/RateLimitService.ts). It is good news for bulk exports and bad news for anyone relying on that mechanism as protection.

The two instrumentation routes

What gets lost over OTLP, and it is one specific thing

The check is simple and the result is blunt: there is not a single occurrence of the word score in the OTLP ingestion processor, a file of 3,864 lines. Scores only come in through three places: the score-create event of the classic ingestion route, the POST /api/public/scores endpoint, or the interface.

This has a design consequence worth being clear about before building evaluators: the trace can travel through the OpenTelemetry collector with everything good that brings, on-disk queue, retries and sampling, but the score an evaluator puts on it has to go down a different channel and with the project keys.

Everything else does travel over OTLP. Whatever is not in the mapping lists is kept as metadata converted to a string, visible but outside the indexed fields.

The precedence chain, which is the map you need to have

Verified in packages/shared/src/server/otel/OtelIngestionProcessor.ts:

User identifier (line 2545), in this order: langfuse.user.id, user.id, the observation metadata, the trace metadata, and lastly the Vercel SDK metadata.

Session identifier (line 2565): langfuse.session.id, session.id, gen_ai.conversation.id, observation metadata, trace metadata, Vercel metadata.

Environment (line 2383): langfuse.environment, deployment.environment.name, deployment.environment; for each key the span attributes are checked first and the resource attributes after.

Model (line 2753): langfuse.observation.model.name, gen_ai.response.model, ai.model.id, gen_ai.request.model, llm.response.model, llm.model_name, model.

Level (line 1250): the native attribute, with accepted aliases where SUCCESS and OK become DEFAULT, WARN becomes WARNING and FATAL or CRITICAL become ERROR; if there is no attribute and the span status is an error, the result is ERROR.

Tags (line 3227): langfuse.trace.tags, langfuse.tags, observation and trace metadata, Vercel metadata, and tag.tags.

Input and output (line 1790): a chain of some twenty frameworks in strict order, starting from Langfuse’s own attributes and going through Genkit, the Vercel SDK, the standard inference operation details event of the conventions from 1.37 onwards, the per-role message events, the prompt and completion attributes, Logfire, MLflow, TraceLoop, OpenInference and the gen_ai.input.messages and gen_ai.output.messages attributes.

That this chain has twenty entries says something about the real state of the conventions: it was already noted that the generative AI conventions were extracted into their own repository without any published version. While that remains the case, the backend has to guess, and it guesses with a precedence list.

The practical part: if Langfuse’s own attributes are emitted, they always win. If you want backend-neutral instrumentation, gen_ai.conversation.id and deployment.environment.name are the two standard attributes that Langfuse does understand.

Metadata, and how it ends up in the table

A metadata value is flattened into two parallel arrays, one of names and one of values, with dot-separated paths and everything converted to a string (otel/utils.ts:239, applied in worker/src/services/IngestionService/index.ts:363). A nested object with twenty keys is twenty entries in each array.

There is a cap of 10,001 positions when reconstructing arrays from OTel attributes (OtelIngestionProcessor.ts:271), which is high but it exists.

Sessions, users and cardinality

A session is a text column with a Bloom filter index at 1 %. The same goes for the user. The Postgres table that accompanies sessions stores only whether it is bookmarked, whether it is public and its environment; there is no counter and no cap. There is no limit on traces per session in the code, and the documentation recommends session identifiers below 200 ASCII characters.

The cardinality difference matters and it is not intuitive: user and session are plain text, so high cardinality does not break the schema, it only inflates the Bloom filters. The environment, by contrast, is declared low cardinality. Generating a dynamic environment value per tenant degrades ClickHouse measurably.

Aggregated cost comes out of a cost details map with materialised input, output and total columns. The classic sessions endpoint is deprecated; the replacement is the version 2 observations one with a filter.

Scores: five types and three sources

The types are NUMERIC, CATEGORICAL, BOOLEAN, CORRECTION and TEXT (domain/scores.ts:45). The boolean requires a value of zero or one. The correction does not accept an associated config. The text one is capped at 500 characters.

The sources are API, EVAL and ANNOTATION. The EVAL source is reserved for the internal evaluators: the public API only accepts the other two. Annotation scores require an associated config except for the correction one.

On what costs money, a clarification that corrects a widespread belief: in 4.35.0 the entitlement for the number of model-based evaluators is unlimited on every plan, including the open edition (features/entitlements/constants/entitlements.ts:60). What is limited by plan is the number of annotation queues and the days of data access. And there is a version nuance: trace-level evaluators are discontinued in version 4.

Masking, and what it does not cover

This is the part to read twice if the deployment has to comply with anything.

The classic Python SDK function has the signature masking_function(*, data, **kwargs) and covers only the data that passes through the SDK calls: observation creation, update, and setting the trace input and output. It does not cover the raw OTel attributes produced by a third-party automatic instrumentation.

The replacement is a function that acts on the attributes and the resource attributes in the export batch. That one does reach automatic instrumentation, with a limitation declared in the documentation: it cannot change the span name, the identifiers, the parent relationship, the resource attributes, the events or the links.

And there is the gap that matters: the generative AI conventions from 1.37 onwards put messages and responses into span events. That is, the route by which prompts arrive in a modern instrumentation is precisely the one the new masking does not touch.

For a deployment with personal data, the conclusion is that client-side masking is not enough and a redaction processor has to go into the collector. It is the same reasoning that led in the previous article to recommending the collector in the middle, and it adds to the already documented problem of MCP tool arguments being written in clear, bypassing the message redaction switch.

On the server side there is no safety net: Langfuse has no server-side personal data masking. The encryption key that gets configured protects secrets, that is, model provider credentials, integration headers and SSO configurations. Trace inputs and outputs are not encrypted at application level.

Volume control

Three levers, in order of how high up they act:

In the SDK: sample_rate, or the LANGFUSE_SAMPLE_RATE variable, with head sampling at trace level. If the trace is not sampled, neither its observations nor its scores travel. LANGFUSE_TRACING_ENABLED set to false switches the whole thing off.

In the collector: probabilistic sampling and tail sampling. It is the only layer where you can decide based on the outcome, that is, keep every error and a fraction of the rest.

In the server: a deterministic per-project sampling, with a SHA-256 hash of the trace identifier (server/ingestion/sampling.ts). Deterministic per trace, so it does not cut traces in half.

What not to put into a trace

With the schema in front of you, the list is concrete.

Do not filter by tags or by status message. In the listings table the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the creation and update dates, the model name, the experiment identifier and the metadata names. The name, the tags, the version, the release and the status message are not. Filtering by them scans.

Do not use metadata as a store. It gets flattened into string arrays, truncated to two hundred characters in the listings table but stored whole in the full table, and on top of that it is indexed with a full-text index segmented on non-alphabetic characters. Bulky metadata multiplies the inverted index, not just the data.

Do not store full prompts without thinking about it. The input and the output carry ZSTD level 3 compression and two full-text indexes over their lowercase versions. Storing the entire context of a 142,000-token agentic call on every turn inflates the inverted index disproportionately. Client-side truncation is the cheap lever.

Do not use the environment to separate tenants. It is low cardinality, it has a 40-character cap and it isolates nothing: same project, same keys, same limits, same retention. It is there to segment dashboards. Real isolation means separate projects, as was already argued in the article on the seams.

Version 3 and the calendar date

The constant is in the code: V3_SUNSET_DATE = "2026-11-16" (features/public-api/server/deprecations.ts:11).

It is worth being precise about what it means, because the public documentation and the code do not say exactly the same thing. The code comment is literal: the classic ingestion route is never switched off and continues to accept score events; trace and observation events fail only in version 4’s exclusive write mode, not in the mixed or classic modes. And when they fail, they fail per event, not per batch.

The x-langfuse-ingestion-version: 4 header activates direct writing to the events table, which is the real-time route. Without it the deferred path is taken. Recent SDKs send it by themselves. Any value above four is rejected.

As for names, trace-level input and output attributes are deprecated in favour of the observation ones on the root span. The classic read endpoints carry a deprecation marker in the response body, and their replacements are version 2 of observations, version 3 of scores and version 2 of metrics. And dataset runs are renamed experiments.

There is a curious piece that deserves a mention because it shows up in the Redis dashboards and is misleading: a queue called version 4 classic API usage, which ingests nothing. It is a periodic task every fifteen minutes that scans the ClickHouse query log to detect which projects are still using the old API, and materialises the result into Redis for the interface’s migration notice.

How this looks in code

Two pieces, the client one and the collector one.

On the application side, what changes compared with the usual instrumentation is the observation type and where the identity comes from:

from langfuse import get_client, propagate_attributes

lf = get_client()

# identity propagates by context, not observation by observation
with propagate_attributes(
 session_id=id_conversacion, # the same across every turn
 user_id=sujeto_del_token, # from the claim, not from the body
 environment="produccion",
 tags=["soporte", "n2"],
):
 # the type matters: the interface groups by it
 with lf.start_as_current_observation(
 as_type="agent", name="resolver-incidencia"
 ) as raiz:

 with lf.start_as_current_observation(
 as_type="tool", name="buscar_en_inventario"
 ) as herramienta:
 resultado = mcp.call("buscar", args)
 # truncate here, not on the server
 herramienta.update(output=str(resultado)[:4000])

 # the score goes down its own channel, never over OTLP
 lf.score_current_trace(name="resuelta", value=1, data_type="BOOLEAN")

The as_type parameter accepts nine literals: span, generation, embedding, agent, tool, chain, retriever, evaluator and guardrail. The tenth type in the model, the event, has its own constructor because it does not open a scope.

On the collector side, the redaction processor that compensates for what the SDK masking does not reach:

processors:
 redaction:
 allow_all_keys: true
 blocked_values:
 - '\b\d{8}[A-HJ-NP-TV-Z]\b' # national ID number
 - '\b[\w.+-]+@[\w-]+\.[\w.]+\b' # email
 summary: silent
 tail_sampling:
 policies:
 - name: errores
 type: status_code
 status_code: {status_codes: [ERROR]}
 - name: resto
 type: probabilistic
 probabilistic: {sampling_percentage: 5}

exporters:
 otlphttp/langfuse:
 endpoint: http://langfuse-web.observabilidad.svc:3000/api/public/otel
 headers:
 Authorization: "Basic ${LANGFUSE_BASIC}"
 x-langfuse-ingestion-version: "4"
 sending_queue:
 enabled: true
 storage: file_storage
 retry_on_failure:
 enabled: true

The endpoint carries the base path without the signal suffix: the exporter adds its own. And the on-disk queue is the main reason for putting the collector in the middle, because the SDK’s batch processor drops when it fills up.

Instrumentation checklist

Review the type of every observation against the list of ten. Tool calls as TOOL, agents as AGENT, guardrails as GUARDRAIL.
Decide where the user identifier comes from and set it in a single place, high up the precedence chain.
Emit gen_ai.conversation.id if you want neutral instrumentation, or Langfuse’s own attribute if coupling does not matter.
Set the environment with deployment.environment.name, with few distinct values and never one per tenant.
Get scores out through their own channel, with the project keys, and do not expect them to travel with the trace.
Truncate input and output on the client before sending them, especially with long agentic contexts.
Put a redaction processor in the collector, because the SDK masking leaves out span events.
Do not filter by tags in dashboards that get queried often.
Send the ingestion version header, or use recent SDKs that send it by themselves.
Review the size of the metadata: there is no schema cap, the cap is the one on the request body.

Traps and things that are not what they look like

What you see in a listing is truncated to two hundred characters by the materialised view, and the full read has another cap of a thousand. The data is whole in the full events table.

Metadata has no size limit in the schema. A blob of megabytes passes validation.

An OTLP span over 9.5 MB is not rejected, it is only logged. There is no signal back to the client.

The environment is declared low cardinality. One value per tenant degrades ClickHouse.

Scores do not travel over OTLP. It is by absence in the code, not by configuration.

The classic masking does not cover automatic instrumentation, and the new one does not cover span events, which is where the recent conventions put the messages.

Langfuse does not encrypt trace inputs or outputs. The encryption key protects integration credentials.

In self-hosted there are no rate limits on the API. It is no use as protection.

The classic ingestion route is not switched off on 16 November, it changes behaviour depending on the write mode, and it fails per event instead of per batch.

Model-based evaluators are not limited by plan in this version, contrary to what is usually assumed. What is limited are the annotation queues and the days of data access.

The series: the ten articles

This is the first. The order is designed so that each one can be read on its own and so that together they cover the whole of day 2.

What goes into a trace (this article): version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Anuncio de Langfuse v4, 17 de agosto de 2026.
Compatibilidad y migración, consultado el 12 de septiembre de 2026.
Sesiones, muestreo y enmascarado, consultados el 12 de septiembre de 2026.
Integración nativa con OpenTelemetry, consultado el 12 de septiembre de 2026.

Langfuse from the inside: the sorting centre that must not become the bottleneck it came to observe

Sat, 06 Jun 2026 06:00:00 +0200

This post closes a trilogy of the Observe layer: in LLM tracing with OpenTelemetry GenAI the SDK → Collector → backend pipeline was set up and Langfuse was treated as a black box that receives spans; in Prompt versioning with Langfuse and MLflow its prompt management layer was used. Here we open the box: what is inside Langfuse, why v3 stopped being a monolith on Postgres, and how it is operated so it holds the traffic of an inference cluster without becoming the problem.

TL;DR

Langfuse v3 (stable since December 2024) is not an application, it is six services: two containers of its own (Web and Worker) and four stateful dependencies (Postgres, ClickHouse, Redis/Valkey and an S3-compatible blob store). The key architectural change compared with v2, which was a Next.js monolith on Postgres, is the asynchronous ingestion pipeline: traces are received in batches, written immediately to S3, only a reference is queued in Redis, and a Worker ingests them into ClickHouse in the background. This decouples reception speed (limited only by Redis write latency, ~1-5 ms) from the cost of persisting and merging into the analytical database. The result: the Web container sustains hundreds of events per second without a spike blocking the client that serves the inference. But that design only performs with the right settings. This post covers the architecture, its interaction with the rest of the on-premise stack, and ten backend knobs, from batching to ClickHouse to queue sharding, from the FINAL modifier to system log table hygiene, that decide the real throughput and the storage cost. And it marks where the async design hides data loss windows worth knowing about before promising “full traceability”.

You are here: OBSERVE (and the layer that holds up the rest)

The analogy: the postal sorting centre

Picture the central mail sorting office of a large city at rush hour. Lorries loaded with sacks (batches of letters) arrive at a rate that does not stop. If the counter clerk had to open every sack, read every letter, decide its destination and file it before accepting the next lorry, the queue of lorries would go round the block in ten minutes. No serious sorting centre works like that.

What they do is decouple reception from processing:

The reception counter accepts the sack, stamps a receipt on it, leaves it in a pigeonhole in the warehouse and drops a ticket on a conveyor belt. Time per sack: seconds. The counter never blocks.
Further back, in the sorting room, a team of operators picks tickets off the belt, retrieves the sack from its pigeonhole, opens it, sorts the letters and files them in the permanent archive, ordered, indexed, searchable.

Langfuse v3 is exactly this sorting centre:

Postal centre	Langfuse v3	Function
Reception counter	Web container (ingestion endpoint)	Accepts event batches, gives an immediate receipt (HTTP 207)
Pigeonhole warehouse	S3 / Blob store (MinIO on-prem)	Stores the raw sack (the full event)
Ticket on the belt	Redis / Valkey (BullMQ queue)	Only the reference to the object in S3, not the content
Sorting room	Worker container	Takes tickets, reads S3, transforms and files
Indexed permanent archive	ClickHouse (OLAP)	Traces, observations and scores, queryable by project+time
Administrative register	Postgres (OLTP)	Users, projects, API keys, prompts, datasets, config

The thesis of the whole post follows from this analogy: the value of Langfuse lies in the counter never blocking the client that serves the inference. An observability tool that adds latency or outages to the token-serving path is worse than having no observability, because it degrades precisely the system it set out to look after. The whole of the v3 design, and every knob in this post, exists to keep that promise under load.

The mechanism itself: six services, two planes

Langfuse v3 separates two planes that in v2 were fused together:

Ingestion and query plane (the two containers of its own, stateless, horizontally scalable): Web and Worker.
State plane (four dependencies, each with its own load profile): Postgres (transactional OLTP), ClickHouse (analytical OLAP), Redis/Valkey (queue + cache), Blob store (raw objects).

What to take away from this diagram:

Web and Worker are interchangeable and stateless. They store nothing locally. You can run 1 or 20 replicas of each; the state lives in the four dependencies. This is what allows scaling by load without choreography.
Redis never carries the event content, only the reference to the object in S3. That is why Redis holds the spike: a Redis write is ~1-5 ms and moves bytes, not kilobytes. The bottleneck of the Web container is, literally, Redis write speed.
Postgres and ClickHouse have opposite profiles. Postgres is OLTP: many small transactional reads and writes (is this API key valid? which version does the production label point to?). ClickHouse is OLAP: few enormous batch writes and analytical queries over billions of rows (give me the p95 TTFT of project X over the last 7 days). Putting traces in Postgres, which is what v2 did, works until it does not: at production volume, Postgres drowns in a workload it was not designed for. That was the reason for the redesign.

The ingestion flow step by step (and the maths of the decoupling)

The heart of the design is the ingestion route. Seen in detail, a POST /api/public/ingestion request with a batch of events does this:

The mathematical point is the early ACK. The latency the client perceives when sending traces is:

$$ t_{\text{client}} = t_{\text{S3 write}} + t_{\text{Redis enqueue}} \approx 10\text{–}40\,\text{ms} $$

whereas the real cost of persisting, reading S3, transforming, merging against the previous version, inserting into ClickHouse, letting the background merges compact, happens off that path, in the Worker, and can take hundreds of ms or seconds without the client caring. The decoupling turns a system whose throughput would be limited by ClickHouse speed into one limited by Redis speed. And Redis, on modest hardware, sustains on the order of 50,000 operations/second.

This has an important sizing consequence. If your inference load generates $E$ events/second (a chat with RAG + 2 tool calls easily produces 6-10 spans = events per request), the Web container absorbs them while $E \ll 50{,}000$. The Worker, by contrast, scales with the cost of processing: that is the component to watch and replicate, and the first knob of the post.

Honest scepticism. The early ACK has a flip side: between the HTTP 207 and persistence in ClickHouse there is a potential loss window. If the event is in S3 and the reference in Redis, and Redis goes down without persistence (AOF/RDB) before the Worker processes it, the reference is lost; the data is still in S3 but nobody claims it any more. More subtly: the Worker buffers writes to ClickHouse in memory and flushes them in batches; a Worker crash with a full buffer loses that batch. There is a reported bug where the ClickhouseWriter discards rows after exhausting flush retries with no dead-letter queue. For observability this is usually tolerable (losing 0.01 % of the traces breaks nothing). For regulatory auditing, where the trace is evidence, it is not, and Langfuse is better treated as “best-effort” and not as a ledger. We will come back to this at the close.

Interaction with the rest of the stack: Langfuse in the example 4×H100 cluster

Langfuse does not live in isolation. In the seven-layer stack it occupies the LLM-aware observability layer, and it relates to almost every other one. On the generic reference cluster we use throughout the blog, 4×H100 SXM 80 GB (320 GB aggregate VRAM), NVLink, 640 GB of system RAM, NVMe-oF, 25/100 GbE network, the telemetry flow looks like this:

Three ideas from this topology:

Langfuse receives from the OTel Collector, not from the application directly (in the recommended pattern). The app SDK or vLLM emit spans with the gen_ai.* semantic conventions; the Collector does batch, tail-sampling (preserving 100 % of errors and high latencies, sampling the rest) and enriches with its own attributes (tenant_id, priority_tier); and it distributes: LLM traces go to Langfuse, infrastructure spans to Tempo, metrics (GPU DCGM, vLLM metrics) to Prometheus. Langfuse is one more exporter, not the only destination. This is covered in detail in the OTel tracing post.
Langfuse runs off the GPUs. It is a consumer of CPU, RAM, disk and network (ClickHouse wants memory, MinIO wants disk, Redis wants CPU for networking) but it does not touch VRAM. On the 4×H100 cluster, Langfuse lives on a CPU node (or on the GPU nodes but with nodeSelector/taints keeping it away from the vLLM pods). Mixing ClickHouse with vLLM on the same node without resource limits is asking for an ingestion spike to steal memory bandwidth from inference. Isolation by design.
The telemetry route is “cold” and the data route is “hot”. The data plane (left) serves tokens with a millisecond latency budget; the telemetry plane (right) tolerates seconds. The early ingestion ACK is what keeps these two clocks apart: the app does not wait for Langfuse to file anything before returning the response to the user.

The 10 backend knobs that move the needle most

These are, in approximate order of impact/frequency, the settings that decide whether your self-hosted Langfuse ingests 50 events/s or 5,000, and whether your disk grows sustainably or explodes in three weeks. They are all environment variables or config injected into the Web and Worker containers (except the ClickHouse ones, which go in its server-side config). The canonical detail is in the Langfuse scaling doc.

Knob 1 — Scale the Worker by load (the first lever, always)

The Worker is the component that saturates first, because it does the expensive work: reading S3, transforming, merging, inserting into ClickHouse. The Langfuse operational rule is simple: a 2-CPU Worker container above 50 % CPU use is saturated; add replicas. Better than CPU, the Worker publishes via statsd the metric langfuse.queue.ingestion.length (ingestion queue length), which is the direct signal for autoscaling: if the queue grows without draining, there are not enough Workers.

# The ideal autoscaler watches queue depth, not just CPU.
# (KEDA ScaledObject over the statsd metric → Prometheus)
triggers:
 - type: prometheus
 metadata:
 query: langfuse_queue_ingestion_length
 threshold: "10000" # if the queue goes past 10k refs, scale

In AWS deployments there is ENABLE_AWS_CLOUDWATCH_METRIC_PUBLISHING=true to push these metrics to CloudWatch. On-premise, the path is statsd → Prometheus → KEDA, slotting into the autoscaling on Kubernetes with KEDA we already covered for vLLM. Always start here: most “Langfuse is slow” problems are simply insufficient Workers, not fine tuning.

Knob 2 — Separate the ingestion deployment from the UI one

When ingestion is heavily loaded, UI queries and the public API become slow because they share the same Web container. The solution is to split langfuse-web into two identical deployments and route by path: everything under /api/public/ingestion*, /api/public/media* and /api/public/otel* goes to the ingestion deployment; the rest (UI, read API) to the interface one.

# Ingress / gateway rule
location ~ ^/api/public/(ingestion|media|otel) {
proxy_pass http://langfuse-web-ingest; # replicas dedicated to writing
}
location / {
proxy_pass http://langfuse-web-ui; # replicas dedicated to reading
}

It is the same idea as the read/write separation of any system with mixed loads: a storm of writes should not starve whoever is trying to look at the dashboard right in the middle of the incident, which is precisely when they need it most.

Knob 3 — Batching writes to ClickHouse (interval + batch size)

ClickHouse hates small, frequent inserts: every INSERT creates a part on disk that then has to be merged, and thousands of tiny inserts generate thousands of parts and a storm of background merges that saturates the disk. The defence is to accumulate in an in-memory buffer in the Worker and flush in large batches:

# Worker: fewer flushes, larger batches → fewer parts, fewer merges
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=1000 # raise e.g. to 2000-5000
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_BATCH_SIZE=10000 # raise if there is throughput

Raising the interval and the batch size reduces the flush frequency and improves sustained throughput. The trade-off is direct and has to be understood: larger, less frequent batches mean more data in the Worker’s volatile buffer, that is, a bigger loss window if the Worker goes down (a knob coupled to the scepticism at the close). Langfuse also uses ClickHouse async_insert, which accumulates server-side before confirming; it adds another layer of buffering to keep in mind.

Knob 4 — Skip the ClickHouse read-before-write in ingestion

By default, when ingesting an event the Worker reads the existing event from ClickHouse and merges it with the incoming one (necessary when legacy SDKs send partial events: a start, then an end, then an update of the same observation). That per-event read loads ClickHouse on the write path and limits total throughput.

If your projects are not migrated from an old version, because the full history already lives in S3, you can disable that read:

# Date earlier than the creation of your first project
LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE=2025-01-01

With modern Langfuse SDKs or with ingestion via OpenTelemetry, this does not affect you negatively and removes one read per event. A warning from the doc itself: if you combine this with aggressive deletion (lifecycle) rules in S3 plus late event updates, you can generate duplicates in the history. Know that before enabling it.

Knob 5 — Write concurrency to S3/Blob storage

In high-throughput scenarios, the S3 client can exhaust its sockets and start queueing and throttling writes. The symptom is unmistakable in the logs of the Web container that handles ingestion:

@smithy/node-http-handler:WARN - socket usage at capacity=150
and 387 additional requests are enqueued.

…accompanied by a rise in memory in that container (queued requests pile up in RAM). The cure is to raise the concurrent write limit from its default of 50:

LANGFUSE_S3_CONCURRENT_WRITES=100 # raise gradually from 50

Each additional socket has a small memory cost, so the official advice is to raise it gradually while observing the behaviour, not to jump to 1000 in one go.

Knob 6 — Redis queue sharding + concurrency per shard

If Redis goes past 90 % CPU, first the obvious: an instance with at least 4 CPUs (so Redis spreads networking and background tasks across different cores) and Redis Cluster mode enabled. If CPU still does not come down, you can shard the queues Langfuse uses:

# Advanced: only if Redis is genuinely drowning and you already did the above
LANGFUSE_INGESTION_QUEUE_SHARD_COUNT=6 # ~2-3× the number of Redis cluster shards
LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT=6
# Concurrency counts PER SHARD; target ~20 per worker
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=3 # 6 shards × ~3 ≈ 18
LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY=3

Two traps the doc underlines and that are worth tattooing: once you shard, do not reduce the number of shards (it breaks the distribution); and concurrency counts per shard, not globally, so if you have 10 shards and want concurrency 20 per worker, set 2, not 20. It is an advanced knob: most on-premise deployments never need it.

Knob 7 — The `FINAL` modifier for OTel-only projects

Langfuse stores observations in a ClickHouse ReplacingMergeTree and, by default, adds the FINAL modifier to API queries so the last version of each row wins at read time. FINAL is necessary when ingestion produces several versions of the same observation (legacy SDKs with their start/end/update events), but it adds merge work to every read and slows it down.

Projects that ingest exclusively over OpenTelemetry write each observation as a single immutable row, so FINAL is redundant for them:

# Recommended in mixed deployments: per-project, flagged in Redis with a 24h TTL
LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true

# Only if ALL projects are OTel-only: global, no Redis lookup
LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL=true

Since on the example cluster the instrumentation is 100 % OTel (gen_ai.* via Collector), this knob is free money in dashboard read latency. Careful with the global version: do not enable it if some project is still using legacy ingestion, or reads may return duplicate or stale rows.

Knob 8 — Separate analytical reads from the write path (compute-compute)

Heavy dashboard queries (percentiles over millions of spans) compete with ingestion inserts and with background merges on the same ClickHouse. If your deployment supports compute-compute separation (ClickHouse Cloud or BYOC), you can route reads to a read-only compute group:

CLICKHOUSE_URL=http://clickhouse-primary:8123 # writes, migrations, ingestion
CLICKHOUSE_READ_ONLY_URL=http://clickhouse-reader:8123 # UI + public API reads

Critical nuance for on-premise, and here it is time to be sceptical about the usefulness of this knob in our context: on a single-node ClickHouse or on a self-managed cluster without compute separation, this variable adds nothing, because the read endpoint would be the same as the write one. It is a knob for cloud architectures with storage separated from compute. On an on-premise 4×H100 cluster with ClickHouse on one node, the real alternative is to scale ClickHouse vertically (the doc recommends ≥16 GiB of RAM for large deployments; ClickHouse scales vertically well) and to make sure every query filters by projectId and time, which is how the tables are indexed. Without a time filter, even the fattest ClickHouse suffers.

Knob 9 — Data retention: TTL in ClickHouse + lifecycle in S3

Disk is the cost that grows on its own. LLM traces carry whole inputs and outputs (sometimes prompts of tens of KB), and ClickHouse also accumulates its own system tables. The first-order lever is a retention policy that nightly deletes traces, observations, scores and media older than N days, coordinating ClickHouse and blob storage. Where the retention feature is not available, it is done by hand:

-- ClickHouse: TTL over the tracing tables
ALTER TABLE traces MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE observations MODIFY TTL toDateTime(start_time) + INTERVAL 90 DAY;
ALTER TABLE scores MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE event_log MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY;

# S3/MinIO: lifecycle rule, e.g. 30 days for the raw event bucket
# WATCH OUT! Do NOT apply retention to the MEDIA bucket:
# - it breaks the files referenced in traces
# - it breaks future uploads (state is tracked by hash in Postgres)

Two operational parameters that avoid nasty surprises on large deletions:

LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS=600000 # default 10 min; raise it if deletions time out
# ClickHouse 25.7+: less mutation pressure on bulk deletions
CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE=lightweight_update
CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE=true

The mental rule: short retention for raw events (S3, 30 days is usually enough, they are recoverable/recomputable), retention by business value for the ClickHouse tables (90 days, 180, whatever compliance asks for), and never touch the media bucket with blind lifecycle rules.

Knob 10 — Hygiene of the ClickHouse system log tables (the silent disk killer)

This is the knob nobody configures and that fills the disk without showing up in any Langfuse metric, because it is not Langfuse data: they are the system tables of ClickHouse itself (trace_log, text_log, opentelemetry_span_log, asynchronous_metric_log, metric_log, latency_log). By default they have no TTL, and the query profiler writes to system.trace_log continuously. On a ClickHouse with traffic, these tables can dominate disk usage while you look for the problem in your traces. Langfuse does not read from them, so they can be trimmed without fear. Two options:

<!-- Option A — disable the ones Langfuse never reads
 (file in /etc/clickhouse-server/config.d/) -->
<clickhouse>
 <trace_log remove="1"/>
 <text_log remove="1"/>
 <opentelemetry_span_log remove="1"/>
 <asynchronous_metric_log remove="1"/>
 <metric_log remove="1"/>
 <latency_log remove="1"/>
</clickhouse>
<!-- Keep query_log, part_log and error_log: useful for debugging and small -->

-- Option B — aggressive TTL + turn off the profiler, if you want to keep them for debugging
-- (in config: query_profiler_real_time_period_ns = 0)
SET max_table_size_to_drop = 0;
TRUNCATE TABLE system.trace_log;
ALTER TABLE system.trace_log MODIFY TTL event_date + INTERVAL 7 DAY;
-- repeat for each log table to cap

To identify which table is eating the disk, the golden query:

SELECT table, formatReadableSize(sum(bytes)) AS size, sum(rows) AS rows
FROM system.parts WHERE active GROUP BY table ORDER BY sum(bytes) DESC;

If you take only one knob from this post to your first real deployment, make it this one: the difference between a ClickHouse that grows 2 GB/day of useful data and one that grows 20 GB/day of system logs nobody looks at.

Summary table of the 10 knobs

#	Knob	Variable / action	When
1	Scale Worker	replicas by CPU>50 % / `langfuse.queue.ingestion.length`	always, first
2	Separate ingestion/UI	route `/ingestion`,`/media`,`/otel*` to a dedicated replica	slow UI under load
3	Batching to ClickHouse	`LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS` / `_BATCH_SIZE`	high throughput
4	Skip CH read-before-write	`LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE`	non-migrated projects
5	S3 concurrency	`LANGFUSE_S3_CONCURRENT_WRITES` (def. 50)	“socket usage at capacity”
6	Redis queue sharding	`LANGFUSE__QUEUE_SHARD_COUNT` + `_CONCURRENCY` (per shard)	Redis CPU >90 %
7	Remove `FINAL` (OTel)	`LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true`	100 % OTel instrumentation
8	CH read/write split	`CLICKHOUSE_READ_ONLY_URL` (cloud/BYOC only)	compute-compute available
9	Retention + TTL	TTL in CH + S3 lifecycle + `LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS`	always (disk cost)
10	CH system log hygiene	`<trace_log remove="1"/>` or aggressive TTL	always (hidden disk)

How to maximise Langfuse on the example 4×H100 cluster

With the architecture and the knobs clear, this is a concrete sizing to get the most out of Langfuse on the generic reference cluster (4×H100 SXM, 320 GB VRAM, 640 GB RAM, NVMe-oF, 25/100 GbE), without stealing a single GB of VRAM from inference.

Component distribution

Langfuse is 100 % CPU/RAM/disk/network load, so its natural place is off the GPU nodes or, if it cohabits, with taints/nodeSelector confining it away from the vLLM pods. Suggested distribution:

nodo-cpu-01 (control + observability, no GPU)
├── langfuse-web-ingest ×3 (2 CPU / 4 GiB each) ← ingestion, scales with load
├── langfuse-web-ui ×2 (2 CPU / 4 GiB each) ← dashboard/read API
├── langfuse-worker ×4 (2 CPU / 4 GiB each) ← the one that scales most
├── redis/valkey ×1 (4 CPU / 4 GiB, cluster mode)
└── postgres ×1 (2 CPU / 8 GiB, replica for HA)
nodo-storage-01 (heavy state, local NVMe)
├── clickhouse ×1 (8 CPU / 32 GiB / NVMe) ← ≥16 GiB is the minimum; 32 is comfortable
└── minio (S3) ×1 (4 CPU / 8 GiB / HDD+NVMe cache)
nodo-gpu-01..02 (4×H100 SXM each) → inference ONLY
└── vLLM, embeddings, reranker, guardrails (emit spans, do not host Langfuse)

Sizing by real load

Let us put numbers on an example load. Suppose the cluster serving 300 requests/second of chat-with-RAG, where each request generates on the order of 8 spans (request, retrieval, rerank, 2× tool, guardrail in, llm, guardrail out):

$$ E = 300\,\tfrac{\text{req}}{\text{s}} \times 8\,\tfrac{\text{spans}}{\text{req}} = 2,400\ \text{events/s} $$

Against the Redis ceiling (~50,000 ops/s), $E = 2{,}400$ leaves the reception counter at ~5 % of its capacity: enormous headroom. The component to watch is the Worker. With a target of ~20 concurrency per Worker and batches of 10,000 events every ~1-2 s, 4 Workers drain 2,400 ev/s with margin; the langfuse.queue.ingestion.length metric should stay flat near zero. If it grows, knob 1 (more Workers) is the answer before any fine tuning.

Tail-sampling is the multiplier that changes the economics. If the Collector preserves 100 % of errors/high latencies but samples normal traffic at, say, 10 %, the 2,400 ev/s you store in ClickHouse drop to ~240-300 effective ev/s without losing the signal that matters. The rule: sample in the Collector, not in Langfuse. Langfuse should receive, already filtered, what deserves to be persisted. This is developed in the OTel tracing post; here it is enough to note that upstream sampling is, in effect, the knob 0 that multiplies all the others.

Storage estimate

An LLM observation with full input+output weighs, compressed in ClickHouse, on the order of 1-3 KB (ClickHouse compresses text very well, 5-10×). With sampling at 10 % over 2,400 ev/s:

$$ 240\,\tfrac{\text{ev}}{\text{s}} \times 2\,\text{KB} \times 86,400\,\tfrac{\text{s}}{\text{day}} \approx 41\ \text{GB/day (raw)} \;\xrightarrow{\text{compression}}\; \sim 5\text{–}8\ \text{GB/day in CH} $$

At 90 days of retention (knob 9), the permanent archive settles at around 500-700 GB in ClickHouse, comfortable on the NVMe of the storage node, plus the raw events in MinIO with a 30-day lifecycle. Without the system log hygiene (knob 10), add easily as much again in rubbish nobody queries. The two disk knobs together are the difference between planning storage once a year and fighting a full disk every month.

“Maximum use” checklist

Sampling in the Collector (tail: 100 % errors + N % normal), before touching anything in Langfuse.
Workers scaled by queue length via KEDA (knob 1), not fixed.
Ingestion separated from UI (knob 2) so the dashboard responds during incidents.
SKIP_FINAL_FOR_OTEL_PROJECTS enabled (knob 7) because the instrumentation is 100 % OTel.
Generous CH batching (knob 3) tuned to the throughput, accepting the loss window.
Retention + TTL + system log hygiene (knobs 9 and 10) configured on day 1, not when the disk screams.
ClickHouse with ≥16 GiB and every query filtering by projectId+time (knob 8 in its on-premise version: vertical scaling).
Langfuse isolated from the GPUs by taints/nodeSelector: not a MB of VRAM, no memory bandwidth contention with vLLM.

Traps and things that are not what they look like

“Langfuse guarantees me full traceability.” No: the design is high-performance best-effort, not a ledger. Between the HTTP 207 and the row in ClickHouse there are volatile buffers (Redis without hard persistence, the Worker’s in-memory buffer, ClickHouse’s server-side async_insert). There is a known bug where the writer discards rows with no dead-letter queue after exhausting retries. For operational observability, losing 0.01 % of spans is irrelevant. For ENS/EU AI Act audit evidence, where the trace is the proof, Langfuse should not be the only record; the regulatory audit log needs durability guarantees this pipeline does not promise. A distinction covered in the technical controls for ENS/42001/EU AI Act.

Raising ClickHouse batching “to go faster” and nothing else. Knob 3 improves throughput at the cost of enlarging the loss window and the latency of data appearing in the dashboard. Batches of 50,000 every 10 s perform wonderfully… until the Worker restarts with 50,000 events in the buffer. Tune with awareness of the trade-off, not by blind maximisation.

Putting ClickHouse on the same node as vLLM with no limits. ClickHouse is voracious with memory bandwidth during merges. Sharing a node with vLLM without resources.limits or NUMA isolation means an ingestion spike can degrade inference TTFT, exactly the original sin this whole architecture wanted to avoid. Isolate.

Forgetting the time filter in your own queries. The ClickHouse tables are indexed by projectId and time. A custom dashboard or an API query without a time filter scans the whole history and takes performance down for everyone. It is not Langfuse “being slow”: it is a badly written query.

Applying lifecycle to the media bucket. It breaks the files referenced in traces and blocks future uploads (state is tracked by hash in Postgres). The media bucket is managed only with the Langfuse retention feature, never with blind S3 rules.

Treating queue sharding as a routine optimisation. It is an advanced knob for a genuinely drowning Redis, irreversible (do not reduce shards) and with per-shard concurrency semantics that are easy to misread. In the vast majority of on-premise deployments it is not needed; if you enable it “just in case”, you complicate your life without gaining anything.

Conclusion

Langfuse v3 solved the structural problem of LLM observability, that the observer should not suffocate the observed, by moving from a monolith on Postgres to a six-service sorting centre with asynchronous ingestion. That design is what allows a cluster serving thousands of tokens per second to be instrumented end to end without the app ever waiting for a trace to be filed. But the design is a necessary condition, not a sufficient one: it performs if the right levers are tuned. Of the ten knobs, three decide almost everything in a typical on-premise deployment, scaling Workers by queue length (1), retention + TTL (9), and system log hygiene (10); the rest are refinements that show up when the load bites. And above them all lives knob 0, which is not a Langfuse one: sampling in the Collector, which decides how much reaches the pipeline before any internal setting matters. Maximising Langfuse on the 4×H100 cluster is not about squeezing its peak throughput: it is about putting it off the GPUs, feeding it already-sampled traffic, sizing the Worker by the queue, and configuring retention on day one, so that the tool that came to tell the story does not end up being the chapter about the incident.

References

Langfuse, Scaling Langfuse Deployments (doc oficial de sizing y todos los env vars de este post): https://langfuse.com/self-hosting/configuration/scaling.
Langfuse, Self-host Langfuse y Configuration via Environment Variables: https://langfuse.com/self-hosting · https://langfuse.com/self-hosting/configuration.
Langfuse, ClickHouse (self-hosted): https://langfuse.com/self-hosting/deployment/infrastructure/clickhouse.
Langfuse, From Zero to Scale: Langfuse’s Infrastructure Evolution (el porqué del rediseño v2→v3): https://langfuse.com/blog/2024-12-langfuse-v3-infrastructure-evolution.
Langfuse, Migrate v2 to v3 (self-hosted): https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v2-to-v3.
ClickHouse, Langfuse and ClickHouse: A new data stack for modern LLM applications: https://clickhouse.com/blog/langfuse-and-clickhouse-a-new-data-stack-for-modern-llm-applications.
Langfuse, issue #13468 — ClickhouseWriter drops rows after max flush attempts with no DLQ (la ventana de pérdida documentada): https://github.com/langfuse/langfuse/issues/13468.
ClickHouse, TTL for tables and columns: https://clickhouse.com/docs/guides/developer/ttl.
OpenTelemetry, Semantic Conventions for Generative AI (gen_ai.*): https://opentelemetry.io/docs/specs/semconv/gen-ai/.

Self-Hosting on lo0 — Tech Blog

Langfuse v4, day 2 (5 of 10): the worker's thirty-nine queues, and the three that decide whether you lose data

TL;DR

You are here: OBSERVE, day 2

The analogy: the sorting hub

The map of the thirty-nine

The sharding that does not shard

Concurrencies, and the ones set to one on purpose

An event’s path, from API to ClickHouse

Retries, and the bin nobody looks at

The metrics, with their exact names

The two probes, and the one that can restart you in a loop

The crons

Splitting roles, which is what all this is for

Checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

TL;DR

You are here: OBSERVE, day 2

The analogy: changing the track gauge without closing the line

The three-rung ladder

The three combinations that refuse to start

What breaks in each mode

The three steps, in order

The ClickHouse migrations backfill nothing

The five-step chain that does backfill history

The failure that does not block

Where the ten minutes come from

The forty-eight-hour TTL

What can and cannot be undone

How this looks in code

Migration checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (3 of 10): two well-known agents instrumented, Open Deep Research and GPT Researcher

TL;DR

You are here: OBSERVE, day 2

The analogy: two factories, one audit

The two agents, and why these

Open Deep Research: the shape of the graph

What one request costs, measured

Zero AGENT observations

Instrumenting it without touching the repository

GPT Researcher: two products in one repository

The myth of the fresh config

The standard mode: eight calls, eight traces

The human node that stays open

What to look at in the trace once it arrives

Checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

TL;DR

You are here: OBSERVE, day 2

The analogy: the file that gets photocopied whole at every signature

Which agentic platform is the most used

How LangGraph gets into Langfuse

The observation type is decided by the node name

What a turn costs, measured

What this means for ClickHouse

Interrupts: two traces where there should be one

Thread, session, user, and what is only read at the root

The filter that drops your own spans

The model and the cost with a gateway in the middle

How this looks in code

Agent instrumentation checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (1 of 10): the data model changed underneath, and what really goes into a trace

TL;DR

You are here: OBSERVE, day 2

Knob 7 — The `FINAL` modifier for OTel-only projects