Langfuse v4, day 2 (5 of 10): the worker's thirty-nine queues, and the three that decide whether you lose data

Mon, 14 Sep 2026 08:30:00 +0200

Fifth article in a series of ten about operating Langfuse v4 in production. The previous one covered migrating from version 3. This one goes inside the worker that is already running. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

Thirty-nine queues and thirty-two switches. The catalogue is in packages/shared/src/server/queues.ts:406. The QUEUE_CONSUMER_*_IS_ENABLED variables are read once at boot, so turning a queue off is a restart, not a hot change.

Sharding spreads nothing without Redis Cluster. The producer only applies the hash if REDIS_CLUSTER_ENABLED === "true", and that variable ships "false". Raising LANGFUSE_INGESTION_QUEUE_SHARD_COUNT on an installation with single-node Redis starts N workers and leaves N−1 with no work.

The dead-letter retrier ships disabled. QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED is the only one of the thirty-two whose default is "false".

And even switched on, it does not cover ingestion. Its list holds five queues and none is ingestion-queue. An event that exhausts its six attempts sits in the failed set.

The ClickHouse writer drops rows. Three attempts with a flat hundred-millisecond delay and then out. The code comment is explicit: // TODO - Add to a dead letter queue in Redis rather than dropping. The metric to watch is called langfuse.queue.clickhouse_writer.rows_dropped.

Turn a queue’s consumer off and you stop seeing its depth. The metrics emitter only polls queues with a worker registered in that container. The queue keeps growing and the dashboard goes flat.

Project deletion runs one job every ten minutes. By limiter, deliberately, because each one scours ClickHouse.

There are two optional health probes and one of them can cause the very restart loop it is meant to fix if configured with less than sixty seconds of initial delay.

You are here: OBSERVE, day 2

The previous article left the installation migrated. This is the map of the process that does the work, and it comes before the capacity article because the queues are where saturation shows up before ClickHouse notices.

The analogy: the sorting hub

A postal sorting hub has separate belts for parcels, registered mail, returns and internal pouches. Each belt has its speed and its staff, and some are throttled on purpose because the destination cannot take more.

The job of whoever runs it is not to watch the fastest belt, it is to know which one jams first, which can stop for an afternoon without anyone noticing and which cannot stop for ten minutes. And above all, where the bin for items nobody could sort is, because every hub has one and in many of them nobody looks at it.

The Langfuse worker has thirty-nine belts. This article is the plan, with the speeds and the location of the bin.

The map of the thirty-nine

Grouped by what they do, with what stops happening if the group stops:

Ingestion (4). ingestion-queue, secondary-ingestion-queue, otel-ingestion-queue, secondary-otel-ingestion-queue. If they stop, the web keeps accepting events and uploading them to object storage, and keeps enqueueing. Nothing reaches ClickHouse. The SDK sees no error at all. This is the group that cannot stop.

Propagation (1). event-propagation-queue, the dual-write one from the previous article, with a global concurrency of one across the cluster.

Evaluation (8). trace-upsert, dataset-run-item-upsert-queue, create-eval-queue, evaluation-execution-queue and its secondary, llm-as-a-judge-execution-queue, code-eval-execution-queue, experiment-create-queue. If they stop, evaluations are neither created nor executed. Traces keep arriving. This is the group that can wait.

Deletion and retention (7). trace-delete, score-delete, dataset-delete-queue, project-delete, data-retention-queue and its processing queue, batch-action-queue. If they stop, neither UI-requested deletions nor per-project retention run. That is not an operational annoyance, it is a breach if you have a retention commitment in writing.

Exports (3). batch-export-queue, core-data-s3-export-queue, metering-data-postgres-export-queue. Downloads a user asked for stay pending.

Outbound integrations (8). The PostHog, Mixpanel and blob storage pairs, plus webhook-queue, entity-change-queue and notification-queue. In each pair, the bare queue is the scheduler that fans out per project and the processing one does the work.

Cloud billing (4). They only register with a Stripe key. On a self-hosted installation they never start.

Maintenance (4). dead-letter-retry-queue, monitor-queue, in-app-agent-run-queue and v4-legacy-api-usage-queue, which ingests nothing: it is a cron that scans ClickHouse’s system.query_log to find which projects still use the old API.

The switches number thirty-two, all QUEUE_CONSUMER_*_IS_ENABLED and all "true" except the dead-letter retrier. Four queues have no switch of their own: the model-as-judge one rides inside the evaluation switch, the two Cloud data export queues have their own variable, and the processing queues share a switch with their scheduler.

The mechanism is a top-level if around each worker registration. A false switch means the worker is never instantiated, not that the queue stops receiving work: producers keep enqueueing regardless. Turning a queue off is not consuming it, not not filling it.

The sharding that does not shard

Nine queues support sharding: the four ingestion ones, three evaluation ones, model-as-judge and trace-upsert. Each has its own shard-count variable, and they all default to 1.

Raising them looks like the obvious first move to scale ingestion. It is not, because of this (packages/shared/src/server/redis/ingestionQueue.ts:48):

 const shardIndex =
 IngestionQueue.getShardIndexFromShardName(shardName) ??
 (env.REDIS_CLUSTER_ENABLED === "true" && shardingKey
 ? getShardIndex(shardingKey, env.LANGFUSE_INGESTION_QUEUE_SHARD_COUNT)
 : 0);

The : 0 at the end is the whole story. The producer only computes a shard if Redis is in cluster mode, and REDIS_CLUSTER_ENABLED ships "false". On an installation with single-node Redis, which is most self-hosted installations, raising the shard count to four starts four workers, three of which stare at empty queues, while all traffic keeps coming through ingestion-queue.

The sharding key, when it does apply, is projectId-eventBodyId run through SHA-256, taking the first eight hex characters and a modulo. Shard names are ingestion-queue for zero and ingestion-queue-N for the rest.

Without Redis Cluster, the lever that does work is concurrency, which is a different thing.

Concurrencies, and the ones set to one on purpose

The defaults, worth knowing because several sit at one for good reasons:

Queue	Variable	Default
`ingestion-queue` (per shard)	`LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	20
`secondary-ingestion-queue`	`LANGFUSE_INGESTION_SECONDARY_QUEUE_PROCESSING_CONCURRENCY`	5
`otel-ingestion-queue`	`LANGFUSE_OTEL_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	5
`trace-upsert`	`LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY`	25
`monitor-queue`	`LANGFUSE_MONITOR_QUEUE_PROCESSING_CONCURRENCY`	10
evaluation and model-as-judge	several	5
`create-eval-queue`	`LANGFUSE_EVAL_CREATOR_WORKER_CONCURRENCY`	2
`trace-delete`, `score-delete`, `dataset-delete-queue`, `project-delete`	several	1
`event-propagation-queue`	fixed in code	1 cluster-wide

Deletions sit at one and carry a rate limiter on top. The project one is the most aggressive (worker/src/app.ts:234):

 limiter: {
 // Process at most `max` delete jobs per LANGFUSE_CLICKHOUSE_PROJECT_DELETION_CONCURRENCY_DURATION_MS (default 10 min)
 max: env.LANGFUSE_PROJECT_DELETE_CONCURRENCY,

One project deletion every ten minutes. If a customer asks you to delete forty projects, that is six hours and forty minutes on the clock, and rightly so, because each of those deletions is a broom sweeping through ClickHouse.

The ingestion queues, by contrast, have no limiter at all. There the brake is concurrency and the ClickHouse writer.

One BullMQ detail that recurs across queues and explains a good number of other people’s stalls (worker/src/app.ts:261):

 // The default lockDuration is 30s and the lockRenewTime 1/2 of that.
 // We set it to 60s to reduce the number of lock renewals and also be less sensitive to high CPU wait times.
 lockDuration: 60000, // 60 seconds
 stalledInterval: 120000, // 120 seconds
 maxStalledCount: 3,

An event’s path, from API to ClickHouse

It is worth following it end to end once, because there are three places it can be lost.

The web groups the batch’s events by body id, uploads one JSON file per group to object storage and aborts if that upload fails: no blob, no enqueue. Then it enqueues one job per body id on ingestion-queue.

The worker picks up the job, optionally checks a recently-processed cache in Redis (off by default, five-minute lifetime), decides whether to redirect the project to the secondary queue, and downloads the file from storage. There is a comment here worth a capacity chart on its own:

Process files in batches. If a user has 5k events, this will likely take 100 seconds.

Then it merges and writes. And writing is not writing: the service does not talk to ClickHouse, it puts the row in a buffer whose defaults are 1,000 rows, 1,000 milliseconds and 3 attempts.

That buffer is the third place things get lost, and the most serious one in the system (worker/src/services/ClickhouseWriter/index.ts:535):

 // Re-add the records to the queue with incremented attempts
 let droppedCount = 0;
 queueItems.forEach((item) => {
 if (item.attempts < this.maxAttempts) {
 entityQueue.push({ ...item, attempts: item.attempts + 1 });
 } else {
 // TODO - Add to a dead letter queue in Redis rather than dropping
 recordIncrement("langfuse.queue.clickhouse_writer.error");
 droppedCount++;
 }
 });

Once the three attempts are spent, the row is dropped. And no queue retry saves it, because the BullMQ job already completed successfully: as far as the queue is concerned, that event was processed. The internal retries are not properly exponential either: the delay is flat at a hundred milliseconds, so all three attempts happen within three hundred milliseconds and a ClickHouse that is unwell for half a second takes the batch with it.

The runbook consequence is direct. langfuse.queue.clickhouse_writer.rows_dropped has to be on a dashboard and it has to have an alert at zero. Any non-zero value is silent data loss.

Retries, and the bin nobody looks at

Each queue defines its own policy. The ones that matter:

Queue	Attempts	Backoff
`ingestion-queue`	6	exponential, 5 s
`secondary-ingestion-queue`	5	exponential, 5 s
`evaluation-execution-queue`	10	exponential, 1 s
`code-eval-execution-queue`	3	exponential, 30 s
`trace-upsert`	2	exponential, 5 s, with a 30 s initial delay
`trace-delete`, `score-delete`	2	exponential, 30 s
`monitor-queue`, `in-app-agent-run-queue`	1	no backoff, deliberately

The two single-attempt ones are a design decision, and the comment explains it well: “Postgres owns correctness (claim CAS + reconcile-on-read); BullMQ is delivery-only, so never redeliver on its own.”

And now the bin. There is no dedicated dead letter queue. What exists is BullMQ’s native failed set per queue, with its own retention, plus a queue that retries from there. That queue has two problems for whoever operates it:

First, it ships disabled. It is the only one of the thirty-two defaulting to "false".

Second, even switched on, its scope is hand-written and has five entries (worker/src/services/dlq/dlqRetryService.ts:9):

 private static retryQueues = [
 QueueName.ProjectDelete,
 QueueName.TraceDelete,
 QueueName.ScoreDelete,
 QueueName.BatchActionQueue,
 QueueName.DataRetentionProcessingQueue,
 ] as const;

ingestion-queue is not there. Ingestion events that exhaust their six attempts sit in the failed set, up to a hundred thousand of them, and nobody ever retries them. To get them out there is an admin route, POST /api/admin/bullmq with action retry, which the file itself describes as the one the managed service uses for exactly this.

The metrics, with their exact names

Each queue’s metric name is built at runtime from the queue name, turning dashes into underscores and stripping the _queue suffix. So ingestion-queue produces langfuse.queue.ingestion, and secondary-ingestion-queue produces langfuse.queue.secondary_ingestion. Sharded queues always add a shard tag.

What comes out of that:

langfuse.queue.<queue>.rate with type in request, completed, failed, error and stalled.
langfuse.queue.<queue>.time_distribution with type in wait and processing.
langfuse.queue.<queue>.depth with type in waiting, failed and active, plus an aggregate series with shard: "all".
langfuse.queue.<queue>.dlq_oldest_age, the age of the oldest failed job, which emits zero when the set empties so the dashboard resets.
From the writer: langfuse.queue.clickhouse_writer.rows_dropped, .error, .processing_time and ingestion_clickhouse_insert_queue_length.

Two things to know about those metrics. The waiting depth includes paused jobs, to match BullMQ’s counter. And the big trap (worker/src/features/queue-metrics-runner/index.ts:70):

 // Only poll queues that have registered workers. This avoids calling
 // getInstance() on queues this worker doesn't consume, which would
 // create unnecessary Redis connections and can trigger side effects

Only queues that container consumes emit metrics. If you split roles and turn a queue’s consumer off across every container, that queue stops appearing on dashboards while it keeps growing in Redis. It is the perfect way not to find out.

The metric that saturates first is still the same one: langfuse.queue.ingestion.depth with type: "waiting". The emission interval defaults to one second.

The two probes, and the one that can restart you in a loop

The worker exposes /api/health and /api/ready, identical but for one thing: the second returns 500 after SIGTERM, which is how draining works. Both always check Postgres with a SELECT 1 and Redis with a ping capped at two seconds, hardcoded.

Then there are two optional checks enabled by query parameter:

?failIfEventPropagationStuck=true looks at the propagator’s heartbeat and returns 503 if it has not been refreshed for more than 35 minutes. A missing heartbeat does not count as stuck, deliberately, so a freshly booted container is not restarted in a loop. And the condition that must be respected:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

?failIfQueueConsumptionStuck=true is an in-memory signal of that container, with no Redis involved: it is marked on the active and completed events of any worker. It returns 503 if that container has gone 60 minutes without picking up or finishing a single job. It exists for a very specific failure mode:

After Redis lock loss BullMQ workers can wedge permanently: the process stays alive and connectivity checks pass, but no queue picks up jobs ever again.

The one-hour threshold leans on the default crons keeping a healthy worker busy at least once an hour. In multi-replica deployments each scheduler tick lands on a single replica, so the comment itself warns about raising the threshold if a replica can legitimately sit idle.

Draining, by the way, has no cap. The order is close the HTTP server, stop the periodic runners, close the BullMQ workers and then flush the ClickHouse writer, which on shutdown empties the whole queue rather than one batch. The only thing bounding that drain is the Kubernetes grace period, so it had better be generous.

The crons

Queue	Pattern	What it does
`event-propagation-queue`	`* * * * *`	propagates one staging partition
`dead-letter-retry-queue`	`0 /10 * * *`	retries failed jobs from five queues
`v4-legacy-api-usage-queue`	`/15 * * *`	scans `system.query_log` for old-API usage
`blobstorage-integration-queue`	`/20 * * *`	fans exports out per project
`cloud-usage-metering-queue`	`5 * * * *`	Cloud billing
`posthog-integration-queue`, `mixpanel-integration-queue`	`30 * * * *`	hourly fan-out
`metering-data-postgres-export-queue`	`30 2 * * *`	daily export
`core-data-s3-export-queue`, `data-retention-queue`	`15 3 * * *`	daily export and retention

And a warning the scheduler file itself writes in capitals: cron registration is fire-and-forget from the queue constructors, and a failure is only logged. A transient Redis failure at boot can leave a cron unscheduled until the next restart, without the container ever being marked unhealthy.

Splitting roles, which is what all this is for

With thirty-two switches, what you can build is one worker per family. The split that makes sense on an installation with agentic load:

# Pool 1: ingestion. This is the one that scales with traffic.
QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_OTEL_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED=true # note: global concurrency 1
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=20
# everything else false

# Pool 2: evaluations. Scales with the number of evaluators, not with traffic.
QUEUE_CONSUMER_EVAL_EXECUTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_CREATE_EVAL_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_TRACE_UPSERT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EXPERIMENT_CREATE_QUEUE_IS_ENABLED=true

# Pool 3: maintenance. One replica is enough.
QUEUE_CONSUMER_TRACE_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_PROJECT_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DATA_RETENTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED=true # switch it on

Two warnings about this split. Each worker registration opens its own Redis connection, so the connection count grows with queues times shards times replicas, and that is checked against Redis maxclients before multiplying replicas. And the moment you split roles you lose the metrics of the queues that container does not consume, so some replica needs to consume every queue or you go blind on that one.

Checklist

There is an alert on langfuse.queue.clickhouse_writer.rows_dropped, threshold zero.
There is an alert on the depth of langfuse.queue.ingestion with type: "waiting".
There is an alert on dlq_oldest_age for the queues that matter, and somebody knows how to use POST /api/admin/bullmq.
The dead-letter retrier is on in some replica, knowing it does not cover ingestion.
Nobody raised the shard count without Redis Cluster.
The optional health probes are configured with initialDelaySeconds of at least 60 seconds.
The Kubernetes termination grace period allows the ClickHouse writer to flush.
Every queue is consumed by at least one replica, so its metrics are not lost.
Redis maxclients copes with queues times shards times replicas.

Traps

Sharding without Redis Cluster does not spread load. It starts the workers and leaves them idle.

The dead-letter retrier ships disabled and does not cover ingestion. Two independent facts, both uncomfortable.

The ClickHouse writer drops rows after three attempts. With a TODO in the code admitting it and three hundred milliseconds of total headroom.

Turning a consumer off also turns its metrics off. The queue keeps growing, the dashboard stays flat.

A cron can end up unscheduled while the container stays healthy. A transient Redis failure at boot, and it is gone until the next restart.

The PostHog and blob storage integrations disable themselves when an error is classified as a customer-config fault, and if the notification also fails they stay off and silent.

A misconfigured propagation probe restarts in a loop. Less than sixty seconds of initial delay and you get the cycle.

Draining has no cap of its own. The Kubernetes grace period bounds it and nothing else.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues (this article): the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/server/queues.ts, packages/shared/src/server/redis/, worker/src/env.ts, worker/src/app.ts, worker/src/queues/, worker/src/services/ClickhouseWriter/, worker/src/services/dlq/, worker/src/features/health/, worker/src/features/queue-metrics-runner/, worker/src/utils/shutdown.ts.
Langfuse self-hosting documentation, checked on 14 September 2026.
BullMQ: going to production, checked on 14 September 2026.

Bullmq on lo0 — Tech Blog