--- title: "Langfuse from the inside: the sorting centre that must not become the bottleneck it came to observe" url: https://blog.lo0.es/en/posts/langfuse-inside-sorting-centre-bottleneck/ markdown: https://blog.lo0.es/en/posts/langfuse-inside-sorting-centre-bottleneck/index.md lang: en date: 2026-06-06 lastmod: 2026-09-13 author: "José Manuel Román" description: "Langfuse v3 is not 'a trace database': it is six services (Web, Worker, Postgres, ClickHouse, Redis/Valkey, S3/Blob) joined by an asynchronous ingestion pipeline. This post takes the architecture apart with diagrams, how it fits with vLLM + OTel Collector + gateway on a generic 4×H100 SXM cluster, the 10 backend knobs that move the needle most on performance and retention, and how to size it so the observer never suffocates the observed." tags: ["langfuse","observability","self-hosting","clickhouse","redis","opentelemetry","llmops","on-premise","fundamentals"] translations: es: https://blog.lo0.es/posts/langfuse-self-hosting-arquitectura-tuning/index.md --- # Langfuse from the inside: the sorting centre that must not become the bottleneck it came to observe > This post closes a trilogy of the **Observe** layer: in [LLM tracing with OpenTelemetry GenAI](https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/index.md) the `SDK → Collector → backend` pipeline was set up and Langfuse was treated as a black box that receives spans; in [Prompt versioning with Langfuse and MLflow](https://blog.lo0.es/en/posts/prompt-versioning-contract-five-word-change/index.md) its prompt management layer was used. Here we open the box: what is inside Langfuse, why v3 stopped being a monolith on Postgres, and how it is operated so it holds the traffic of an inference cluster without becoming the problem. ## TL;DR Langfuse v3 (stable since December 2024) **is not an application, it is six services**: two containers of its own (Web and Worker) and four stateful dependencies (Postgres, ClickHouse, Redis/Valkey and an S3-compatible blob store). The key architectural change compared with v2, which was a Next.js monolith on Postgres, is the **asynchronous ingestion pipeline**: traces are received in batches, written *immediately* to S3, only a *reference* is queued in Redis, and a Worker ingests them into ClickHouse in the background. This decouples reception speed (limited only by Redis write latency, ~1-5 ms) from the cost of persisting and merging into the analytical database. The result: the Web container sustains hundreds of events per second without a spike blocking the client that serves the inference. But that design only performs with the right settings. This post covers the architecture, its interaction with the rest of the on-premise stack, and **ten backend knobs**, from batching to ClickHouse to queue sharding, from the `FINAL` modifier to system log table hygiene, that decide the real throughput and the storage cost. And it marks where the async design hides data loss windows worth knowing about before promising "full traceability". ## You are here: OBSERVE (and the layer that holds up the rest) > **Diagram:** you are here: Observe — [see the HTML version](https://blog.lo0.es/en/posts/langfuse-inside-sorting-centre-bottleneck/) ## The analogy: the postal sorting centre Picture the central mail sorting office of a large city at rush hour. Lorries loaded with **sacks** (batches of letters) arrive at a rate that does not stop. If the counter clerk had to **open every sack, read every letter, decide its destination and file it** before accepting the next lorry, the queue of lorries would go round the block in ten minutes. No serious sorting centre works like that. What they do is **decouple reception from processing**: 1. The **reception counter** accepts the sack, stamps a receipt on it, leaves it in a **pigeonhole** in the warehouse and drops a **ticket** on a conveyor belt. Time per sack: seconds. The counter never blocks. 2. Further back, in the **sorting room**, a team of operators picks tickets off the belt, retrieves the sack from its pigeonhole, opens it, sorts the letters and files them in the **permanent archive**, ordered, indexed, searchable. Langfuse v3 is exactly this sorting centre: | Postal centre | Langfuse v3 | Function | |---|---|---| | Reception counter | **Web** container (ingestion endpoint) | Accepts event batches, gives an immediate receipt (HTTP 207) | | Pigeonhole warehouse | **S3 / Blob store** (MinIO on-prem) | Stores the raw sack (the full event) | | Ticket on the belt | **Redis / Valkey** (BullMQ queue) | Only the *reference* to the object in S3, not the content | | Sorting room | **Worker** container | Takes tickets, reads S3, transforms and files | | Indexed permanent archive | **ClickHouse** (OLAP) | Traces, observations and scores, queryable by project+time | | Administrative register | **Postgres** (OLTP) | Users, projects, API keys, prompts, datasets, config | The thesis of the whole post follows from this analogy: **the value of Langfuse lies in the counter never blocking the client that serves the inference**. An observability tool that adds latency or outages to the token-serving path is worse than having no observability, because it degrades precisely the system it set out to look after. The whole of the v3 design, and every knob in this post, exists to keep that promise under load. ## The mechanism itself: six services, two planes Langfuse v3 separates two planes that in v2 were fused together: - **Ingestion and query plane** (the two containers of its own, *stateless*, horizontally scalable): Web and Worker. - **State plane** (four dependencies, each with its own load profile): Postgres (transactional OLTP), ClickHouse (analytical OLAP), Redis/Valkey (queue + cache), Blob store (raw objects). > **Diagram:** Six-service architecture of Langfuse v3 — [see the HTML version](https://blog.lo0.es/en/posts/langfuse-inside-sorting-centre-bottleneck/) What to take away from this diagram: - **Web and Worker are interchangeable and stateless.** They store nothing locally. You can run 1 or 20 replicas of each; the state lives in the four dependencies. This is what allows scaling by load without choreography. - **Redis never carries the event content, only the reference** to the object in S3. That is why Redis holds the spike: a Redis write is ~1-5 ms and moves bytes, not kilobytes. The bottleneck of the Web container is, literally, *Redis write speed*. - **Postgres and ClickHouse have opposite profiles.** Postgres is OLTP: many small transactional reads and writes (is this API key valid? which version does the `production` label point to?). ClickHouse is OLAP: few enormous batch writes and analytical queries over billions of rows (give me the p95 TTFT of project X over the last 7 days). Putting traces in Postgres, which is what v2 did, works until it does not: at production volume, Postgres drowns in a workload it was not designed for. That was the reason for the redesign. ## The ingestion flow step by step (and the maths of the decoupling) The heart of the design is the ingestion route. Seen in detail, a `POST /api/public/ingestion` request with a batch of events does this: > **Diagram:** Langfuse asynchronous ingestion sequence — [see the HTML version](https://blog.lo0.es/en/posts/langfuse-inside-sorting-centre-bottleneck/) The mathematical point is the **early ACK**. The latency the client perceives when sending traces is: $$ t_{\text{client}} = t_{\text{S3 write}} + t_{\text{Redis enqueue}} \approx 10\text{–}40\,\text{ms} $$ whereas the real cost of persisting, reading S3, transforming, merging against the previous version, inserting into ClickHouse, letting the background merges compact, happens **off that path**, in the Worker, and can take hundreds of ms or seconds without the client caring. The decoupling turns a system whose throughput would be limited by ClickHouse speed into one limited by Redis speed. And Redis, on modest hardware, sustains on the order of **50,000 operations/second**. This has an important sizing consequence. If your inference load generates $E$ events/second (a chat with RAG + 2 tool calls easily produces 6-10 spans = events per request), the Web container absorbs them while $E \ll 50{,}000$. The Worker, by contrast, scales with the cost of *processing*: that is the component to watch and replicate, and the first knob of the post. > **Honest scepticism.** The early ACK has a flip side: between the HTTP 207 and persistence in ClickHouse there is a **potential loss window**. If the event is in S3 and the reference in Redis, and Redis goes down without persistence (AOF/RDB) before the Worker processes it, the reference is lost; the data is still in S3 but nobody claims it any more. More subtly: the Worker buffers writes to ClickHouse in memory and flushes them in batches; a Worker crash with a full buffer loses that batch. There is a [reported bug](https://github.com/langfuse/langfuse/issues/13468) where the `ClickhouseWriter` discards rows after exhausting flush retries **with no dead-letter queue**. For observability this is usually tolerable (losing 0.01 % of the traces breaks nothing). For *regulatory auditing*, where the trace is evidence, it is not, and Langfuse is better treated as "best-effort" and not as a ledger. We will come back to this at the close. ## Interaction with the rest of the stack: Langfuse in the example 4×H100 cluster Langfuse does not live in isolation. In the [seven-layer stack](https://blog.lo0.es/en/posts/anatomy-on-premise-llm-inference-stack-seven-layers/index.md) it occupies the LLM-aware observability layer, and it relates to almost every other one. On the generic reference cluster we use throughout the blog, **4×H100 SXM 80 GB (320 GB aggregate VRAM), NVLink, 640 GB of system RAM, NVMe-oF, 25/100 GbE network**, the telemetry flow looks like this: > **Diagram:** Interaction of Langfuse with the inference stack on a 4xH100 cluster — [see the HTML version](https://blog.lo0.es/en/posts/langfuse-inside-sorting-centre-bottleneck/) Three ideas from this topology: 1. **Langfuse receives from the OTel Collector, not from the application directly** (in the recommended pattern). The app SDK or vLLM emit spans with the `gen_ai.*` semantic conventions; the Collector does `batch`, `tail-sampling` (preserving 100 % of errors and high latencies, sampling the rest) and enriches with its own attributes (`tenant_id`, `priority_tier`); and it *distributes*: LLM traces go to Langfuse, infrastructure spans to Tempo, metrics (GPU DCGM, vLLM metrics) to Prometheus. Langfuse is **one more exporter**, not the only destination. This is covered in detail in [the OTel tracing post](https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/index.md). 2. **Langfuse runs off the GPUs.** It is a consumer of CPU, RAM, disk and network (ClickHouse wants memory, MinIO wants disk, Redis wants CPU for networking) but **it does not touch VRAM**. On the 4×H100 cluster, Langfuse lives on a CPU node (or on the GPU nodes but with `nodeSelector`/`taints` keeping it away from the vLLM pods). Mixing ClickHouse with vLLM on the same node without resource limits is asking for an ingestion spike to steal memory bandwidth from inference. Isolation by design. 3. **The telemetry route is "cold" and the data route is "hot".** The data plane (left) serves tokens with a millisecond latency budget; the telemetry plane (right) tolerates seconds. The early ingestion ACK is what keeps these two clocks apart: the app does not wait for Langfuse to file anything before returning the response to the user. ## The 10 backend knobs that move the needle most These are, in approximate order of impact/frequency, the settings that decide whether your self-hosted Langfuse ingests 50 events/s or 5,000, and whether your disk grows sustainably or explodes in three weeks. They are all environment variables or config injected into the **Web and Worker** containers (except the ClickHouse ones, which go in its server-side config). The canonical detail is in the [Langfuse scaling doc](https://langfuse.com/self-hosting/configuration/scaling). ### Knob 1 — Scale the Worker by load (the first lever, always) The Worker is the component that saturates first, because it does the expensive work: reading S3, transforming, merging, inserting into ClickHouse. The Langfuse operational rule is simple: **a 2-CPU Worker container above 50 % CPU use is saturated**; add replicas. Better than CPU, the Worker publishes via statsd the metric `langfuse.queue.ingestion.length` (ingestion queue length), which is the direct signal for autoscaling: if the queue grows without draining, there are not enough Workers. ```yaml # The ideal autoscaler watches queue depth, not just CPU. # (KEDA ScaledObject over the statsd metric → Prometheus) triggers: - type: prometheus metadata: query: langfuse_queue_ingestion_length threshold: "10000" # if the queue goes past 10k refs, scale ``` In AWS deployments there is `ENABLE_AWS_CLOUDWATCH_METRIC_PUBLISHING=true` to push these metrics to CloudWatch. On-premise, the path is statsd → Prometheus → KEDA, slotting into the [autoscaling on Kubernetes with KEDA](https://blog.lo0.es/en/posts/llm-inference-autoscaling-kubernetes-hpa-keda/index.md) we already covered for vLLM. **Always start here**: most "Langfuse is slow" problems are simply insufficient Workers, not fine tuning. ### Knob 2 — Separate the ingestion deployment from the UI one When ingestion is heavily loaded, UI queries and the public API become slow because they share the same Web container. The solution is to **split langfuse-web into two identical deployments** and route by path: everything under `/api/public/ingestion*`, `/api/public/media*` and `/api/public/otel*` goes to the ingestion deployment; the rest (UI, read API) to the interface one. ``` # Ingress / gateway rule location ~ ^/api/public/(ingestion|media|otel) { proxy_pass http://langfuse-web-ingest; # replicas dedicated to writing } location / { proxy_pass http://langfuse-web-ui; # replicas dedicated to reading } ``` It is the same idea as the read/write separation of any system with mixed loads: a storm of writes should not starve whoever is trying to *look at* the dashboard right in the middle of the incident, which is precisely when they need it most. ### Knob 3 — Batching writes to ClickHouse (interval + batch size) ClickHouse hates small, frequent inserts: every `INSERT` creates a *part* on disk that then has to be merged, and thousands of tiny inserts generate thousands of parts and a storm of background merges that saturates the disk. The defence is to **accumulate in an in-memory buffer in the Worker and flush in large batches**: ```bash # Worker: fewer flushes, larger batches → fewer parts, fewer merges LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=1000 # raise e.g. to 2000-5000 LANGFUSE_INGESTION_CLICKHOUSE_WRITE_BATCH_SIZE=10000 # raise if there is throughput ``` Raising the interval and the batch size **reduces the flush frequency** and improves sustained throughput. The trade-off is direct and has to be understood: larger, less frequent batches mean **more data in the Worker's volatile buffer**, that is, a bigger loss window if the Worker goes down (a knob coupled to the scepticism at the close). Langfuse also uses ClickHouse `async_insert`, which accumulates server-side before confirming; it adds another layer of buffering to keep in mind. ### Knob 4 — Skip the ClickHouse read-before-write in ingestion By default, when ingesting an event the Worker **reads the existing event from ClickHouse and merges it** with the incoming one (necessary when legacy SDKs send partial events: a `start`, then an `end`, then an `update` of the same observation). That per-event read loads ClickHouse on the write path and limits total throughput. If your projects are not migrated from an old version, because the full history already lives in S3, you can **disable that read**: ```bash # Date earlier than the creation of your first project LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE=2025-01-01 ``` With modern Langfuse SDKs or with ingestion via OpenTelemetry, this does not affect you negatively and removes one read per event. A warning from the doc itself: if you combine this with aggressive deletion (lifecycle) rules in S3 plus late event updates, you can generate duplicates in the history. Know that before enabling it. ### Knob 5 — Write concurrency to S3/Blob storage In high-throughput scenarios, the S3 client can **exhaust its sockets** and start queueing and throttling writes. The symptom is unmistakable in the logs of the Web container that handles ingestion: ``` @smithy/node-http-handler:WARN - socket usage at capacity=150 and 387 additional requests are enqueued. ``` …accompanied by a rise in memory in that container (queued requests pile up in RAM). The cure is to raise the concurrent write limit from its default of 50: ```bash LANGFUSE_S3_CONCURRENT_WRITES=100 # raise gradually from 50 ``` Each additional socket has a small memory cost, so the official advice is to raise it **gradually** while observing the behaviour, not to jump to 1000 in one go. ### Knob 6 — Redis queue sharding + concurrency *per shard* If Redis goes past 90 % CPU, first the obvious: an instance with **at least 4 CPUs** (so Redis spreads networking and background tasks across different cores) and **Redis Cluster mode** enabled. If CPU still does not come down, you can **shard the queues** Langfuse uses: ```bash # Advanced: only if Redis is genuinely drowning and you already did the above LANGFUSE_INGESTION_QUEUE_SHARD_COUNT=6 # ~2-3× the number of Redis cluster shards LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT=6 # Concurrency counts PER SHARD; target ~20 per worker LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=3 # 6 shards × ~3 ≈ 18 LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY=3 ``` Two traps the doc underlines and that are worth tattooing: **once you shard, do not reduce the number of shards** (it breaks the distribution); and concurrency counts **per shard**, not globally, so if you have 10 shards and want concurrency 20 per worker, set `2`, not `20`. It is an advanced knob: most on-premise deployments never need it. ### Knob 7 — The `FINAL` modifier for OTel-only projects Langfuse stores observations in a ClickHouse `ReplacingMergeTree` and, by default, adds the `FINAL` modifier to API queries so the last version of each row wins at read time. `FINAL` is necessary when ingestion produces several versions of the same observation (legacy SDKs with their `start`/`end`/`update` events), but **it adds merge work to every read and slows it down**. Projects that ingest **exclusively over OpenTelemetry** write each observation as a single immutable row, so `FINAL` is redundant for them: ```bash # Recommended in mixed deployments: per-project, flagged in Redis with a 24h TTL LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true # Only if ALL projects are OTel-only: global, no Redis lookup LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL=true ``` Since on the example cluster the instrumentation is 100 % OTel (`gen_ai.*` via Collector), this knob is **free money in dashboard read latency**. Careful with the global version: do not enable it if some project is still using legacy ingestion, or reads may return duplicate or stale rows. ### Knob 8 — Separate analytical reads from the write path (compute-compute) Heavy dashboard queries (percentiles over millions of spans) compete with ingestion inserts and with background merges on the *same* ClickHouse. If your deployment supports **compute-compute separation** (ClickHouse Cloud or BYOC), you can route reads to a read-only compute group: ```bash CLICKHOUSE_URL=http://clickhouse-primary:8123 # writes, migrations, ingestion CLICKHOUSE_READ_ONLY_URL=http://clickhouse-reader:8123 # UI + public API reads ``` **Critical nuance for on-premise**, and here it is time to be sceptical about the usefulness of this knob in our context: on a **single-node** ClickHouse or on a self-managed cluster without compute separation, this variable **adds nothing**, because the read endpoint would be the same as the write one. It is a knob for cloud architectures with storage separated from compute. On an on-premise 4×H100 cluster with ClickHouse on one node, the real alternative is to **scale ClickHouse vertically** (the doc recommends ≥16 GiB of RAM for large deployments; ClickHouse scales vertically well) and to make sure **every query filters by `projectId` and time**, which is how the tables are indexed. Without a time filter, even the fattest ClickHouse suffers. ### Knob 9 — Data retention: TTL in ClickHouse + lifecycle in S3 Disk is the cost that grows on its own. LLM traces carry whole inputs and outputs (sometimes prompts of tens of KB), and ClickHouse also accumulates its own system tables. The first-order lever is a **retention policy** that nightly deletes traces, observations, scores and media older than N days, coordinating ClickHouse and blob storage. Where the retention feature is not available, it is done by hand: ```sql -- ClickHouse: TTL over the tracing tables ALTER TABLE traces MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY; ALTER TABLE observations MODIFY TTL toDateTime(start_time) + INTERVAL 90 DAY; ALTER TABLE scores MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY; ALTER TABLE event_log MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY; ``` ``` # S3/MinIO: lifecycle rule, e.g. 30 days for the raw event bucket # WATCH OUT! Do NOT apply retention to the MEDIA bucket: # - it breaks the files referenced in traces # - it breaks future uploads (state is tracked by hash in Postgres) ``` Two operational parameters that avoid nasty surprises on large deletions: ```bash LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS=600000 # default 10 min; raise it if deletions time out # ClickHouse 25.7+: less mutation pressure on bulk deletions CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE=lightweight_update CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE=true ``` The mental rule: **short retention for raw events** (S3, 30 days is usually enough, they are recoverable/recomputable), **retention by business value for the ClickHouse tables** (90 days, 180, whatever compliance asks for), and **never touch the media bucket with blind lifecycle rules**. ### Knob 10 — Hygiene of the ClickHouse *system log tables* (the silent disk killer) This is the knob nobody configures and that fills the disk without showing up in any Langfuse metric, because **it is not Langfuse data**: they are the system tables of ClickHouse itself (`trace_log`, `text_log`, `opentelemetry_span_log`, `asynchronous_metric_log`, `metric_log`, `latency_log`). By default **they have no TTL**, and the query profiler writes to `system.trace_log` continuously. On a ClickHouse with traffic, these tables can **dominate disk usage** while you look for the problem in your traces. Langfuse does not read from them, so they can be trimmed without fear. Two options: ```xml ``` ```sql -- Option B — aggressive TTL + turn off the profiler, if you want to keep them for debugging -- (in config: query_profiler_real_time_period_ns = 0) SET max_table_size_to_drop = 0; TRUNCATE TABLE system.trace_log; ALTER TABLE system.trace_log MODIFY TTL event_date + INTERVAL 7 DAY; -- repeat for each log table to cap ``` To identify which table is eating the disk, the golden query: ```sql SELECT table, formatReadableSize(sum(bytes)) AS size, sum(rows) AS rows FROM system.parts WHERE active GROUP BY table ORDER BY sum(bytes) DESC; ``` If you take only one knob from this post to your first real deployment, make it this one: the difference between a ClickHouse that grows 2 GB/day of useful data and one that grows 20 GB/day of system logs nobody looks at. ### Summary table of the 10 knobs | # | Knob | Variable / action | When | |---|---|---|---| | 1 | Scale Worker | replicas by CPU>50 % / `langfuse.queue.ingestion.length` | always, first | | 2 | Separate ingestion/UI | route `/ingestion*`,`/media*`,`/otel*` to a dedicated replica | slow UI under load | | 3 | Batching to ClickHouse | `LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS` / `_BATCH_SIZE` | high throughput | | 4 | Skip CH read-before-write | `LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE` | non-migrated projects | | 5 | S3 concurrency | `LANGFUSE_S3_CONCURRENT_WRITES` (def. 50) | "socket usage at capacity" | | 6 | Redis queue sharding | `LANGFUSE_*_QUEUE_SHARD_COUNT` + `*_CONCURRENCY` (per shard) | Redis CPU >90 % | | 7 | Remove `FINAL` (OTel) | `LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true` | 100 % OTel instrumentation | | 8 | CH read/write split | `CLICKHOUSE_READ_ONLY_URL` (cloud/BYOC only) | compute-compute available | | 9 | Retention + TTL | TTL in CH + S3 lifecycle + `LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS` | always (disk cost) | | 10 | CH system log hygiene | `` or aggressive TTL | always (hidden disk) | ## How to maximise Langfuse on the example 4×H100 cluster With the architecture and the knobs clear, this is a concrete sizing to get the most out of Langfuse on the generic reference cluster (**4×H100 SXM, 320 GB VRAM, 640 GB RAM, NVMe-oF, 25/100 GbE**), without stealing a single GB of VRAM from inference. ### Component distribution Langfuse is **100 % CPU/RAM/disk/network load**, so its natural place is **off the GPU nodes** or, if it cohabits, with `taints`/`nodeSelector` confining it away from the vLLM pods. Suggested distribution: ``` nodo-cpu-01 (control + observability, no GPU) ├── langfuse-web-ingest ×3 (2 CPU / 4 GiB each) ← ingestion, scales with load ├── langfuse-web-ui ×2 (2 CPU / 4 GiB each) ← dashboard/read API ├── langfuse-worker ×4 (2 CPU / 4 GiB each) ← the one that scales most ├── redis/valkey ×1 (4 CPU / 4 GiB, cluster mode) └── postgres ×1 (2 CPU / 8 GiB, replica for HA) nodo-storage-01 (heavy state, local NVMe) ├── clickhouse ×1 (8 CPU / 32 GiB / NVMe) ← ≥16 GiB is the minimum; 32 is comfortable └── minio (S3) ×1 (4 CPU / 8 GiB / HDD+NVMe cache) nodo-gpu-01..02 (4×H100 SXM each) → inference ONLY └── vLLM, embeddings, reranker, guardrails (emit spans, do not host Langfuse) ``` ### Sizing by real load Let us put numbers on an example load. Suppose the cluster serving **300 requests/second** of chat-with-RAG, where each request generates on the order of **8 spans** (request, retrieval, rerank, 2× tool, guardrail in, llm, guardrail out): $$ E = 300\,\tfrac{\text{req}}{\text{s}} \times 8\,\tfrac{\text{spans}}{\text{req}} = 2,400\ \text{events/s} $$ Against the Redis ceiling (~50,000 ops/s), $E = 2{,}400$ leaves the reception counter at **~5 % of its capacity**: enormous headroom. The component to watch is the Worker. With a target of ~20 concurrency per Worker and batches of 10,000 events every ~1-2 s, 4 Workers drain 2,400 ev/s with margin; the `langfuse.queue.ingestion.length` metric should stay flat near zero. If it grows, knob 1 (more Workers) is the answer before any fine tuning. **Tail-sampling is the multiplier that changes the economics.** If the Collector preserves 100 % of errors/high latencies but samples normal traffic at, say, 10 %, the 2,400 ev/s you *store* in ClickHouse drop to ~240-300 effective ev/s without losing the signal that matters. The rule: **sample in the Collector, not in Langfuse**. Langfuse should receive, already filtered, what deserves to be persisted. This is developed in [the OTel tracing post](https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/index.md); here it is enough to note that upstream sampling is, in effect, the knob 0 that multiplies all the others. ### Storage estimate An LLM observation with full input+output weighs, compressed in ClickHouse, on the order of **1-3 KB** (ClickHouse compresses text very well, 5-10×). With sampling at 10 % over 2,400 ev/s: $$ 240\,\tfrac{\text{ev}}{\text{s}} \times 2\,\text{KB} \times 86,400\,\tfrac{\text{s}}{\text{day}} \approx 41\ \text{GB/day (raw)} \;\xrightarrow{\text{compression}}\; \sim 5\text{–}8\ \text{GB/day in CH} $$ At 90 days of retention (knob 9), the permanent archive settles at around **500-700 GB in ClickHouse**, comfortable on the NVMe of the storage node, plus the raw events in MinIO with a 30-day lifecycle. **Without** the system log hygiene (knob 10), add easily as much again in rubbish nobody queries. The two disk knobs together are the difference between planning storage once a year and fighting a full disk every month. ### "Maximum use" checklist 1. **Sampling in the Collector** (tail: 100 % errors + N % normal), before touching anything in Langfuse. 2. **Workers scaled by queue length** via KEDA (knob 1), not fixed. 3. **Ingestion separated from UI** (knob 2) so the dashboard responds during incidents. 4. **`SKIP_FINAL_FOR_OTEL_PROJECTS`** enabled (knob 7) because the instrumentation is 100 % OTel. 5. **Generous CH batching** (knob 3) tuned to the throughput, accepting the loss window. 6. **Retention + TTL + system log hygiene** (knobs 9 and 10) configured on day 1, not when the disk screams. 7. **ClickHouse with ≥16 GiB and every query filtering by `projectId`+time** (knob 8 in its on-premise version: vertical scaling). 8. **Langfuse isolated from the GPUs** by `taints`/`nodeSelector`: not a MB of VRAM, no memory bandwidth contention with vLLM. ## Traps and things that are not what they look like **"Langfuse guarantees me full traceability."** No: the design is **high-performance best-effort**, not a ledger. Between the HTTP 207 and the row in ClickHouse there are volatile buffers (Redis without hard persistence, the Worker's in-memory buffer, ClickHouse's server-side `async_insert`). There is a [known bug](https://github.com/langfuse/langfuse/issues/13468) where the writer discards rows with no dead-letter queue after exhausting retries. For operational observability, losing 0.01 % of spans is irrelevant. For **ENS/EU AI Act audit evidence**, where the trace *is* the proof, Langfuse should not be the only record; the regulatory audit log needs durability guarantees this pipeline does not promise. A distinction covered in [the technical controls for ENS/42001/EU AI Act](https://blog.lo0.es/en/posts/technical-controls-ens-iso-42001-eu-ai-act-cross-mapping/index.md). **Raising ClickHouse batching "to go faster" and nothing else.** Knob 3 improves throughput at the cost of enlarging the loss window and the latency of data appearing in the dashboard. Batches of 50,000 every 10 s perform wonderfully… until the Worker restarts with 50,000 events in the buffer. Tune with awareness of the trade-off, not by blind maximisation. **Putting ClickHouse on the same node as vLLM with no limits.** ClickHouse is voracious with memory bandwidth during merges. Sharing a node with vLLM without `resources.limits` or NUMA isolation means an ingestion spike can degrade inference TTFT, exactly the original sin this whole architecture wanted to avoid. Isolate. **Forgetting the time filter in your own queries.** The ClickHouse tables are indexed by `projectId` and time. A custom dashboard or an API query without a time filter scans the whole history and takes performance down for everyone. It is not Langfuse "being slow": it is a badly written query. **Applying lifecycle to the media bucket.** It breaks the files referenced in traces and blocks future uploads (state is tracked by hash in Postgres). The media bucket is managed **only** with the Langfuse retention feature, never with blind S3 rules. **Treating queue sharding as a routine optimisation.** It is an advanced knob for a genuinely drowning Redis, irreversible (do not reduce shards) and with *per-shard* concurrency semantics that are easy to misread. In the vast majority of on-premise deployments it is not needed; if you enable it "just in case", you complicate your life without gaining anything. ## Conclusion Langfuse v3 solved the structural problem of LLM observability, that the observer should not suffocate the observed, by moving from a monolith on Postgres to a six-service sorting centre with asynchronous ingestion. That design is what allows a cluster serving thousands of tokens per second to be instrumented end to end without the app ever waiting for a trace to be filed. But the design is a necessary condition, not a sufficient one: it performs if the right levers are tuned. Of the ten knobs, three decide almost everything in a typical on-premise deployment, **scaling Workers by queue length (1), retention + TTL (9), and system log hygiene (10)**; the rest are refinements that show up when the load bites. And above them all lives knob 0, which is not a Langfuse one: **sampling in the Collector**, which decides how much reaches the pipeline before any internal setting matters. Maximising Langfuse on the 4×H100 cluster is not about squeezing its peak throughput: it is about putting it off the GPUs, feeding it already-sampled traffic, sizing the Worker by the queue, and configuring retention on day one, so that the tool that came to tell the story does not end up being the chapter about the incident. ## See also - [LLM tracing with OpenTelemetry GenAI](https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/index.md) — the `SDK → Collector → backend` pipeline that feeds Langfuse. There Langfuse is treated as a destination; here it is opened up. The two-layer sampling of that post is the knob 0 that multiplies the ten in this one. - [Prompt versioning with Langfuse and MLflow](https://blog.lo0.es/en/posts/prompt-versioning-contract-five-word-change/index.md) — the prompt management layer that lives in Postgres (not in ClickHouse). The `prompt_id@version` that post propagates as a span attribute lands in the tracing tables described here. - [Evals: the layer after tracing](https://blog.lo0.es/en/posts/evals-llm-layer-after-tracing/index.md) — the Langfuse datasets and evaluators rest on this same backend; the stored traces are the input of continuous eval. - [The OSS catalogue for LLMOps in six stages](https://blog.lo0.es/en/posts/oss-catalogue-llmops-six-stages-tool-by-tool/index.md) — the Langfuse entry alongside Phoenix and the rest of the observability ecosystem. - [The on-premise LLM inference stack in seven layers](https://blog.lo0.es/en/posts/anatomy-on-premise-llm-inference-stack-seven-layers/index.md) — where Langfuse fits (layer 5, LLM-aware observability) in the whole building and how it is sized on the same 4×H100 cluster. - [Autoscaling LLMs on Kubernetes with KEDA](https://blog.lo0.es/en/posts/llm-inference-autoscaling-kubernetes-hpa-keda/index.md) — the concrete mechanism for scaling the Langfuse Workers by `langfuse.queue.ingestion.length` (knob 1), the same pattern as for vLLM. - [Technical controls: ENS, ISO 42001 and the EU AI Act](https://blog.lo0.es/en/posts/technical-controls-ens-iso-42001-eu-ai-act-cross-mapping/index.md) — why Langfuse is best-effort observability and does not replace the regulatory audit log with durability guarantees. - [LiteLLM and Langfuse: the operational pair](https://blog.lo0.es/en/posts/litellm-langfuse-operational-pair/index.md) — the cable between the gateway and this backend: integration routes, `traceparent` propagation and the four bounded queues that drop events before they reach the ingestion pipeline described here. - [Langfuse v4: what goes into a trace](https://blog.lo0.es/en/posts/langfuse-v4-what-goes-into-a-trace/index.md) — version 4 changed the data model underneath this architecture: there is no trace table any more. It opens the series of eight about operations. ## References - Langfuse, *Scaling Langfuse Deployments* (doc oficial de sizing y todos los env vars de este post): . - Langfuse, *Self-host Langfuse* y *Configuration via Environment Variables*: · . - Langfuse, *ClickHouse (self-hosted)*: . - Langfuse, *From Zero to Scale: Langfuse's Infrastructure Evolution* (el porqué del rediseño v2→v3): . - Langfuse, *Migrate v2 to v3 (self-hosted)*: . - ClickHouse, *Langfuse and ClickHouse: A new data stack for modern LLM applications*: . - Langfuse, issue #13468 — *ClickhouseWriter drops rows after max flush attempts with no DLQ* (la ventana de pérdida documentada): . - ClickHouse, *TTL for tables and columns*: . - OpenTelemetry, *Semantic Conventions for Generative AI* (`gen_ai.*`): .