Real ClickHouse capacity in Langfuse v4: 2,782 bytes per observation, and why almost half of that is not data

Sixth article in the series on operating self-hosted Langfuse v4. The previous five cover what goes into a trace, putting LangGraph in front, two well-known agents instrumented, migrating with no window and the worker queues. This one measures what all of that takes up. Benchmark run on 15 September 2026 against the event schema of version 4.36.0, identical to that of 4.35.0.

TL;DR

Langfuse publishes no capacity figure at all. Not bytes per row, not events per second, not expected size. I searched their code with every reasonable combination and there is nothing. The only quantified thing is the commercial quotas of their managed service, which say nothing about disk.

One observation takes 2,782 bytes of disk. Split across the two event tables: 2,347 in the full table and 435 in the listings one. Measured over two hundred thousand observations shaped the way a real agent produces them.

41.8 % of the full table’s disk is not data, it is indexes. Four full-text indexes add up to 187 MiB against 260 MiB of data. The one covering the input costs 523 bytes per observation, almost three quarters of what the column it indexes costs.

The materialized view trims the text eight times over and saves eighteen per cent of disk. Truncating input and output to two hundred characters leaves the text at 64 MiB of the original 540, and even so the second table adds 18.5 % on top of the first. What survives the trimming is the seventy-eight-column skeleton and a metadata field that is truncated element by element, not as a whole.

In the listings table, metadata costs more than input and output together. 132.6 bytes per observation against 101.8. And its full-text index adds another 111.9, almost as much as the column.

A compression ratio quoted without saying what was compressed is worth nothing. My first pass, with repeated text, gave 38 times. With text of an entropy close to that of natural language it gives 3.68. Same schema, same indexes, same version.

Generations are 11 % of the observations and 65 % of the bytes. Their mean input is 15,655 bytes and the 95th percentile is 55,205, because each one reserialises the full history. It is the quadratic law from the second article in the series, landed on disk.

On deletion, the disk does not go down. A delete stops showing the rows and leaves the space occupied. A second step is needed, applying the deleted mask, and the cron that does it ships disabled, just like the retention one. On top of that the cleaner works table by table, so the second table keeps the rows the first one no longer shows.

And automatic retention is a paid feature when self-hosted. The data-retention entitlement is not in the open edition. Without it, the event tables grow without limit: they carry no expiry clause.

You are here: the sizing you cannot do by reading

The previous five articles describe how version 4 works and how to put it to work. The moment arrives to ask for disk, and there the series runs out of sources.

The official documentation gives a container sizing: two CPUs and eight gibibytes for ClickHouse, in Kubernetes eight of request and sixteen of limit, a hundred gibibytes of disk and three minimum replicas. It is a reasonable starting point and it does not answer the only question that matters, which is how long that disk lasts.

I went looking for it in the code, which is what this series does. I searched for events per second, rows per observation, bytes per row, capacity, benchmark, sizing, across the whole repository and its documentation. There is no figure at all. The closest thing is the ingestion quotas of their managed service, which are commercial API limits, and some working notes on a synthetic load generator that refer to the tables of the previous version and bring no results, only parameters.

So the only honest way out was to measure it. This article is that measurement, with the whole method laid out so that anyone can repeat it and contrast it with their own load.

The analogy: the archive and the index card

A municipal archive keeps case files. Each file has its folder with everything inside, and on top of that the archivist fills in an index card with the header data and the first lines of the subject, so that the file can be searched without pulling out the folder.

Intuition says the cards take up nothing compared with the folders. And that is true if you look only at the paper. It stops being true when the archivist, in order to be able to search by any word, also builds a term file: a box with one card per distinct word appearing in any case file, pointing to where it occurs. That box grows with the vocabulary, not with the number of case files, and ends up taking almost as much room as the case files themselves.

In this setup the folders are the full table, the cards are the listings table, and the term file is the full-text indexes. The article measures the three things separately, because whoever sizes by looking only at the folders falls short by almost half.

The benchmark

I want this to be reproducible, so the method goes in whole.

The schema is the real one. I extracted the ClickHouse migration DDL from the repository, in its single-instance variant. There are not three migrations but six: the three that create the two tables and the materialized view, plus one that adds three ingestion attribution columns, another that adds an n-gram index over the metadata, and another that adds three evaluation columns and two indexes. The result is 78 columns per table, eleven skip indexes in the full table and fourteen in the listings one.

I verified that this DDL is identical between the version I have pinned across the series and the current head of the repository: the diff of the migrations directory is empty.

The engine is ClickHouse 26.9.1. The minimum version 4 demands is 25.12, because of the text indexes the schema needs. I am using a newer version, and that is a difference from an installation that sticks to the minimum.

The data is synthetic and correctly shaped. The benchmark does not invent the rows: the columns and their typical values come from the worker function that builds the record. I write only into the full table, which is what the worker does, and let the materialized view populate the second one. That reproduces the real write amplification.

The topology of each trace reproduces the law we measured in the second article of the series: four fixed observations plus five for each tool call. The number of calls per turn comes from a distribution with a low median and a long tail, capped at fifteen. Generations carry the full message history, which grows with the turn, because that is exactly what the instrumentation does and it is the origin of the quadratic growth in bytes.

Result: 200,012 observations across 22,563 traces, that is 8.86 observations per trace.

And the entropy of the text is calibrated, which is the part that almost ruined the benchmark. My first pass used a repeated paragraph as filler. With that text, the full table compressed 38.1 times and the figures came out beautiful and false. A repeated lorem ipsum is the best possible input for a compressor. I redid the generator with a wide vocabulary sampled from a Zipf distribution and fifteen per cent high-entropy identifiers, calibrated to give the ratio that natural language mixed with identifiers gives.

With that text, the same table compresses 3.68 times. Ten times less. Any ClickHouse capacity figure that does not say what was compressed is noise, and mine would be too without this paragraph.

What one observation takes up

The two tables, after forcing the final merge:

The figure to take away is the last row added up: 2,782 bytes of disk per observation, counting both tables, with one replica and no retention.

And it is worth looking at the distance between the last two rows. The compressed data of the full table is 1,364.6 bytes per observation, but on disk it takes 2,347. The difference, 982 bytes per observation, is the next section.

Where the bytes go

Broken down by column, with what each one costs per observation:

Three things jump out.

The first is that the skeleton is cheap. Sixty-seven columns, most of them empty in a normal observation, cost seventeen bytes between them all. ClickHouse compresses a column of empty strings down to almost nothing. Anyone who feared that the thirteen experiment columns or the telemetry ones would weigh can stop fearing it.

The second is that in the full table the input is half of everything. 717.5 of 1,364.6 compressed bytes. It is the direct consequence of each generation reserialising the history.

The third is the one I was not expecting. In the listings table, metadata costs more than input and output together: 132.6 against 101.8. The reason lies in how the materialized view truncates. It trims input and output to two hundred characters each, but it trims the metadata element by element: an array with forty values can survive with eight thousand characters. The table that exists in order to be light fills up with the one thing its truncation does not control.

The split by observation type

Generations are 11 % of the observations and contribute 65 % of the input and output bytes. The 95th percentile of their input is 55 kilobytes, in a benchmark where no conversation goes beyond fifteen tool calls.

This has a sizing consequence worth saying out loud: the size of the database is not determined by the number of requests, it is determined by the depth of the turns. Doubling the users doubles the disk. Doubling the tool calls per turn multiplies it by rather more than two.

The indexes, which are almost half

Here is the finding that changes a sizing exercise the most.

The marks and the primary key are irrelevant: thirty-two kilobytes and three hundred bytes respectively, with the granule size of 64 mebibytes the full table has declared.

Broken down, and ordered by what they cost:

The full-text index over the input costs 523 bytes per observation when the column it indexes costs 717. That is, indexing the input costs almost three quarters of what storing it costs. And the metadata one in the listings table costs 111.9 when its column costs 132.6: eighty-five per cent.

The Bloom filters, on the other hand, are free as far as sizing goes. The four identifier ones and those for model name, experiment and evaluator add up to less than two bytes per observation between them all.

This is not a criticism of the design. Those indexes are what make text search in the interface respond, and without them the experience would be another thing. It is sizing information that is published nowhere: when asking for disk for ClickHouse you have to multiply what the columns say by 1.7.

The materialized view as an amplifier

The second table exists so that listings do not have to touch the large fields. It truncates input and output to two hundred characters and does without the text indexes over them.

The trimming works: the benchmark’s 540.55 MiB of input and output come down to 64.68 MiB, eight and a half times less.

And even so the second table adds 18.5 % of disk on top of the first. For the reasons already seen: the skeleton survives, the metadata truncated element by element survives, and its own text index over that metadata costs another 111.9 bytes.

It is worth keeping in mind because it is counter-intuitive. You would think a table storing two hundred characters of text is an appendix. It is four hundred and thirty-five bytes per observation, almost a fifth of the total, and it is paid always, because the materialized view cannot be switched off without leaving the interface with no listings.

What happens on deletion

The retention cron runs a lightweight delete. I reproduced it: I deleted the observations older than a date, which removed 67,395 rows of the 200,012.

The delete frees not one byte. The rows stop being visible and the space stays occupied. The disk only goes down at the second step, applying the deleted mask, which frees 33.2 %.

And that second step is run by a different cron that ships disabled, just like the retention one. With the default values, whoever switches retention on in the interface will see the traces disappear from the view and will never see the disk go down, until an ordinary merge drags in the affected parts, which can take however long it takes.

There is a third detail. The cleaner works table by table, with an independent deleter for each one. In my test I deleted in the full table and the listings one kept its 200,012 rows, because the materialized view does not propagate deletes. In a real installation both deleters run, but they are two separate operations that can diverge if one fails.

On top of this you have to add what the fourth article already covered: neither of the two tables carries an expiry clause. There is no automatic expiry in the engine. The whole lifecycle depends on those crons. And per-project retention requires an enterprise edition entitlement, so in the open edition the only retention possible is the one you write yourself.

Projection

With the measured 2,782 bytes per observation, one replica and no retention:

The conversion to turns uses the benchmark’s 8.86 observations per trace, which depend on the depth of the agents. With deeper agents, the same turn figure produces more observations.

And there is still a multiplication left. Langfuse recommends three minimum replicas of ClickHouse in Kubernetes. With one million observations a day and three replicas, a year is 2.8 tebibytes of raw disk. The official sizing of a hundred gibibytes per replica covers about thirty-eight days at that rate.

That is the number that is published nowhere and that has to go into the spreadsheet before committing to a service.

What this benchmark does not measure

For the sake of honesty, and because anyone repeating this is going to get different numbers:

The data is synthetic. The shape of the row and the topology of the traces are taken from the code and from the measurements in the previous articles, but the text is generated by a sampler, not by a model. I have calibrated its entropy so that compression resembles that of natural language, and that calibration is the largest source of error in the whole article. A load with a lot of structured JSON will compress better; one with a lot of code or a lot of base64, worse.

It is a single instance, with no replication. In a replicated installation there is a different engine and there are copies.

It is ClickHouse 26.9.1, above the 25.12 minimum. The text index implementations have changed between versions and I have not compared.

It does not measure performance: not ingestion per second, not query latency, not the CPU cost of merges. Disk only. The merge cost of full-text indexes is a topic with a life of its own and is left for the saturation runbook.

And it does not measure object storage, which in version 4 is mandatory and keeps a copy of the original event. That is additional disk this benchmark does not touch.

Checklist

• Do not size using the official container sizing: it covers CPU and memory, not disk growth.
• Count 2,782 bytes per observation as a starting point, and adjust by measuring your own load.
• Multiply by the number of replicas before asking for the volume.
• Multiply what the columns suggest by 1.7, because the indexes are 41.8 % of the full table’s disk.
• Measure your own installation with the system tables instead of trusting this article: active parts for the total, columns for the breakdown, skip indexes for what is not data.
• Watch the metadata, which is the only thing the materialized view’s truncation does not control, because it trims each element and not the whole.
• Size by turn depth, not by number of requests.
• Switch on both crons, the retention one and the deleted mask one, and check that the disk really goes down after the second.
• Reckon on per-project retention requiring a licence, and budget the work of writing it if you do not have one.
• Remember that the tables have no expiry in the engine: with no cron, they grow for ever.

Traps and things that are not what they look like

• Langfuse publishes no capacity figure at all, neither in its code nor in its documentation.
• A compression ratio without saying what was compressed means nothing: the same schema gives 38 times with repeated text and 3.68 with realistic text.
• 41.8 % of the full table’s disk is indexes, not data.
• Indexing the input costs almost as much as storing it: 523 bytes per observation against 717.
• The 78-column skeleton is irrelevant: seventeen bytes per observation across the sixty-seven small columns.
• In the listings table metadata costs more than input and output together.
• The materialized view’s truncation trims each element of the metadata, not the whole array.
• The listings table is not an appendix: it adds 18.5 % of the first table’s disk despite storing two hundred characters.
• A delete frees no disk. The mask has to be applied, and that cron ships disabled.
• The retention one also ships disabled out of the box.
• The cleaner works table by table; the materialized view does not propagate deletes.
• The event tables carry no expiry clause.
• Per-project retention is a paid feature when self-hosted.
• Generations are 11 % of the observations and 65 % of the bytes.
• Bloom filters are free as far as disk goes; the full-text ones are not.

Closing

What I take from this benchmark is not the figure, although the figure was needed. It is that the split is not where you look for it.

When somebody sizes a trace database, they look at the size of the prompts. It is the natural thing to do: they are the large fields and they are the ones you see in the interface. And it turns out that in this architecture the prompts are half the compressed data, and the compressed data is a little over half the disk. The rest is search structures nobody counts because they do not show up when you look at a row.

There is a pattern there that goes beyond this tool. An observability database is, by definition, a database queried in unforeseen ways: by free text, by user, by session, by whatever is needed when something goes wrong at three in the morning. That ability to search by anything is paid for in disk, always, and it is paid at write time and not at search time. Langfuse’s design pays it fairly explicitly, with four full-text indexes declared in the migration, in plain sight for anyone who wants to read them.

What was missing is for somebody to put the number alongside. Now it is there, with its margin of error and its method on the table, so that the next person to measure it can tell me where I got it wrong.

The series: the ten articles

1. What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
2. Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
3. Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
4. Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
5. The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
6. Real ClickHouse capacity and cost (this article): how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
7. Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
8. Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
9. Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
10. Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

See also

• Self-hosted Langfuse: architecture and tuning — the architecture of the previous version, where this setup comes from.
• LiteLLM and Langfuse: the operational pair — who produces the traces measured here.
• Sizing for agents — the other side of sizing, that of the gateway and the inference engine.
• deepagents on a cluster of your own — the harness that generates these observations and where it keeps its own state.
• Technical controls for ENS, 42001 and the AI Act — the framework where retention and deletion fit.

Sources

• Langfuse, migraciones de ClickHouse (packages/shared/clickhouse/migrations/canonical/, ficheros 0039 a 0047): https://github.com/langfuse/langfuse.
• Langfuse, escritura del worker en ClickHouse (worker/src/services/IngestionService/, worker/src/services/ClickhouseWriter/): https://github.com/langfuse/langfuse.
• Langfuse, limpiador de retención por lotes y limpiador de máscaras de borrado (worker/src/features/batch-data-retention-cleaner/, worker/src/features/deleted-mask-cleaner/): https://github.com/langfuse/langfuse.
• Langfuse, guía de autoalojamiento y requisitos de infraestructura: https://langfuse.com/self-hosting.
• ClickHouse, tablas del sistema parts, columns y data_skipping_indices: https://clickhouse.com/docs/en/operations/system-tables.
• ClickHouse, índices de salto y de texto completo: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree.
• ClickHouse, borrados ligeros y máscara de borrado: https://clickhouse.com/docs/en/guides/developer/lightweight-delete.