Runbook on lo0 — Tech Blog

Langfuse v4, day 2 (5 of 10): the worker's thirty-nine queues, and the three that decide whether you lose data

Mon, 14 Sep 2026 08:30:00 +0200

Fifth article in a series of ten about operating Langfuse v4 in production. The previous one covered migrating from version 3. This one goes inside the worker that is already running. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

Thirty-nine queues and thirty-two switches. The catalogue is in packages/shared/src/server/queues.ts:406. The QUEUE_CONSUMER_*_IS_ENABLED variables are read once at boot, so turning a queue off is a restart, not a hot change.

Sharding spreads nothing without Redis Cluster. The producer only applies the hash if REDIS_CLUSTER_ENABLED === "true", and that variable ships "false". Raising LANGFUSE_INGESTION_QUEUE_SHARD_COUNT on an installation with single-node Redis starts N workers and leaves N−1 with no work.

The dead-letter retrier ships disabled. QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED is the only one of the thirty-two whose default is "false".

And even switched on, it does not cover ingestion. Its list holds five queues and none is ingestion-queue. An event that exhausts its six attempts sits in the failed set.

The ClickHouse writer drops rows. Three attempts with a flat hundred-millisecond delay and then out. The code comment is explicit: // TODO - Add to a dead letter queue in Redis rather than dropping. The metric to watch is called langfuse.queue.clickhouse_writer.rows_dropped.

Turn a queue’s consumer off and you stop seeing its depth. The metrics emitter only polls queues with a worker registered in that container. The queue keeps growing and the dashboard goes flat.

Project deletion runs one job every ten minutes. By limiter, deliberately, because each one scours ClickHouse.

There are two optional health probes and one of them can cause the very restart loop it is meant to fix if configured with less than sixty seconds of initial delay.

You are here: OBSERVE, day 2

The previous article left the installation migrated. This is the map of the process that does the work, and it comes before the capacity article because the queues are where saturation shows up before ClickHouse notices.

The analogy: the sorting hub

A postal sorting hub has separate belts for parcels, registered mail, returns and internal pouches. Each belt has its speed and its staff, and some are throttled on purpose because the destination cannot take more.

The job of whoever runs it is not to watch the fastest belt, it is to know which one jams first, which can stop for an afternoon without anyone noticing and which cannot stop for ten minutes. And above all, where the bin for items nobody could sort is, because every hub has one and in many of them nobody looks at it.

The Langfuse worker has thirty-nine belts. This article is the plan, with the speeds and the location of the bin.

The map of the thirty-nine

Grouped by what they do, with what stops happening if the group stops:

Ingestion (4). ingestion-queue, secondary-ingestion-queue, otel-ingestion-queue, secondary-otel-ingestion-queue. If they stop, the web keeps accepting events and uploading them to object storage, and keeps enqueueing. Nothing reaches ClickHouse. The SDK sees no error at all. This is the group that cannot stop.

Propagation (1). event-propagation-queue, the dual-write one from the previous article, with a global concurrency of one across the cluster.

Evaluation (8). trace-upsert, dataset-run-item-upsert-queue, create-eval-queue, evaluation-execution-queue and its secondary, llm-as-a-judge-execution-queue, code-eval-execution-queue, experiment-create-queue. If they stop, evaluations are neither created nor executed. Traces keep arriving. This is the group that can wait.

Deletion and retention (7). trace-delete, score-delete, dataset-delete-queue, project-delete, data-retention-queue and its processing queue, batch-action-queue. If they stop, neither UI-requested deletions nor per-project retention run. That is not an operational annoyance, it is a breach if you have a retention commitment in writing.

Exports (3). batch-export-queue, core-data-s3-export-queue, metering-data-postgres-export-queue. Downloads a user asked for stay pending.

Outbound integrations (8). The PostHog, Mixpanel and blob storage pairs, plus webhook-queue, entity-change-queue and notification-queue. In each pair, the bare queue is the scheduler that fans out per project and the processing one does the work.

Cloud billing (4). They only register with a Stripe key. On a self-hosted installation they never start.

Maintenance (4). dead-letter-retry-queue, monitor-queue, in-app-agent-run-queue and v4-legacy-api-usage-queue, which ingests nothing: it is a cron that scans ClickHouse’s system.query_log to find which projects still use the old API.

The switches number thirty-two, all QUEUE_CONSUMER_*_IS_ENABLED and all "true" except the dead-letter retrier. Four queues have no switch of their own: the model-as-judge one rides inside the evaluation switch, the two Cloud data export queues have their own variable, and the processing queues share a switch with their scheduler.

The mechanism is a top-level if around each worker registration. A false switch means the worker is never instantiated, not that the queue stops receiving work: producers keep enqueueing regardless. Turning a queue off is not consuming it, not not filling it.

The sharding that does not shard

Nine queues support sharding: the four ingestion ones, three evaluation ones, model-as-judge and trace-upsert. Each has its own shard-count variable, and they all default to 1.

Raising them looks like the obvious first move to scale ingestion. It is not, because of this (packages/shared/src/server/redis/ingestionQueue.ts:48):

 const shardIndex =
 IngestionQueue.getShardIndexFromShardName(shardName) ??
 (env.REDIS_CLUSTER_ENABLED === "true" && shardingKey
 ? getShardIndex(shardingKey, env.LANGFUSE_INGESTION_QUEUE_SHARD_COUNT)
 : 0);

The : 0 at the end is the whole story. The producer only computes a shard if Redis is in cluster mode, and REDIS_CLUSTER_ENABLED ships "false". On an installation with single-node Redis, which is most self-hosted installations, raising the shard count to four starts four workers, three of which stare at empty queues, while all traffic keeps coming through ingestion-queue.

The sharding key, when it does apply, is projectId-eventBodyId run through SHA-256, taking the first eight hex characters and a modulo. Shard names are ingestion-queue for zero and ingestion-queue-N for the rest.

Without Redis Cluster, the lever that does work is concurrency, which is a different thing.

Concurrencies, and the ones set to one on purpose

The defaults, worth knowing because several sit at one for good reasons:

Queue	Variable	Default
`ingestion-queue` (per shard)	`LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	20
`secondary-ingestion-queue`	`LANGFUSE_INGESTION_SECONDARY_QUEUE_PROCESSING_CONCURRENCY`	5
`otel-ingestion-queue`	`LANGFUSE_OTEL_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	5
`trace-upsert`	`LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY`	25
`monitor-queue`	`LANGFUSE_MONITOR_QUEUE_PROCESSING_CONCURRENCY`	10
evaluation and model-as-judge	several	5
`create-eval-queue`	`LANGFUSE_EVAL_CREATOR_WORKER_CONCURRENCY`	2
`trace-delete`, `score-delete`, `dataset-delete-queue`, `project-delete`	several	1
`event-propagation-queue`	fixed in code	1 cluster-wide

Deletions sit at one and carry a rate limiter on top. The project one is the most aggressive (worker/src/app.ts:234):

 limiter: {
 // Process at most `max` delete jobs per LANGFUSE_CLICKHOUSE_PROJECT_DELETION_CONCURRENCY_DURATION_MS (default 10 min)
 max: env.LANGFUSE_PROJECT_DELETE_CONCURRENCY,

One project deletion every ten minutes. If a customer asks you to delete forty projects, that is six hours and forty minutes on the clock, and rightly so, because each of those deletions is a broom sweeping through ClickHouse.

The ingestion queues, by contrast, have no limiter at all. There the brake is concurrency and the ClickHouse writer.

One BullMQ detail that recurs across queues and explains a good number of other people’s stalls (worker/src/app.ts:261):

 // The default lockDuration is 30s and the lockRenewTime 1/2 of that.
 // We set it to 60s to reduce the number of lock renewals and also be less sensitive to high CPU wait times.
 lockDuration: 60000, // 60 seconds
 stalledInterval: 120000, // 120 seconds
 maxStalledCount: 3,

An event’s path, from API to ClickHouse

It is worth following it end to end once, because there are three places it can be lost.

The web groups the batch’s events by body id, uploads one JSON file per group to object storage and aborts if that upload fails: no blob, no enqueue. Then it enqueues one job per body id on ingestion-queue.

The worker picks up the job, optionally checks a recently-processed cache in Redis (off by default, five-minute lifetime), decides whether to redirect the project to the secondary queue, and downloads the file from storage. There is a comment here worth a capacity chart on its own:

Process files in batches. If a user has 5k events, this will likely take 100 seconds.

Then it merges and writes. And writing is not writing: the service does not talk to ClickHouse, it puts the row in a buffer whose defaults are 1,000 rows, 1,000 milliseconds and 3 attempts.

That buffer is the third place things get lost, and the most serious one in the system (worker/src/services/ClickhouseWriter/index.ts:535):

 // Re-add the records to the queue with incremented attempts
 let droppedCount = 0;
 queueItems.forEach((item) => {
 if (item.attempts < this.maxAttempts) {
 entityQueue.push({ ...item, attempts: item.attempts + 1 });
 } else {
 // TODO - Add to a dead letter queue in Redis rather than dropping
 recordIncrement("langfuse.queue.clickhouse_writer.error");
 droppedCount++;
 }
 });

Once the three attempts are spent, the row is dropped. And no queue retry saves it, because the BullMQ job already completed successfully: as far as the queue is concerned, that event was processed. The internal retries are not properly exponential either: the delay is flat at a hundred milliseconds, so all three attempts happen within three hundred milliseconds and a ClickHouse that is unwell for half a second takes the batch with it.

The runbook consequence is direct. langfuse.queue.clickhouse_writer.rows_dropped has to be on a dashboard and it has to have an alert at zero. Any non-zero value is silent data loss.

Retries, and the bin nobody looks at

Each queue defines its own policy. The ones that matter:

Queue	Attempts	Backoff
`ingestion-queue`	6	exponential, 5 s
`secondary-ingestion-queue`	5	exponential, 5 s
`evaluation-execution-queue`	10	exponential, 1 s
`code-eval-execution-queue`	3	exponential, 30 s
`trace-upsert`	2	exponential, 5 s, with a 30 s initial delay
`trace-delete`, `score-delete`	2	exponential, 30 s
`monitor-queue`, `in-app-agent-run-queue`	1	no backoff, deliberately

The two single-attempt ones are a design decision, and the comment explains it well: “Postgres owns correctness (claim CAS + reconcile-on-read); BullMQ is delivery-only, so never redeliver on its own.”

And now the bin. There is no dedicated dead letter queue. What exists is BullMQ’s native failed set per queue, with its own retention, plus a queue that retries from there. That queue has two problems for whoever operates it:

First, it ships disabled. It is the only one of the thirty-two defaulting to "false".

Second, even switched on, its scope is hand-written and has five entries (worker/src/services/dlq/dlqRetryService.ts:9):

 private static retryQueues = [
 QueueName.ProjectDelete,
 QueueName.TraceDelete,
 QueueName.ScoreDelete,
 QueueName.BatchActionQueue,
 QueueName.DataRetentionProcessingQueue,
 ] as const;

ingestion-queue is not there. Ingestion events that exhaust their six attempts sit in the failed set, up to a hundred thousand of them, and nobody ever retries them. To get them out there is an admin route, POST /api/admin/bullmq with action retry, which the file itself describes as the one the managed service uses for exactly this.

The metrics, with their exact names

Each queue’s metric name is built at runtime from the queue name, turning dashes into underscores and stripping the _queue suffix. So ingestion-queue produces langfuse.queue.ingestion, and secondary-ingestion-queue produces langfuse.queue.secondary_ingestion. Sharded queues always add a shard tag.

What comes out of that:

langfuse.queue.<queue>.rate with type in request, completed, failed, error and stalled.
langfuse.queue.<queue>.time_distribution with type in wait and processing.
langfuse.queue.<queue>.depth with type in waiting, failed and active, plus an aggregate series with shard: "all".
langfuse.queue.<queue>.dlq_oldest_age, the age of the oldest failed job, which emits zero when the set empties so the dashboard resets.
From the writer: langfuse.queue.clickhouse_writer.rows_dropped, .error, .processing_time and ingestion_clickhouse_insert_queue_length.

Two things to know about those metrics. The waiting depth includes paused jobs, to match BullMQ’s counter. And the big trap (worker/src/features/queue-metrics-runner/index.ts:70):

 // Only poll queues that have registered workers. This avoids calling
 // getInstance() on queues this worker doesn't consume, which would
 // create unnecessary Redis connections and can trigger side effects

Only queues that container consumes emit metrics. If you split roles and turn a queue’s consumer off across every container, that queue stops appearing on dashboards while it keeps growing in Redis. It is the perfect way not to find out.

The metric that saturates first is still the same one: langfuse.queue.ingestion.depth with type: "waiting". The emission interval defaults to one second.

The two probes, and the one that can restart you in a loop

The worker exposes /api/health and /api/ready, identical but for one thing: the second returns 500 after SIGTERM, which is how draining works. Both always check Postgres with a SELECT 1 and Redis with a ping capped at two seconds, hardcoded.

Then there are two optional checks enabled by query parameter:

?failIfEventPropagationStuck=true looks at the propagator’s heartbeat and returns 503 if it has not been refreshed for more than 35 minutes. A missing heartbeat does not count as stuck, deliberately, so a freshly booted container is not restarted in a loop. And the condition that must be respected:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

?failIfQueueConsumptionStuck=true is an in-memory signal of that container, with no Redis involved: it is marked on the active and completed events of any worker. It returns 503 if that container has gone 60 minutes without picking up or finishing a single job. It exists for a very specific failure mode:

After Redis lock loss BullMQ workers can wedge permanently: the process stays alive and connectivity checks pass, but no queue picks up jobs ever again.

The one-hour threshold leans on the default crons keeping a healthy worker busy at least once an hour. In multi-replica deployments each scheduler tick lands on a single replica, so the comment itself warns about raising the threshold if a replica can legitimately sit idle.

Draining, by the way, has no cap. The order is close the HTTP server, stop the periodic runners, close the BullMQ workers and then flush the ClickHouse writer, which on shutdown empties the whole queue rather than one batch. The only thing bounding that drain is the Kubernetes grace period, so it had better be generous.

The crons

Queue	Pattern	What it does
`event-propagation-queue`	`* * * * *`	propagates one staging partition
`dead-letter-retry-queue`	`0 /10 * * *`	retries failed jobs from five queues
`v4-legacy-api-usage-queue`	`/15 * * *`	scans `system.query_log` for old-API usage
`blobstorage-integration-queue`	`/20 * * *`	fans exports out per project
`cloud-usage-metering-queue`	`5 * * * *`	Cloud billing
`posthog-integration-queue`, `mixpanel-integration-queue`	`30 * * * *`	hourly fan-out
`metering-data-postgres-export-queue`	`30 2 * * *`	daily export
`core-data-s3-export-queue`, `data-retention-queue`	`15 3 * * *`	daily export and retention

And a warning the scheduler file itself writes in capitals: cron registration is fire-and-forget from the queue constructors, and a failure is only logged. A transient Redis failure at boot can leave a cron unscheduled until the next restart, without the container ever being marked unhealthy.

Splitting roles, which is what all this is for

With thirty-two switches, what you can build is one worker per family. The split that makes sense on an installation with agentic load:

# Pool 1: ingestion. This is the one that scales with traffic.
QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_OTEL_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED=true # note: global concurrency 1
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=20
# everything else false

# Pool 2: evaluations. Scales with the number of evaluators, not with traffic.
QUEUE_CONSUMER_EVAL_EXECUTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_CREATE_EVAL_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_TRACE_UPSERT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EXPERIMENT_CREATE_QUEUE_IS_ENABLED=true

# Pool 3: maintenance. One replica is enough.
QUEUE_CONSUMER_TRACE_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_PROJECT_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DATA_RETENTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED=true # switch it on

Two warnings about this split. Each worker registration opens its own Redis connection, so the connection count grows with queues times shards times replicas, and that is checked against Redis maxclients before multiplying replicas. And the moment you split roles you lose the metrics of the queues that container does not consume, so some replica needs to consume every queue or you go blind on that one.

Checklist

There is an alert on langfuse.queue.clickhouse_writer.rows_dropped, threshold zero.
There is an alert on the depth of langfuse.queue.ingestion with type: "waiting".
There is an alert on dlq_oldest_age for the queues that matter, and somebody knows how to use POST /api/admin/bullmq.
The dead-letter retrier is on in some replica, knowing it does not cover ingestion.
Nobody raised the shard count without Redis Cluster.
The optional health probes are configured with initialDelaySeconds of at least 60 seconds.
The Kubernetes termination grace period allows the ClickHouse writer to flush.
Every queue is consumed by at least one replica, so its metrics are not lost.
Redis maxclients copes with queues times shards times replicas.

Traps

Sharding without Redis Cluster does not spread load. It starts the workers and leaves them idle.

The dead-letter retrier ships disabled and does not cover ingestion. Two independent facts, both uncomfortable.

The ClickHouse writer drops rows after three attempts. With a TODO in the code admitting it and three hundred milliseconds of total headroom.

Turning a consumer off also turns its metrics off. The queue keeps growing, the dashboard stays flat.

A cron can end up unscheduled while the container stays healthy. A transient Redis failure at boot, and it is gone until the next restart.

The PostHog and blob storage integrations disable themselves when an error is classified as a customer-config fault, and if the notification also fails they stay off and silent.

A misconfigured propagation probe restarts in a loop. Less than sixty seconds of initial delay and you get the cycle.

Draining has no cap of its own. The Kubernetes grace period bounds it and nothing else.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues (this article): the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/server/queues.ts, packages/shared/src/server/redis/, worker/src/env.ts, worker/src/app.ts, worker/src/queues/, worker/src/services/ClickhouseWriter/, worker/src/services/dlq/, worker/src/features/health/, worker/src/features/queue-metrics-runner/, worker/src/utils/shutdown.ts.
Langfuse self-hosting documentation, checked on 14 September 2026.
BullMQ: going to production, checked on 14 September 2026.

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

Mon, 14 Sep 2026 08:00:00 +0200

Fourth article in a series of ten about operating Langfuse v4 in production. The previous three looked at the client side: what goes into a trace, what an agent turn costs and two well-known agents instrumented. This is the first one on the server side. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

The default write mode is the destination, not the origin. LANGFUSE_MIGRATION_V4_WRITE_MODE ships as events_only (packages/shared/src/env.ts:337). That is right for a new installation and wrong for a migration: in that mode the traces and observations tables stop being written.

The worker refuses to start on three combinations. legacy with direct OTel writes, events_only without the preview opt-in, and events_only with OTel behaviour set to dual. The code comment names them: combinations that would silently lose data (worker/src/env.ts:722).

In events_only twenty-four REST routes return 404. Traces, observations, sessions, v1 and v2 scores, metrics, spans, generations and the dataset-run ones. Not a bug: a deliberate short circuit so stale data from tables nobody writes is never served.

History is backfilled by a chain of five background migrations, not by the ClickHouse migrations. Not one of the ten new ClickHouse migrations copies a single old row.

A failed background migration does not block the next one. The manager has no dependency model and skips them. The chain protects itself with predecessor guards that the first two steps do not declare, so a failure of step 1 does not stop step 2 from starting.

The old API’s ten-minute delay has an exact explanation. Three-minute partitions in the staging table, a lock four minutes after creation, and a per-minute cron that only touches partitions older than ten minutes, with a global concurrency of one.

That staging table has a forty-eight-hour TTL. If the propagator stops over a long weekend, those partitions disappear and the hole in events_full is permanent. No code detects it and none repairs it.

The point of no return is in Postgres, not in ClickHouse. A Prisma migration runs DROP TABLE on traces, observations and scores, and Prisma Migrate has no down migrations.

You are here: OBSERVE, day 2

The first three articles measured what an agent produces. This one starts on operating the backend that receives it, and it comes before the capacity article because you cannot size what has not been migrated yet.

The analogy: changing the track gauge without closing the line

Changing the gauge of a line in service is not done in one go. The third rail is laid, rolling stock of both gauges runs for a while, and only when every train is of the new gauge is the old rail lifted.

The Langfuse migration has exactly that shape. dual mode is the third rail: the old tables and the new ones are written at the same time. And as with the track, the danger is not the day of the change, it is at both ends. Laying the new rail and lifting the old one on the same day strands trains. And once the old rail is up, going back is not a matter of will, it is a matter of the material no longer being there.

The three-rung ladder

Everything turns on one variable with three values, defined three times over in the three packages that have to agree, with the same enum and the same default (packages/shared/src/env.ts:337, worker/src/env.ts:586, web/src/env.mjs:610):

LANGFUSE_MIGRATION_V4_WRITE_MODE: z
 .enum(["legacy", "dual", "events_only"])
 .default("events_only"),

The two helpers that read it say what each rung does better than any explanation (worker/src/env.ts:705):

export const v4WritesToEventsTable = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "legacy";

export const v4WritesToLegacyTables = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "events_only";

What gets written in each mode:

Table	`legacy`	`dual`	`events_only`
`traces`, `observations`	yes	yes	no
`scores`	yes	yes	yes
`dataset_run_items_rmt`	yes	yes	no
`observations_batch_staging`	no	yes	no
`events_full`	no	yes	yes
`events_core`	no	yes (through the materialised view)	yes

The middle rung is the only one where both coexist, and therefore the only place you can step back from without losing anything. The code itself offers it as the remedy when something breaks in events_only (web/src/pages/api/public/ingestion.ts:296):

As a temporary migration bridge, set LANGFUSE_MIGRATION_V4_WRITE_MODE=dual on both the web and worker services and redeploy.

The both in that sentence matters. All three packages read the variable and the code comment insists three separate times on keeping them in sync. Deploying the worker with one value and the web with another is the first way to break this.

The three combinations that refuse to start

The worker validates its configuration at boot and throws in three cases, under a comment that leaves no doubt (worker/src/env.ts:718):

 // Hard errors: combinations that would silently lose data.
 if (mode === "legacy" && otel === "direct") {
 throw new Error(
 "Invalid V4 config: LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct " +
 "requires LANGFUSE_MIGRATION_V4_WRITE_MODE in {dual, events_only}. " +
 "Direct OTel writes target events_full, which is not read in legacy mode.",
 );
 }

The second one bites in a real migration: events_only also requires LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true, because otherwise the web reads would target the legacy tables that mode no longer writes. The third forbids events_only with OTel behaviour set to dual.

A failed boot is the good news. A failed boot shows up in thirty seconds. The combinations it does not validate take days to surface.

What breaks in each mode

This is the table to have in front of you before moving the variable, because the two ends break different things.

In legacy the event tables are not written, so whatever depends on them disappears. Monitor routes return 404 through a dedicated middleware (web/src/server/api/trpc.ts:413). Metrics v2 is unavailable. Dashboard widgets stay at version 1 (DashboardService.ts:39). The v4 preview is forced off and cannot be switched on.

In events_only the legacy tables are not written, so twenty-four REST routes short-circuit with a 404 (web/src/features/public-api/server/createAuthedProjectAPIRoute.ts:319):

 // Short-circuit routes that read from legacy traces/observations tables
 // when the deployment is in events_only mode — those tables are no longer
 // populated, so the response would be stale or empty.

The list covers traces, traces/{id}, observations, observations/{id}, sessions, sessions/{id}, scores and scores/{id} in v1 and v2, metrics, metrics/daily, spans, generations, events, dataset-run-items and the two datasets/{name}/runs routes. If you have your own integrations against those routes, they break on cutover day, and returning 404 rather than stale data is the right call even though it stings.

One detail that is usually reported wrongly: /api/public/ingestion is never shut down. The deprecation notice says so itself (deprecations.ts:104):

This route is never removed: scores (and sdk-log) stay accepted.

In events_only that route stays alive and keeps accepting two event types (web/src/pages/api/public/ingestion.ts:274):

const EVENTS_ONLY_ALLOWED_TYPES = new Set<string>([
 eventTypes.SCORE_CREATE,
 eventTypes.SDK_LOG,
]);

The rest are rejected per event, not per batch: the batch comes back as a 207 with 400s inside. An old client sending traces and scores mixed together will see the scores go through and the rest fail, which is exactly the kind of partial failure that takes a long time to notice.

The three steps, in order

Given all of the above, the order of a no-window migration is this:

Finish the v3 background migrations before upgrading. This comes before everything else and there are no shortcuts. v4 deletes the rows of seven old migrations, and the comment explains why (prisma/migrations/20260723124010_drop_dataset_run_items_table/migration.sql:7): their worker scripts are deleted alongside, so an unfinished row would leave the manager with no script to resolve. If one was halfway, that data is lost and the row disappears quietly.
Upgrade to v4 with LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy, apply the ClickHouse and Prisma migrations. Here the installation behaves like v3 and you verify everything still stands.
Move to dual. From then on both table families are written and the propagator starts. Let it run for as long as it takes and launch the historic backfill.
Once history is complete and verified, move to events_only with the preview opt-in enabled, and only then lift the old rail.

All the work is between 3 and 4, and it can take weeks. There is no rush: dual is a stable state.

The ClickHouse migrations backfill nothing

There are 96 files in packages/shared/clickhouse/migrations/canonical/, 48 up and 48 down, which have to be materialised before running (pnpm ch:migrations:materialize) and cleaned afterwards, including after a failed migration. The runner is golang-migrate.

The ten v4 ones are 0039 through 0048. The first three create events_full, events_core and the materialised view that feeds the second from the first. The three drop migrations remove event_log, project_environments and dataset_run_items. The ones in between add columns and indexes.

What matters for planning is what they do not do. None copies an old row. The materialised view only sees rows inserted into events_full after it exists. The indexes in migrations 0043 and 0047 are added without materialising historical parts, and 0047 says so in writing:

Skip indexes are added without materializing historical parts to keep this migration metadata-only.

Translated into operations: during and after the migration, searches over older data do full scans. Not a bug, the price of a migration that does not block.

And one cluster trap: migrations 0042, 0043, 0047 and 0048 carry alter_sync = 2, meaning every ALTER waits for all replicas. One slow or down replica hangs the whole migration. On a single-replica installation you never notice; on three, you do.

The five-step chain that does backfill history

The backfill is done by five background migrations running in the worker, registered in Postgres with names that carry the order inside them:

20260701_v4_step_1_create_root_spans_from_traces: creates virtual root spans in events_full from existing traces, with the identifier concat('t-', t.id). It skips traces referenced by a dataset run item, which belong to step 4. Chunked by monthly partition.
20260701_v4_step_2_rewrite_observations_to_pid_tid_sorting: copies observations into a scratch table with a different key order. Chunked by monthly partition.
20260701_v4_step_3_backfill_events_full_from_observations: the bulk of it. Chunked by ClickHouse part, not by partition, and the comment explains why that is the only safe granularity:

One todo per part keeps each INSERT bounded to a single ClickHouse part (single-digit GBs in most cases), which is the only granularity that’s safe to assume self-hoster hardware can chew through without OOM or memory-limit failures.
20260701_v4_step_4_backfill_events_full_from_dataset_run_items: walks the dataset run items by cursor and writes the root and its descendants. Batches of 200, a cap of 50,000 descendants per item, a 90-day window.
20260701_v4_step_5_drop_pid_tid_sorting_tables: drops step 2’s scratch tables.

There are only two switches for the five. The first four share LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL, which ships true, and the fifth has its own, LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES, which ships false on purpose so the scratch tables stay around until the operator confirms the backfill went well.

One migration at a time across the whole cluster. The manager takes a lock in Postgres, refreshes it with a heartbeat every 15 seconds and treats a lock older than 60 seconds as expired, all inside a serialisable transaction. Adding worker replicas does not speed the backfill up by one minute. That is the first thing that surprises anyone used to scaling out to go faster.

They are resumable, and rather better than this usually is. Progress lives in a JSONB field on the row, and on resume the worker re-attaches to the queries the previous worker left running, by query id, instead of relaunching them:

In-flight queries keep running server-side; the next run re-attaches to them via the persisted queryIds.

On how long they take, the code says nothing. Not the files, not the folder’s README, not the migrations. The only time threshold anywhere is a design one: “A good threshold is something that takes more than 5 minutes to run”. Any figure in hours or days you read elsewhere, including the one I repeated myself before checking, is not in the code.

The failure that does not block

This is the finding that changes a runbook the most. When a background migration fails, the manager marks it with failedAt and moves on. And the predicate it uses to pick the next one excludes exactly the rows carrying failedAt, so a failed migration is skipped and does not block the ones after it.

The code acknowledges the gap and explains the patch (worker/src/backgroundMigrations/utils/backfillBase.ts:107):

The BackgroundMigrationManager has no dependency model — it runs every eligible row in name order and marks each finished/failed independently, so a failed upstream step does NOT by itself stop a downstream step from running on partial data. Each downstream step calls this from its validate() […] Because the next step in turn guards on its own predecessor, a single failure transitively halts the rest of the chain.

The guard exists, but only in steps 3 and 4. Steps 1 and 2 declare no predecessor. A failure of step 1 does not stop step 2 from starting; what saves the situation is that step 3 does check step 2.

Practical consequence: the only reliable signal that the backfill is going well is to look at the background_migrations table in Postgres and check that all five rows have finishedAt and none has failedAt. And to restart a failed one you clear failedAt by hand and relaunch with --retry-failed, as the error message itself says.

Two concrete failure modes that deserve a place in the runbook:

Step 3 breaks if ClickHouse merges a part midway. The message says it all: “Part … no longer active after processing — its rows are in a merged successor part that is not in this run’s todo list. Clear failedAt and re-run with state.chunksLoaded=false”. Recovery is manual, editing the JSONB.
Step 4 stops dead on one giant trace, if it exceeds 50,000 descendants. The three ways out the error offers are raising the cap, skipping the item (and losing that trace entirely) or fixing it by hand in SQL.

And an asymmetry worth knowing before comparing data: the historic backfill does not copy trace metadata down to child spans, deliberately, whereas the live path does concatenate them. History and new data are not identical.

Where the ten minutes come from

The v3 API warns that its data runs about ten minutes behind (deprecations.ts:16):

Data on this API is delayed by about 10 minutes. Real-time is only OpenTelemetry writes plus v2 observations and v2 metrics reads.

That delay is the sum of four things, all visible in the code:

The staging table partitions on three-minute intervals.
A partition is treated as closed four minutes after creation: “We ’lock’ partitions 4min after their creation, i.e. the 15:00:00 partition should stop receiving updates at 15:04:00.”
The propagation cron runs every minute and only looks at partitions older than LANGFUSE_EXPERIMENT_EVENT_PROPAGATION_PARTITION_DELAY_MINUTES, which defaults to 10.
That cron has a global concurrency of one across the cluster and processes a single partition per invocation, so a backlog makes the delay grow rather than recover.

The way out of the delay is not to lower that variable, it is to have clients write over OTLP with direct writes, which is the real-time route. The header that enables it is x-langfuse-ingestion-version: 4, sent automatically by the Python SDK from 4.0.0 and the JavaScript one from 5.0.0, and a value above 4 is rejected with a 400.

The forty-eight-hour TTL

This is the frightening one, and it appears in no migration guide.

The staging table carries TTL toDateTime(s3_first_seen_timestamp) + INTERVAL 48 HOUR with ttl_only_drop_parts = 1. If the propagator stops for more than forty-eight hours, a long bank holiday for instance, those partitions delete themselves. The data is already in the legacy tables, so an installation in dual keeps working, but the hole in events_full is permanent, and no code detects it and none repairs it. The health probe detects stuck, not lost.

The defence is an alert, and one exists for this. The worker health endpoint accepts ?failIfEventPropagationStuck=true and returns 503 if the propagator’s heartbeat has not been refreshed for more than LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES, which defaults to 35. The comment carries a condition to read before putting it in a Kubernetes probe:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

Another detail of the same mechanism: the propagation cursor lives in Redis, not in Postgres. If Redis is flushed, the cursor resets and every live partition of the staging table is reprocessed.

What can and cannot be undone

The ClickHouse down migrations exist, and they are honest about what they do. The one for 0039 warns in capitals:

WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this table may predate this migration […] Rolling back drops that data regardless of which path created the table.

The ones for the dropped tables recreate the schema and nothing else: “Any data held in event_log before the up migration dropped it is NOT restored.”

But the real point of no return is not in ClickHouse, it is in Postgres. There is a Prisma migration called 20260723150000_drop_legacy_tracing_tables that does exactly what it says:

DROP TABLE IF EXISTS "traces";
DROP TABLE IF EXISTS "observations";
DROP TABLE IF EXISTS "scores";

Prisma Migrate has no down migrations. It applies, and that is that.

Summarising the real reversibility: the write mode can go down from events_only to dual whenever you want, and that is the panic button. What does not come back is what was not written while you were up there: the legacy tables receive nothing retroactively. And step 5 of the chain, the one that drops the scratch tables, ships disabled precisely so you can do forensics if something went wrong.

One last note on the ground you are standing on. The comment in the export-source policy drops something worth reading twice: “legacy-writes-disabled, which in practice only surfaces on self-hosted (Cloud does not run events_only)”. The events_only path is the one you will use and not the one the vendor’s managed service uses. It is less battle-tested in production than dual.

How this looks in code

The minimum runbook, in variables:

# Step 2: upgrade to v4 behaving like v3
LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy
LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=dual_write # "direct" will not start here
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=false

# Step 3: third rail. Both services, web and worker, together.
LANGFUSE_MIGRATION_V4_WRITE_MODE=dual
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=true
LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES=35

# Step 4: only once all five background_migrations rows have finishedAt
LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only
LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true # without this it will not start
LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES=true # last of all

And the query that answers the only question that matters for weeks:

SELECT name, started_at, finished_at, failed_at, failed_reason,
 state->>'chunksLoaded' AS chunks
FROM background_migrations
WHERE name LIKE '20260701_v4_%'
ORDER BY name;

Migration checklist

Every v3 background migration is finished before the image is upgraded.
There is a Postgres and a ClickHouse backup from before the first DROP TABLE.
The write mode has the same value in the worker and in the web, and both change together.
The upgrade went to legacy first, and the service was verified unchanged.
There is an alert on ?failIfEventPropagationStuck=true, with initialDelaySeconds of at least 60 seconds.
Somebody looks at the background_migrations table every day while the backfill runs.
The in-house integrations calling the twenty-four routes that will return 404 have been inventoried.
Client SDKs are on Python 4.0.0 or JavaScript 5.0.0 at minimum before moving to events_only.
Step 5 of the chain is enabled last, and only after verifying the backfill.

Traps

The default is the end state. A v3 installation booting v4 without touching the variable lands in events_only and stops writing the old tables from the first second.

A failed background migration does not block the next one. And the first two steps of the chain declare no predecessor.

More worker replicas do not speed up the backfill. One migration at a time across the cluster, by design.

Forty-eight hours of stopped propagator is a permanent hole. No detection and no repair.

New indexes do not cover old data. Historical searches scan until somebody materialises the indexes by hand.

On a cluster, every ALTER waits for all replicas. One down replica hangs the migration.

Classic ingestion rejects per event. A mixed batch returns 207 and half of it is lost without anyone reading the response body.

History and the live path do not produce quite the same thing. Trace metadata does not descend to children in the backfill.

The route you will take is not the one the vendor takes. Cloud does not run events_only.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window (this article): the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/env.ts, worker/src/env.ts, web/src/env.mjs, web/src/features/public-api/server/deprecations.ts, web/src/pages/api/public/ingestion.ts, web/src/features/public-api/server/createAuthedProjectAPIRoute.ts, worker/src/backgroundMigrations/, worker/src/features/eventPropagation/, packages/shared/clickhouse/migrations/canonical/, packages/shared/prisma/migrations/.
Langfuse compatibility and migration, checked on 14 September 2026.
OpenTelemetry integration migration to v4, checked on 14 September 2026.

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

Tue, 09 Jun 2026 17:00:00 +0200

Operational companion to The contractor with the master key. That post explains the why and the where: the threat model, the five isolation families, which domain uses each. This one is the how, with commands. If you have not read it, read it first: here I take for granted what the blast radius is, why bwrap runs without root and what Tetragon watches. The procedure comes in two independent tracks, client and cluster, because, as the sibling post argues, the control extrapolates but the primitive gets rewritten.

TL;DR

Two reproducible procedures. Client (workstation): install ai-jail (it wraps bubblewrap), generate the per-project .ai-jail, audit with --dry-run, fix the allowlists with --bootstrap, use --lockdown for anything you do not trust, and leave the agent without git push permission. Cluster (RKE2 with Cilium + Tetragon): put the pod baseline in place (unprivileged securityContext, seccomp: RuntimeDefault, default-deny NetworkPolicy), put the untrusted agent’s pod in a microVM with runtimeClassName: kata, and deploy the Tetragon TracingPolicy files in two phases: observe with action: Post to raise the baseline, then promote to action: Sigkill on tcp_connect (egress) and security_file_open (secret paths). The golden rule of the Tetragon phase: adopt first, block later; never put a Sigkill into production without having seen the events in observation mode first.

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

ai-jail wraps the sandbox; on Linux it needs bubblewrap separately, on macOS it needs no extra dependency.

# ai-jail (macOS and Linux)
brew tap akitaonrails/tap && brew install ai-jail
# or, with cargo:
cargo install ai-jail
# or, with mise:
mise use -g ubi:akitaonrails/ai-jail

# bubblewrap on Linux (pick your distro)
sudo pacman -S bubblewrap # Arch
sudo apt install bubblewrap # Debian / Ubuntu
sudo dnf install bubblewrap # Fedora

Check the binary is there and that bwrap runs without root (it must not ask for sudo):

ai-jail --version
bwrap --ro-bind / / --unshare-all echo "bwrap ok without root"

If bwrap fails asking for privileges, your kernel has unprivileged user namespaces disabled; enable them (sysctl kernel.unprivileged_userns_clone=1 on older Debian/Ubuntu) before continuing.

A1 — The per-project .ai-jail file

On the first launch inside the project, ai-jail creates a .ai-jail (TOML) that is committable to the repo: any colleague who clones inherits the same policy.

cd ~/Projects/mi-app
ai-jail claude # creates .ai-jail and launches Claude Code inside the sandbox

The generated file:

# .ai-jail — sandbox configuration (commit it to the repo)
command = ["claude"]
rw_maps = [] # extra writable directories
ro_maps = [] # extra read-only directories

Before trusting the sandbox, audit it. --dry-run --verbose prints every mount point, every isolation flag and the full bwrap command, without executing anything:

ai-jail --dry-run --verbose claude

Read the output and confirm three things: that $HOME is mounted as a tmpfs (not the real one), that ~/.ssh, ~/.aws and ~/.gnupg do not appear among the mounts, and that the only writable directory is the project one. If you need an extra directory:

ai-jail --rw-map ~/Projects/shared-lib claude # extra, writable
ai-jail --map /opt/datasets claude # extra, read-only

Other agents, same binary:

ai-jail codex
ai-jail opencode
ai-jail bash # bare shell to debug the sandbox
ai-jail -- python script.py # any command

A2 — Permission allowlists with –bootstrap

--bootstrap generates each agent’s permission configuration, with sensible allow/deny/ask, and takes a backup before overwriting:

ai-jail --bootstrap

What it produces, in summary:

Agent	File	Base policy
Claude Code	`~/.claude/settings.json`	allow: `git status/diff/log`, `ls`, `grep`, `cargo`, `npm`, `python`, `docker compose` · ask: `git push`, `rm`, `docker run` · deny: `rm -rf`, `sudo`, `chmod 777`, `git push --force`
Codex	`~/.codex/config.toml`	`approval_policy = "on-request"`
OpenCode	`~/.config/opencode/opencode.json`	permissions for `bash`, `edit`, `write`

The operational key: git push is under ask, not allow, and git push --force under deny. The agent can commit, branch and rebase locally as much as it wants; none of that touches the remote. (If you use the Claude Code /sandbox, also set "allowUnsandboxedCommands": false to close the dangerouslyDisableSandbox escape hatch, which is opt-out out of the box.)

A3 — Lockdown for anything you do not trust

To audit third-party code or run an agent over a project you do not know, --lockdown goes further: project mounted read-only, GPU/Docker/display disabled, --rw-map/--map ignored, $HOME a pure tmpfs with no host dotfiles, network cut with --unshare-net and the environment wiped with --clearenv.

ai-jail --lockdown bash

It is the most restrictive sandbox possible short of a VM. Use it as your mental default for anything that is not your own code on your own machine.

A4 — The safety net: git with no push

It is not a flag, it is a property of the environment that changes the risk arithmetic. If the project is in git with a remote, and the agent does not have push permission, the worst case, that it corrupts every file in the project, is reverted with:

git checkout . # back to the last commit
# and if it touched .git (unlikely): delete the dir and re-clone

The remote was never touched. Sandbox for the filesystem + git for the code + manual push is already a reasonable level for daily use: ai-jail protects your data and the system, git protects the code, and the decision to publish is still yours.

Track B — Cluster (RKE2 with Cilium + Tetragon)

The untrusted agent, or the inference that executes generated code, runs as a pod. The same principle as on the client, different primitives. We assume a generic RKE2 cluster with Cilium as CNI and Tetragon already deployed (the eBPF agent DaemonSet on every node).

B0 — The pod baseline

Before any eBPF, the standard kit. Unprivileged securityContext, read-only root, default seccomp:

apiVersion: v1
kind: Pod
metadata:
 name: ai-agent
 namespace: agentes
 labels:
 app: ai-agent
spec:
 securityContext:
 runAsNonRoot: true
 runAsUser: 10001
 seccompProfile:
 type: RuntimeDefault
 containers:
 - name: agent
 image: registry.interno/ai-agent:pinned
 securityContext:
 allowPrivilegeEscalation: false
 readOnlyRootFilesystem: true
 capabilities:
 drop: ["ALL"]
 volumeMounts:
 - { name: work, mountPath: /work }  # the only writable one
 volumes:
 - name: work
 emptyDir: {}

And the default egress cut, the cluster twin of --unshare-net. A default-deny egress NetworkPolicy in the namespace, opening only DNS and the bare essentials:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
 name: default-deny-egress
 namespace: agentes
spec:
 podSelector: {}
 policyTypes: ["Egress"]
 egress:
 - to:
 - namespaceSelector:
 matchLabels: { kubernetes.io/metadata.name: kube-system }
 ports:
 - { protocol: UDP, port: 53 }
 - { protocol: TCP, port: 53 }

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

For genuinely untrusted code, take it out of the shared kernel. With Kata deployed there is a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
 name: kata
handler: kata

And the pod asks for it with one line, runtimeClassName: kata, running in its own microVM with a dedicated kernel instead of sharing the node’s:

spec:
 runtimeClassName: kata  # ← the pod runs in a microVM, not on the node's kernel
 # ...the rest the same as B0

It is the cluster twin of isolation by construction: a kernel exploit inside the pod does not reach the node.

B2 — Tetragon, observation phase (Post)

Now the layer that separates a platform with runtime visibility from one without. Observe first, never kill from the outset. A TracingPolicyNamespaced, scoped to the namespace and to the agent’s label, that reports (does not kill) three things: process executions, network connections and opens of sensitive paths. action: Post only emits the event.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-observa
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- outbound connections ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchActions:
 - action: Post
 # --- opens of sensitive files ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets"
 - "/work/.git/config"
 matchActions:
 - action: Post

(Process executions need no kprobe: Tetragon emits process_exec/process_exit natively.) Deploy it and watch the events live from the node’s Tetragon pod:

kubectl apply -f agente-observa.yaml
# readable events, filtered by namespace:
kubectl exec -n kube-system ds/tetragon -c tetragon -- \
 tetra getevents -o compact --namespace agentes

Leave this running for a typical working day of the agent. Note which destinations it really connects to (your internal registry, your HF mirror, your vLLM endpoint) and which paths it opens. That is your baseline: the list of what is legitimate. Without this step, a Sigkill kills good work and generates an availability incident, precisely what ENS asks you to avoid.

B3 — Tetragon, enforcement phase (Sigkill)

With the baseline in hand, promote to blocking. Two rules. The first: kill any connection whose destination is not on the allowlist — NotDAddr inverts the match: it fires for everything that is not those networks. The second: kill any attempt to open a secrets path.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicyNamespaced
metadata:
 name: agente-enforce
 namespace: agentes
spec:
 podSelector:
 matchLabels:
 app: ai-agent
 kprobes:
 # --- egress: kill everything that is NOT on the allowlist ---
 - call: "tcp_connect"
 syscall: false
 args:
 - index: 0
 type: "sock"
 selectors:
 - matchArgs:
 - index: 0
 operator: "NotDAddr"
 values:
 - "127.0.0.1"
 - "10.0.0.0/8" # internal cluster network
 - "172.16.10.20" # internal registry (example)
 matchActions:
 - action: Sigkill
 # --- reading secrets: kill the process ---
 - call: "security_file_open"
 syscall: false
 args:
 - index: 0
 type: "file"
 selectors:
 - matchArgs:
 - index: 0
 operator: "Prefix"
 values:
 - "/var/run/secrets/kubernetes.io/serviceaccount/token"
 - "/work/.ssh"
 matchActions:
 - action: Sigkill

kubectl apply -f agente-enforce.yaml

Now the agent can do whatever it likes inside the pod, but the instant it tries to connect to a disallowed destination or read the service account token, Tetragon kills it in the kernel, before the packet leaves or the read returns bytes. It is the cluster twin of the curl blocklist and the unmounted ~/.ssh, but applied at runtime and over any binary, not just the ones you know about.

Operational warning. Enforcement with Sigkill requires a recent kernel with eBPF support for the action (5.10+ is safe). Deploy agente-enforce in a test namespace first, and keep agente-observa active in parallel: if the block fires, the Post event tells you exactly what triggered it. Adopt first, block later.

The client ↔ cluster equivalence table

The same vector, the two primitives. This is “extrapolating the technology” made explicit:

Threat vector	Client (workstation)	Cluster (RKE2)
`$HOME` / writable root	`$HOME` as ephemeral tmpfs (`bwrap`)	`readOnlyRootFilesystem: true` + `emptyDir`
Arbitrary egress	`curl`/`wget` blocklist · `--unshare-net`	default-deny NetworkPolicy + Tetragon `NotDAddr`→`Sigkill`
Reading secrets	`~/.ssh`/`~/.aws`/`~/.gnupg` not mounted	secrets outside the pod + Tetragon `security_file_open`→`Sigkill`
Kernel escape	Landlock (2nd VFS barrier)	`runtimeClassName: kata` (microVM, own kernel)
No escape hatch	process inside `bwrap`, no way out	no `privileged`, `drop ALL`, `allowPrivilegeEscalation:false`
Damage to the code	git remote with no `push` → `git checkout .`	GitOps + PR review, the agent does not apply to `main`
Visibility	`--dry-run --verbose` (static, pre-run)	Tetragon `tetra getevents` (dynamic, at runtime)

Checklist of gotchas

Do not put in a Sigkill without passing through Post. The observation baseline is not optional: it is what separates “blocking a C2” from “killing your own fine-tuning job”.
The .ai-jail gets committed; secrets do not. The TOML is policy, not credentials. Check you are not putting paths holding sensitive data into rw_maps.
readOnlyRootFilesystem breaks apps that write to /tmp. Mount an emptyDir at /tmp as well as the work one.
A NetworkPolicy with no DNS rule leaves the pod blind. Open port 53 to kube-system or nothing resolves.
Kata is not free. It adds startup latency and not every workload with special devices (GPU passthrough) fits; reserve it for the untrusted, not for everything.
The Claude Code /sandbox does not cover MCP or hooks unless you enable sandbox-runtime. If your agent uses MCP servers, assume they run with full permissions until you do.
NotDAddr with literal IPs ages badly. Document the allowlist and review it when the registry or the inference endpoint changes; consider stable internal CIDRs instead of loose IPs.

References

ai-jail (Fabio Akita), GPL-3.0: https://github.com/akitaonrails/ai-jail
bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io
Tetragon — TracingPolicy: https://tetragon.io/docs/concepts/tracing-policy/
Tetragon — enforcement (Sigkill/Override): https://tetragon.io/docs/concepts/enforcement/
Kata Containers — Kubernetes RuntimeClass: https://katacontainers.io
Kubernetes — Pod Security & seccomp: https://kubernetes.io/docs/tutorials/security/seccomp/
Kubernetes — Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
Cilium: https://cilium.io

QLoRA runbook: from dataset to adapter served in multi-LoRA (operational procedure)

Tue, 09 Jun 2026 03:00:00 +0000

This is the operational companion to QLoRA and multi-LoRA at the limit on small models. That post takes apart the why, NF4, double quantisation, paged optimizers, the adapter maths; this is the how, with commands you copy and paste. If you have not read the fundamentals one, read it first: here we assume you know what an adapter is, why the base lives in 4-bit and why the gradient only touches the adapter.

TL;DR

A reproducible procedure in five phases: (1) pin the environment with fixed versions; (2) prepare the dataset in chat format; (3) train the QLoRA adapter with TRL + PEFT on an RTX 4090 (24 GB, Ada Lovelace) using gradient checkpointing, gradient accumulation and paged_adamw_8bit; (4) validate and version the adapter as an artefact of megabytes; (5) serve it in vLLM with --enable-lora, loading it hot without restarting the server and resolving it from object storage. All on-premise, on consumer hardware, without a single data item leaving the perimeter. What follows are the exact commands and the memory budget that separates “it fits” from “OOM”.

The end-to-end flow

Phase 0 — Environment and versions

QLoRA is sensitive to the versions of bitsandbytes, transformers, peft and trl: misaligned combinations give dequant errors or adapters that do not load in vLLM. Pin the environment and do not touch it mid-campaign. Reference versions as of June 2026 (check the specific ones in your index; the exact pin matters less than the coherence between them):

python -m venv .venv && source .venv/bin/activate
pip install --upgrade pip

# Training (producer)
pip install "torch>=2.4" \
 "transformers>=4.50" \
 "peft>=0.14" \
 "trl>=0.15" \
 "bitsandbytes>=0.45" \
 "accelerate>=1.2" \
 "datasets>=3.2"

# Serving (consumer) — in its own environment/image
pip install "vllm>=0.8"

What each piece does and why it is pinned:

Package	Role in the flow	Why the version matters
`torch`	tensor runtime and CUDA kernels	the CUDA ABI has to match the driver and `bitsandbytes`; a major jump breaks the 4-bit kernels.
`transformers`	loads the base, the tokenizer and the `chat_template`	it has to know the architecture of the SLM you use; a new model needs a version that supports it.
`peft`	implements LoRA/QLoRA: injects the `A,B` matrices and writes the `adapter_config.json`	that `adapter_config.json` is what vLLM reads when serving; old versions write fields the serving side does not understand.
`trl`	the `SFTTrainer`: the supervised training loop	it integrates `peft` natively; its API (`SFTConfig`) changes between versions, hence the pin.
`bitsandbytes`	the NF4 quantisation and `paged_adamw_8bit`	the most sensitive piece: a badly compiled binary gives corrupt dequant or hangs on the first step.
`accelerate`	orchestrates device, mixed precision and `device_map`	the silent backend of almost everything; misaligning it with `transformers` gives cryptic errors.
`datasets`	loads the JSONL (and allows streaming if the corpus is large)	not very sensitive; any recent 3.x will do.
`vllm`	the multi-LoRA serving	separate environment or image: do not mix its stack with the training `bitsandbytes`.

The golden rule: coherence between the four above (transformers, peft, trl, bitsandbytes) matters more than the exact number of each. Pin them when starting a campaign and do not move them until it closes.

Check that the GPU and CUDA are healthy before starting; a badly compiled bitsandbytes shows up late:

python -c "import torch, bitsandbytes; print(torch.cuda.get_device_name(0), torch.cuda.is_available())"
nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv

For 100 % sovereignty: download the base once from your internal Hugging Face mirror (or a MinIO with the weights) and export HF_HOME to a local volume. Nothing in this flow needs to leave the perimeter.

Phase 1 — Preparing the dataset

The canonical format for a conversational task is JSONL, one conversation per line, with the model’s chat template. Do not invent your own format: use the base tokenizer’s chat_template, because any mismatch between how you train and how you serve degrades quality silently.

{"messages":[{"role":"system","content":"You are a network support assistant."},{"role":"user","content":"The north wing AP does not come up after the outage."},{"role":"assistant","content":"First confirm the PoE on the port..."}]}
{"messages":[{"role":"user","content":"Generate the VLAN change for customer 42."},{"role":"assistant","content":"interface GigabitEthernet0/3\n switchport access vlan 42..."}]}

What each field is and why:

Field	What it is	Operational note
`messages`	the complete conversation, a list of turns	one conversation per JSONL line; it is what `apply_chat_template` turns into tokens.
`role`	who is speaking: `system`, `user`, `assistant`	the adapter learns to produce the `assistant` turns; the `user`/`system` ones are context, not target.
`content`	the text of the turn	the `system` one sets the persona/task; keep it identical to the one you will use in production or the adapter drifts out of alignment.

Operational rules that save grief: watch the ratio of examples (a well-curated narrow-task dataset of 2,000-20,000 examples performs better than 200,000 noisy ones), deduplicate, and set aside 5-10 % as a validation split that does NOT go into training. Building the corpus from production signal is covered by Retrain: closing the loop.

Phase 2 — The training script

A minimal, complete script with TRL + PEFT. It trains an r=8 adapter on an 8B SLM quantised to NF4. Each block has its rationale commented.

# train_qlora.py
import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

BASE = "Qwen/Qwen3-8B" # or whichever SLM you serve; ALWAYS use the same one in train and serve
OUT = "adapters/soporte-redes-v1"

# 1) Base frozen and quantised to 4-bit NF4 with double quantisation
bnb = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_quant_type="nf4", # NormalFloat, quantile-optimal for Gaussian weights
 bnb_4bit_use_double_quant=True, # quantises the scale constants -> ~0.37 bits/param less
 bnb_4bit_compute_dtype=torch.bfloat16 # the matmuls run in BF16 after dequant on the fly
)

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
 BASE, quantization_config=bnb, torch_dtype=torch.bfloat16, device_map={"": 0}
)

# 2) The adapter: low rank, attention projections only (aggressive). Raise target_modules if the eval asks for it.
peft_cfg = LoraConfig(
 r=8, lora_alpha=16, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM",
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
)

ds = load_dataset("json", data_files={"train": "data/train.jsonl",
 "eval": "data/eval.jsonl"})

# 3) Training config designed to fit in 24 GB
cfg = SFTConfig(
 output_dir=OUT,
 per_device_train_batch_size=1, # small real batch
 gradient_accumulation_steps=16, # EFFECTIVE batch = 1*16 = 16, without paying its VRAM all at once
 gradient_checkpointing=True, # recomputes activations in backward: trades compute for memory
 optim="paged_adamw_8bit", # paged optimizer: the airbag against VRAM peaks
 learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03,
 num_train_epochs=3, bf16=True,
 max_length=2048, # bounds the sequence: activations scale with it
 logging_steps=10, eval_strategy="steps", eval_steps=100, save_steps=200,
 report_to="none",
)

trainer = SFTTrainer(model=model, args=cfg, peft_config=peft_cfg,
 train_dataset=ds["train"], eval_dataset=ds["eval"],
 processing_class=tok)
trainer.train()
trainer.save_model(OUT) # saves ONLY the adapter (MB), not the base

`BitsAndBytesConfig` — how the base is quantised

Option	What it does	Why this value / when to change it
`load_in_4bit=True`	loads the base weights in 4-bit	it is the foundation of QLoRA: without it the 8B does not even fit to train.
`bnb_4bit_quant_type="nf4"`	uses the NF4 format (quantile-optimal for Gaussian weights)	`"fp4"` exists, but NF4 performs better on transformer weights; leave NF4.
`bnb_4bit_use_double_quant=True`	quantises the scale constants themselves	saves ~0.37 bits/param (hundreds of MB on an 8B); the margin separating “it fits” from “OOM”. Leave it at `True`.
`bnb_4bit_compute_dtype=torch.bfloat16`	matmul precision after undoing the quantisation on the fly	BF16 on Ada/Hopper (4090, H100); use `float16` only on GPUs without BF16.

`LoraConfig` — the shape of the adapter

Option	What it does	Why this value / when to change it
`r=8`	adapter rank: its correction capacity	4-8 for a narrow task (aggressive); raise it to 16-64 only if the eval shows underfitting.
`lora_alpha=16`	scale factor of the delta (effective `α/r`)	common convention `α=2r`; it modulates how much the adapter “weighs” on the base.
`lora_dropout=0.05`	regularisation on the adapter	0.05-0.1 with small datasets (avoids overfit); 0 if the corpus is large.
`bias="none"`	does not train the bias terms	`"none"` is the standard; `"all"`/`"lora_only"` rarely help and cost params.
`task_type="CAUSAL_LM"`	objective/head type	fixed for a generative LLM.
`target_modules=[q,k,v,o]`	which matrices get an adapter	attention only = cheap and aggressive; add `gate_proj`/`up_proj`/`down_proj` (MLP) if the task demands rewriting more behaviour and the eval asks for it.

`SFTConfig` — the memory budget and the loop

Option	What it does	Why this value / when to change it
`per_device_train_batch_size=1`	microbatch per GPU	1 on 24 GB; the real batch is built by `gradient_accumulation_steps`.
`gradient_accumulation_steps=16`	accumulates 16 microbatches before updating	effective batch = 1×16 = 16 without paying its VRAM all at once; raise it if you shorten the sequence and want a bigger effective batch.
`gradient_checkpointing=True`	recomputes activations in the backward pass instead of storing them	essential on a 4090: ~20-30 % slower in exchange for much less VRAM.
`optim="paged_adamw_8bit"`	Adam optimizer in 8-bit + states pageable to RAM	less state VRAM and the airbag that avoids the OOM at the peaks.
`learning_rate=2e-4`	learning rate of the adapter	1e-4-3e-4 is the typical QLoRA range; adapters tolerate a higher LR than a full fine-tune.
`lr_scheduler_type="cosine"`	LR decay curve	`cosine` or `linear`; cosine usually gives a smooth drop at the end.
`warmup_ratio=0.03`	warms the LR over the first 3 % of steps	avoids the instability of the first steps.
`num_train_epochs=3`	complete passes over the dataset	1-3; watch the eval loss so as not to overfit.
`bf16=True`	compute and adapter precision	BF16 on Ada/Hopper; `fp16=True` if your GPU has no BF16.
`max_length=2048`	maximum sequence length	the #1 VRAM lever for activations: shorten it first if there is an OOM.
`eval_strategy`/`eval_steps`/`save_steps`	validation and checkpoint cadence	tune them to the dataset size; evaluating often costs time.

The four pieces that make it fit on a 4090 are: per_device_train_batch_size=1 + gradient_accumulation_steps (a large effective batch without its memory cost all at once), gradient_checkpointing=True (recomputing activations instead of storing them) and optim="paged_adamw_8bit" (paging states to RAM at the peaks). Remove any of the three with long sequences and you will see the OOM.

A declarative alternative with Axolotl if you prefer YAML over Python (same result):

base_model: Qwen/Qwen3-8B
load_in_4bit: true
adapter: qlora
lora_r: 8
lora_alpha: 16
lora_target_modules: [q_proj, k_proj, v_proj, o_proj]
sequence_len: 2048
micro_batch_size: 1
gradient_accumulation_steps: 16
gradient_checkpointing: true
optimizer: paged_adamw_8bit
learning_rate: 0.0002
num_epochs: 3
bf16: true
datasets:
 - path: data/train.jsonl
 type: chat_template

Phase 3 — Launch and monitor

# Simple launch on one GPU
python train_qlora.py

# In another terminal: watch VRAM. If it gets close to the ceiling, lower max_length or raise grad accumulation.
watch -n 2 nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu --format=csv

Approximate VRAM budget when training the 8B on the 4090, and what to touch when it gets tight:

Component	Approx. VRAM	Lever if there is an OOM
8B NF4 base (frozen)	~4.0 GB	— (fixed)
Adapter + grad + Adam states	~0.3-0.7 GB	lower `r`
Activations (batch × sequence)	~6-14 GB	lower `max_length`, `batch_size`; raise `grad_accum`
Dequant buffers / workspace	~1-2 GB	—

Table of quick OOM remedies, in order of cost: (1) lower max_length; (2) confirm gradient_checkpointing=True; (3) raise gradient_accumulation_steps and lower per_device_train_batch_size to 1; (4) use paged_adamw_8bit (already in the script); (5) as a last resort lower r. If after all that it still does not fit, the sequence or the model is too large for 24 GB: either you bound it, or you move up the hardware.

Phase 4 — Validating the adapter

Never promote an adapter on the strength of the training loss. Measure against the validation split you set aside and against a handful of real prompts.

# quick_eval.py
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
 bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B", quantization_config=bnb, device_map={"": 0})
model = PeftModel.from_pretrained(base, "adapters/soporte-redes-v1") # base + adapter

msgs = [{"role": "user", "content": "The north wing AP does not come up after the outage."}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(0)
print(tok.decode(model.generate(ids, max_new_tokens=256)[0], skip_special_tokens=True))

For a serious verdict, run the adapter through your eval suite (the layer described by LLM evals) and compare against the base without an adapter and against the previous version of the adapter. Promote only if it wins on the task metric without regressing on safety/format.

Phase 5 — Versioning the adapter as an artefact

The adapter is a pair of files of a few MB (adapter_model.safetensors + adapter_config.json). Treat it as a versioned, signed, traceable artefact, not as a loose file.

# Reproducible checksum + upload to internal object storage (MinIO/S3)
sha256sum adapters/soporte-redes-v1/adapter_model.safetensors > adapters/soporte-redes-v1/SHA256
aws --endpoint-url https://minio.interno s3 cp \
 adapters/soporte-redes-v1/ s3://adapters/soporte-redes/v1/ --recursive

A convention that works: s3://adapters/<task-or-customer>/<version>/. Immutable per version, with its SHA256. Deleting a customer is deleting a prefix of a few MB, not retraining anything. Versioning 500 adapters costs what versioning 500 heavy configuration files costs.

Phase 6 — Serving in multi-LoRA with vLLM

The consumer loads one shared base and applies the adapter delta per request. Startup with static adapters declared:

VLLM_ALLOW_RUNTIME_LORA_UPDATING=True \
vllm serve Qwen/Qwen3-8B \
 --enable-lora \
 --max-loras 8 \  # max nº of DIFFERENT adapters per batch (not the total loadable)
 --max-lora-rank 8 \  # = the maximum rank of your adapters; do not inflate it (wastes memory)
 --max-cpu-loras 64 \  # adapters cached in RAM for fast swap to VRAM
 --lora-modules soporte-redes=/srv/adapters/soporte-redes/v1

Each flag, what it controls and how to size it:

Flag / variable	What it controls	How to size it
`--enable-lora`	turns on adapter support	mandatory; without it, vLLM ignores any `model` that is an adapter.
`--max-loras 8`	nº of different adapters in a single batch	more adapters per batch makes the SGMV kernels more expensive; 8-32 is reasonable. It is not the total loadable.
`--max-lora-rank 8`	maximum rank the server reserves	set it equal to the real rank of your adapters (8 here); inflating it wastes VRAM and performance.
`--max-cpu-loras 64`	adapters cached in RAM ready to page to VRAM	≥ nº of active adapters; it is the “bench” from which fast swapping happens.
`--lora-modules name=path`	declares static adapters at startup	useful for the fixed ones; leave it out if everything goes through dynamic loading/Resolver.
`VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`	enables the hot load/unload endpoints	essential for `/v1/load_lora_adapter`; without it, the server is static.

--max-loras limits the different adapters per batch, not how many you can have loaded; the bulk lives on CPU (--max-cpu-loras) and is paged to VRAM on demand. Set --max-lora-rank to the real rank (8 here): inflating it wastes memory and performance. Requests pick the adapter through the model field:

curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
 "model": "soporte-redes",
 "messages": [{"role":"user","content":"The north wing AP does not come up after the outage."}]
}'
# model:"Qwen/Qwen3-8B" (no adapter) uses the bare base on the same server

Hot loading of a new adapter without restarting (thanks to VLLM_ALLOW_RUNTIME_LORA_UPDATING=True):

curl -X POST http://localhost:8000/v1/load_lora_adapter -H "Content-Type: application/json" -d '{
 "lora_name": "cliente-42",
 "lora_path": "/srv/adapters/cliente-42/v3"
}'
# and to free VRAM/CPU when a customer goes idle:
curl -X POST http://localhost:8000/v1/unload_lora_adapter -H "Content-Type: application/json" -d '{"lora_name":"cliente-42"}'

For multi-tenant at scale, avoid declaring hundreds of adapters by hand: the LoRAResolver resolves and loads the adapter from local storage or S3 the first time an unknown model arrives, so the server stays lean and the adapters are pulled lazily from your MinIO. The internals of how thousands of concurrent adapters are batched (SGMV kernels, unified paging, the heterogeneous gather/scatter) are in Multi-LoRA serving; this runbook only switches them on. To squeeze the base’s decode throughput on a 4090, combine this with Optimising decode in vLLM.

Serving multi-adapter vs merging per task

Two deployment architectures, and the procedure changes:

Serving multi-LoRA (the above). A shared base + N hot adapters. It is the default sovereign pattern: minimum footprint, per-customer isolation, hot-swap. Use standard QLoRA and merge nothing.

Merging per task. If you want a single quantised-and-adapted artefact per task (no adapter at runtime), do not merge a standard QLoRA adapter into the 4-bit base: the merge reintroduces precision that NF4 does not represent and on requantising you lose part of what was learnt. For that case, train with QA-LoRA (quantization-aware), which merges cleanly onto a quantised base. It is an architecture decision, not a quality one; the conceptual detail is in the fundamentals post.

Checklist of operational gotchas

Coherent chat template between training and serving. The most common and most silent mismatch: you train with one chat_template and serve with another. Use the base’s on both sides.
Exactly the same base (revision included) in train and serve. An adapter trained on Qwen3-8B is not valid on another revision of the model.
--max-lora-rank ≥ the rank of ALL the adapters served together, but no more: inflating it wastes VRAM.
KV budget vs --max-loras. The bottleneck in serving is not the adapters (MB), it is the KV cache and concurrency; see Inverted roofline for the SLM regime.
r too low = underfitting if the task demands rewriting a lot of behaviour. Raise r only if the eval asks for it.
Do not promote on training loss. Validate against the reserved split + real prompts + a safety regression.
Version and make immutable every adapter with its SHA256; never overwrite a served version.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB) the same machine is producer and consumer: you train a customer’s adapter in hours and serve it on the same server on top of the shared base. It is the canonical case for multi-tenant demos and platform prototypes.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8) QLoRA stops being necessary in order to fit, but it serves to parallelise production (several adapter jobs at once) and to keep the quantised format consistent between training and serious serving of hundreds of concurrent adapters. The base can run in native FP8; the mechanics of the runbook do not change, only the scale.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hugging Face TRL — PEFT integration (SFTTrainer + QLoRA): https://huggingface.co/docs/trl/peft_integration
Hugging Face PEFT: https://github.com/huggingface/peft
bitsandbytes: https://github.com/bitsandbytes-foundation/bitsandbytes
vLLM — LoRA Adapters (serving, dynamic loading, LoRAResolver): https://docs.vllm.ai/en/stable/features/lora/
Axolotl: https://github.com/axolotl-ai-cloud/axolotl
Xu, Y. et al. QA-LoRA: Quantization-Aware Low-Rank Adaptation. ICLR 2024. https://arxiv.org/abs/2309.14717

Runbook on lo0 — Tech Blog

Langfuse v4, day 2 (5 of 10): the worker's thirty-nine queues, and the three that decide whether you lose data

TL;DR

You are here: OBSERVE, day 2

The analogy: the sorting hub

The map of the thirty-nine

The sharding that does not shard

Concurrencies, and the ones set to one on purpose

An event’s path, from API to ClickHouse

Retries, and the bin nobody looks at

The metrics, with their exact names

The two probes, and the one that can restart you in a loop

The crons

Splitting roles, which is what all this is for

Checklist

Traps

The series: the ten articles

See also

Sources

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

TL;DR

You are here: OBSERVE, day 2

The analogy: changing the track gauge without closing the line

The three-rung ladder

The three combinations that refuse to start

What breaks in each mode

The three steps, in order

The ClickHouse migrations backfill nothing

The five-step chain that does backfill history

The failure that does not block

Where the ten minutes come from

The forty-eight-hour TTL

What can and cannot be undone

How this looks in code

Migration checklist

Traps

The series: the ten articles

See also

Sources

Runbook: caging the AI agent — bubblewrap on the client, Tetragon on the cluster

TL;DR

The flow of the two tracks

Track A — Client (the developer’s workstation)

A0 — Install ai-jail and bubblewrap

A1 — The per-project .ai-jail file

A2 — Permission allowlists with –bootstrap

A3 — Lockdown for anything you do not trust

A4 — The safety net: git with no push

Track B — Cluster (RKE2 with Cilium + Tetragon)

B0 — The pod baseline

B1 — Kata RuntimeClass: the untrusted pod in its own microVM

B2 — Tetragon, observation phase (Post)

B3 — Tetragon, enforcement phase (Sigkill)

The client ↔ cluster equivalence table

Checklist of gotchas

See also

References

QLoRA runbook: from dataset to adapter served in multi-LoRA (operational procedure)

TL;DR

The end-to-end flow

Phase 0 — Environment and versions

Phase 1 — Preparing the dataset

Phase 2 — The training script

BitsAndBytesConfig — how the base is quantised

LoraConfig — the shape of the adapter

SFTConfig — the memory budget and the loop

Phase 3 — Launch and monitor

Phase 4 — Validating the adapter

Phase 5 — Versioning the adapter as an artefact

Phase 6 — Serving in multi-LoRA with vLLM

Serving multi-adapter vs merging per task

Checklist of operational gotchas

Applied to on-premise infrastructure

See also

References

`BitsAndBytesConfig` — how the base is quantised

`LoraConfig` — the shape of the adapter

`SFTConfig` — the memory budget and the loop