--- title: "LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard" url: https://blog.lo0.es/en/posts/litellm-langfuse-operational-pair/ markdown: https://blog.lo0.es/en/posts/litellm-langfuse-operational-pair/index.md lang: en date: 2026-09-07 lastmod: 2026-09-13 author: "José Manuel Román" description: "The gateway and the tracing backend are installed in one afternoon and operated for years. This post walks through the seam between LiteLLM and Langfuse as of September 2026: the three integration routes and the one that expires on 16 November, why the cost comes out as zero on a self-hosted model and how to fix it, how the client's traceparent is propagated (and broken) on its way to the vLLM engine, the four serial queues where events are silently dropped, and what can honestly be promised to an auditor about best-effort data." tags: ["litellm","langfuse","observability","opentelemetry","gateway","vllm","finops","llmops","on-premise","operations"] translations: es: https://blog.lo0.es/posts/litellm-langfuse-par-operativo/index.md --- # LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard > This post opens the operational track of the control layer. The previous articles left the pieces chosen and assembled: [choosing the OSS gateway](https://blog.lo0.es/en/posts/choosing-oss-gateway-llm-inference/index.md) and [the L7 inference router](https://blog.lo0.es/en/posts/llm-inference-router-l7-switchboard/index.md) explain why LiteLLM ends up in front of the fleet, and [Langfuse from the inside](https://blog.lo0.es/en/posts/langfuse-inside-sorting-centre-bottleneck/index.md) takes apart the backend that receives the traces. Here we deal with the seam between the two, which is where the work of the next three years lives. ## TL;DR The LiteLLM plus Langfuse pair is configured with a list of callbacks and two API keys, and from there four matters decide whether the dashboard is good for operating or only for showing in a demo. **There are three integration routes and they are not equivalent.** The native `langfuse` callback is tied to the Python SDK v2 (`langfuse>=2.59.7,<3.0` in LiteLLM's `pyproject.toml`), which writes against the legacy ingestion that Langfuse Cloud retires on **16 November 2026**. The `langfuse_otel` preset exports over OTLP and is the path recommended in the current documentation of both projects. The OTel v2 route, behind the `LITELLM_OTEL_V2=true` flag, changes span names, leaves the success status at `UNSET` and, above all, its Langfuse mapper does not read the request metadata: `session_id`, `trace_user_id`, `tags` and `trace_id` stop arriving. **The cost of a self-hosted model comes out as zero unless a price is registered.** `get_model_info()` raises an exception for a model that is not in the pricing map, `completion_cost()` propagates it, the logger catches it, stores it in a debugging field and returns `None`, and the payload turns it into `0.0`. There is no warning at the default log level. A team's budget is never exhausted because its spend is always zero. **Client-gateway-engine correlation does not come out of the box.** LiteLLM does continue the client's trace if a `traceparent` arrives, and it also extracts from it the 32-hex trace-id to use as `litellm_trace_id`, which makes the Langfuse identifier equal to the W3C one. Upstream the story changes: `forward_traceparent_to_llm_provider` is `False` by default, the function that applies it returns early if no `otel` logger is loaded, and when it does act it forwards the client's original header, so the vLLM span ends up a sibling of LiteLLM's rather than a child. Add to that the fact that vLLM exports OTLP over gRPC by default and Langfuse does not accept gRPC. **Between the request and the dashboard there are four bounded queues that drop events.** LiteLLM's logging thread pool (100 threads, 10,000 pending tasks, dropping once full), the asynchronous `LoggingWorker` (50,000, and once full it aggressively flushes half), OTel's `BatchSpanProcessor` (2,048 spans by default) and Langfuse's `ClickhouseWriter`, which drops rows after exhausting the retries with no dead-letter queue. None of the four applies back pressure on the request, which is exactly what you want, and none of them guarantees delivery, which is exactly what an auditor asks about. The rest of the article develops the four points with configuration applicable to a generic 4×H100 cluster with vLLM, the overhead figures the project itself publishes, and the honest limit of what this data sustains before ENS or ISO/IEC 42001. ## You are here: the seam between DEPLOY and OBSERVE In the [seven-layer stack](https://blog.lo0.es/en/posts/anatomy-on-premise-llm-inference-stack-seven-layers/index.md), LiteLLM lives in the gateway layer and Langfuse in the observability one. The separation is clean on the diagram and false in operation: the gateway is the only point in the system that sees the complete request before chopping it up and the complete response after reassembling it, so it is also the only reasonable place from which to emit the trace. The inference engine sees tokens and latencies; it does not see who is asking or how much that team has spent this month. That turns LiteLLM into a telemetry producer as well as a traffic distributor, and moves onto the gateway a responsibility that was not in the original design: if the callback fails, the request is served all the same and nobody notices that the dashboard has been incomplete for two days. ## The analogy: the toll booth that issues the ticket A motorway toll booth does two independent jobs. It raises the barrier, which is its visible function, and it issues a ticket with number plate, time, section and amount, which is the function the concession lives off. The two share a booth and do not share a destination: the barrier responds in a hundred milliseconds because there is a car waiting, and the ticket travels to a processing centre over a slower channel that batches thousands of records before sending them. The design is well thought out as long as its consequence is understood. If the processing centre goes down, cars keep going through. If the ticket queue fills up, tickets are thrown away, traffic is not stopped. The concession accepts losing records rather than causing a jam, and charges on a statistical sample it knows to be imperfect. LiteLLM and Langfuse work the same way. The four queues in the corresponding section are the slow channel, and all of them prefer to drop rather than block. Whoever builds the pair inheriting the complete analogy gets the operation right. Whoever thinks the ticket is an accounting entry ends up explaining to an auditor why traces from a Tuesday afternoon are missing. ## Three routes, and one with an expiry date As of September 2026 three paths coexist for getting a request from LiteLLM to Langfuse. LiteLLM's documentation already labels the first as legacy. | Route | How it is enabled | Transport | Status | |---|---|---|---| | Native `langfuse` callback | `success_callback: ["langfuse"]` | Langfuse Python SDK v2, legacy ingestion | Maintained for compatibility. Tied to `langfuse<3.0` | | `langfuse_otel` preset (v1) | `callbacks: ["langfuse_otel"]` | OTLP/HTTP against `/api/public/otel` | Recommended by both projects | | OTel v2 route | `callbacks: ["langfuse_otel"]` + `LITELLM_OTEL_V2=true` | OTLP/HTTP, new mapper | Opt-in. Changes semantics and loses metadata | The date that orders the decision is **16 November 2026**. Langfuse's compatibility matrix marks the legacy batch ingestion of the Python SDK v2 as retired on that date for Cloud, at which point trace ingestion through that route stops working. On self-hosted the endpoint survives, but a v4 server in `events_only` mode returns 400 for every event type except `score-create` and `sdk-log`. Since LiteLLM's native callback is pinned to `langfuse>=2.59.7,<3.0` in its own `pyproject.toml`, and its code calls APIs that only exist in SDK v2 (`self.Langfuse.trace(...)`, `trace.generation(...)`), there is no way to move up an SDK without changing route. Anyone who today has `success_callback: ["langfuse"]` against Langfuse Cloud has a migration with a date on it. Anyone who has it against a self-hosted Langfuse has one with some slack, because v3 receives security patches until the end of January 2027, but they have one all the same. One detail of the OTLP route worth checking in older deployments: the `x-langfuse-ingestion-version: 4` header is the one that enables real-time ingestion. Without it the traces arrive, but with a delay that Langfuse's documentation puts at around ten minutes (the compatibility page says fifteen, and that discrepancy remains unresolved in their own documents). LiteLLM started sending it in version 1.95.0, of August 2026. A proxy older than that version against a Langfuse v4 produces the classic symptom: the dashboard works, but it is always behind and nobody knows why. ## The configuration that works today For a generic 4×H100 cluster with vLLM behind it and Langfuse self-hosted in the same Kubernetes, the OTLP route in its v1 variant is the one that gives the most functionality with the fewest surprises. ```yaml model_list: - model_name: llama-70b litellm_params: model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct api_base: http://vllm-llama70b.inference.svc.cluster.local:8000/v1 api_key: os.environ/VLLM_API_KEY # Price derived from the cluster's €/GPU-hour. Without this, spend = 0.0 input_cost_per_token: 0.00000031 output_cost_per_token: 0.00000089 litellm_settings: callbacks: ["langfuse_otel"] turn_off_message_logging: false # a conscious decision, see the PII section langfuse_default_tags: ["model_group", "user_api_key_team_alias", "cache_hit"] general_settings: disable_error_logs: true # provider errors inflate SpendLogs proxy_batch_write_at: 60 ``` The credentials travel through the environment. `LANGFUSE_PUBLIC_KEY`, `LANGFUSE_SECRET_KEY` and `LANGFUSE_HOST` are enough: the preset builds the `Authorization: Basic base64(public:secret)` header and adds the ingestion version one, so there is no need to touch `OTEL_EXPORTER_OTLP_HEADERS` by hand. The preset's default host points to the US region of Langfuse Cloud, a relevant detail for anyone building a platform with data sovereignty requirements: if the variable is not defined, the telemetry leaves Spain without anyone having decided it. For real multi-tenancy, LiteLLM supports per-team Langfuse credentials (`POST /team//callback` with `callback_vars`) and per-key ones (`metadata.logging[]` when generating the key), and the key's wins over the team's without mixing. There is a limitation that breaks the architecture of anyone who planned to separate tenants across different Langfuse instances: under OTel v2, a per-key or per-team `langfuse_host` does not move that tenant to another host, because the exporter fixes the destination at startup and the only thing that varies per tenant are the headers. And one operational warning with years of history behind it: `MAX_LANGFUSE_INITIALIZED_CLIENTS` is 50 by default because each Langfuse client starts a thread. The comment in LiteLLM's code is explicit about the incident that motivated it, with the CPU at 100% from repeated initialisations. A deployment with per-team credentials and many teams hits that ceiling. ## What is lost when moving to OTel v2 The `LITELLM_OTEL_V2=true` flag brings a step up in quality in the proxy's general instrumentation, and at the same time a concrete regression on the path to Langfuse that the documentation acknowledges without underlining it. The Langfuse mapper of the v2 route emits `langfuse.observation.type`, `langfuse.observation.model.name`, the observation identifier, the model parameters, input and output, `usage_details` and `cost_details`, plus `langfuse.trace.metadata.team_id` and `team_alias`. It does not emit `session.id`, nor `user.id`, nor `langfuse.trace.tags`, nor `langfuse.trace.id`, nor the generation name, nor `parent_observation_id`. LiteLLM's documents put it this way: those attributes are set by the preset from the request and the response, not from metadata supplied by the client. The practical consequence is that an application that today groups conversations by sending `metadata: {"session_id": "...", "trace_user_id": "..."}` in the request body stops grouping them when the flag is enabled, with no error and no warning. The traces keep arriving, loose. The other v2 changes are semantic and affect alerts already written. The root span goes from being called `Received Proxy Server Request` to being named after the route (`POST /v1/chat/completions`), the inference span goes from `litellm_request` to `{operation} {model}`, the guardrail spans stop hanging off the inference span and hang off the root instead, and the success status stays at `UNSET` instead of `OK`. Any rule that fires on `status == OK` stops firing. The way back is to remove the flag and restart. ## Cost: why the trace says zero This is the part that takes the most time to diagnose and the least to fix. LiteLLM computes the cost of a request from a per-model pricing map. For commercial provider models the map comes filled in. For `hosted_vllm/whatever`, which is all there is in an in-house inference factory, the map is empty, and the failure path is as follows: 1. `litellm.get_model_info()` raises an exception with the message `This model isn't mapped yet`. 2. `completion_cost()` and `response_cost_calculator()` propagate the exception. 3. `Logging._response_cost_calculator()` catches it, writes a `StandardLoggingModelCostFailureDebugInformation` into `response_cost_failure_debug_information`, logs it at **debug** level and returns `None`. 4. The standard logging payload does `llm_response_cost = raw_response_cost or 0.0`. The system delivers a spend of zero, with no visible error at the default log level, and writes it into the `spend` column of `LiteLLM_SpendLogs`. The per-key, per-team and per-organisation budgets are evaluated against that zero and are never exhausted. The Langfuse dashboard shows a total cost of zero for everything self-hosted, which is precisely the part you wanted to measure. ### Registering the price The pricing keys are accepted both in `litellm_params` and in `model_info`, and those in `litellm_params` win when they are in both places. The complete set has 91 fields, but for a deployment with vLLM two are enough, and there are three things to know about them. The first is that **the price is per individual token, in USD**. There are no per-thousand variants: searching for `per_1k` or `per_1000` in the pricing model returns nothing. A three-orders-of-magnitude error here is easy to make and hard to see, because it produces plausible figures. The second is that `input_cost_per_request` exists in the schema and **is not applied** in chat calls: a model registered with that field returns a cost of 0.0. To charge per request it has to be converted into a cost per token. The third is that `input_cost_per_second` needs the response time and is computed as price times `response_time_ms / 1000`, so outside the proxy's request path it silently returns zero. The number that goes into those two keys comes from the article on [FinOps and multi-tenancy with LiteLLM](https://blog.lo0.es/en/posts/finops-multi-tenancy-gpu-cluster-who-pays/index.md) and the one on [cost per token and per request](https://blog.lo0.es/en/posts/gpu-hour-to-cost-per-token-on-prem-vs-cloud/index.md): amortisation plus energy divided by tokens served. The example arithmetic, to set the order of magnitude: a cluster whose fully loaded cost is 9.60 USD per hour sustaining 3,000 output tokens per second produces 10.8 million tokens an hour, that is 0.00000089 USD per output token. That is the value that goes into `output_cost_per_token`. If the cluster's average utilisation is 30 % and not 100 %, the divisor changes and the price triples, which is the same lever that article develops. Before sending traffic, the `/spend/calculate` endpoint accepts the same parameters as `completion_cost` and returns the computed cost. It is the cheap way to check that the price resolves. ### How that number reaches Langfuse, and who wins The two integration routes write the cost into different fields. The native callback sends it as `usage.total_cost` of the v2 schema, and duplicates it in the trace metadata as `litellm_response_cost`. The OTLP preset emits `langfuse.observation.cost_details` as a JSON string of the form `{"total": cost}`, and `langfuse.observation.usage_details` with input, output and total. The rule on the Langfuse side is documented and is the right one for this case: when there is ingested cost and inferred cost, **the ingested one takes priority**. Langfuse only computes from its own pricing table when the emitter sends no cost, and changes to a model definition apply to new generations, with no retroactive recalculation. A self-hosted model therefore needs no pricing definition in Langfuse: it is enough for LiteLLM to send `cost_details`. One detail remains that explains a puzzling symptom. Langfuse stores the ingested cost and the computed one in separate columns, `totalCost` and `calculatedTotalCost`, and the session header totals add up only the first. A mixed deployment where part of the traffic brings cost and part lets Langfuse compute it shows sessions at zero while the individual generations show cost. There is an open issue about it. ### The tokens of a streaming response Almost all of an assistant's traffic is streamed, and there the OpenAI `usage` object only appears if the client asks for `stream_options.include_usage`. LiteLLM solves this by injecting the option on its own: on the asynchronous completion routes with `stream: true`, if the caller did not set it and all the candidate deployments support it, it adds it and marks `_litellm_strip_stream_usage` so the usage block is used in the cost calculation and not returned to the client. The `general_settings.always_include_stream_usage` flag forces or disables that behaviour, and is worth setting to `True` when capability detection does not recognise the vLLM deployment. When the usage is missing anyway, LiteLLM reassembles the response with `stream_chunk_builder()` and counts tokens with the tokeniser. It is an approximation, and it produces the second path towards a cost of zero. The two paths to zero are told apart by looking at `total_tokens` in `LiteLLM_SpendLogs`: zero tokens with a correct price is a usage problem, correct tokens with zero cost is a price problem. ## Correlation: from the client to the engine The operational question that justifies the whole setup is being able to take a user complaint, find their request and see where the time went. That requires the trace identifier to survive three hops. > **Diagram:** Propagation of the client's traceparent to the LiteLLM gateway and the vLLM engine, with the points where correlation breaks — [see the HTML version](https://blog.lo0.es/en/posts/litellm-langfuse-operational-pair/) ### What does work If the client sends `traceparent`, LiteLLM's spans hang off the existing trace. In addition, and independently of the OTel subsystem, `litellm_pre_call_utils.py` extracts by regular expression the 32-character hexadecimal trace-id and assigns it to `data["litellm_trace_id"]` as a last resort, and takes the `session_id` from the W3C `baggage` header. That makes the Langfuse trace identifier equal to the W3C trace identifier, which is the most robust correlation key available: with it, an identifier copied from the application log opens the trace in Langfuse with no translation. For the routes that are not OTel v2, moreover, headers prefixed with `langfuse_` inject per-request metadata. `langfuse_trace_id`, `langfuse_trace_user_id` and `langfuse_trace_metadata` are documented, but the code strips the prefix and writes the rest into the metadata, so any body key works as a header. Two warnings about this: header values arrive as strings and are not interpreted as JSON, and Nginx drops headers with underscores by default (`underscores_in_headers off`), which produces the symptom that the metadata works locally and disappears behind the ingress. ### What does not work without effort Towards the engine, propagation is off. The flag is `litellm.forward_traceparent_to_llm_provider`, it is `False`, and its implementation has three behaviours to know about before enabling it: - The function returns immediately if `open_telemetry_logger is None`. A deployment with `callbacks: ["langfuse_otel"]` and without the generic `otel` callback propagates nothing, whatever the flag is set to. - When it does propagate, it copies the client's `traceparent` header as is, instead of injecting the context of LiteLLM's active span. The vLLM span ends up hanging off the client's span-id, a sibling of the gateway call's span. The trace is one, the hierarchy is wrong. - If the client sends no `traceparent`, none is created. Add to that the fact that LiteLLM's general policy is not to forward client headers to the provider, and that the `forward_client_headers_to_llm_api` allowlist covers headers prefixed with `x-`, among which `traceparent` is not. This is the only route. ### The vLLM side, and why the Collector stops being optional vLLM enables traces with `--otlp-traces-endpoint`, and `--collect-detailed-traces` with `model`, `worker` or `all` adds forward and execute times, with a performance impact its own documentation warns about. The attributes it emits are in `vllm/tracing/utils.py` and are the most useful part of the setup: `gen_ai.latency.time_in_queue`, `time_to_first_token`, `time_in_scheduler`, `time_in_model_prefill`, `time_in_model_decode`, `time_in_model_forward`, `time_in_model_execute`, `time_in_model_inference` and `e2e`, plus the token counters. That breakdown is what turns a "it's slow" into "the time goes in the queue, not in decode", which is the conversation you want to have during an incident. It fits with what was already covered in [instrumenting vLLM with OTel](https://blog.lo0.es/en/posts/instrumenting-vllm-otel-measuring-optimisations/index.md) and with the metrics in [DCGM and vLLM](https://blog.lo0.es/en/posts/anatomy-dcgm-vllm-metrics-analogies-documented-anomalies/index.md). There are three constraints that decide the architecture: vLLM exports over gRPC unless `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf` is set, and its gRPC exporter is built with `insecure=True` hard-coded. Langfuse, for its part, does not support gRPC. And vLLM emits the old convention names (`gen_ai.usage.prompt_tokens`, `gen_ai.usage.completion_tokens`), renamed in the specification to `input_tokens` and `output_tokens`, plus a `gen_ai.latency.*` family that was never standard and that the file itself documents as its own attributes until they are normalised. An OpenTelemetry Collector in between solves all three at once: it receives gRPC, exports HTTP with the Basic header towards `/api/public/otel/v1/traces`, and renames attributes with a `transform`. It is also the only place where tail sampling can be done, which is the next point. And add the warning from the llm-d community, which holds for any deployment with Envoy or sidecars in between: `OTEL_SERVICE_NAME` has to be set per engine, because vLLM sends no service name and without it every span arrives as `unknown_service`. ## The four serial queues A request's telemetry crosses four bounded buffers before it is queryable. All four drop once full and none blocks the request. | Queue | Capacity | What it does once full | |---|---|---| | LiteLLM logging thread pool | 100 threads, 10,000 pending tasks | Drops the new ones with a rate-limited warning | | Asynchronous `LoggingWorker` | 50,000 events, concurrency 100 | Aggressively flushes 50 % of the queue; `enqueue()` documents that it never blocks | | OTel's `BatchSpanProcessor` | 2,048 spans (`OTEL_BSP_MAX_QUEUE_SIZE`), batch 512, sent every 5 s | Drops spans silently | | Langfuse's `ClickhouseWriter` | Bounded retries | Drops rows after exhausting retries, with no dead-letter queue | The first is documented in LiteLLM's own code with a sentence that saves half an hour of discussion: logging is best-effort and, once the cap is reached, new entries are dropped with a rate-limited warning instead of being queued forever. The third is a barely visible consequence: LiteLLM builds the `BatchSpanProcessor` without adjusting `max_queue_size`, so the OTel SDK defaults govern, and no LiteLLM documentation mentions the `OTEL_BSP_*` variables that change them. The Langfuse SDK does the same on its own account: it sets batch size and interval, and leaves the queue size at the default. On top of this comes a fifth loss point that is not a queue: LiteLLM does not call `flush()` per request on the native route, so a `SIGKILL` on the pod takes the last batch with it. On Kubernetes, this is an argument for a generous `terminationGracePeriodSeconds` and a `preStop` that gives some slack. None of this is a defect. It is the right design decision for a piece that sits on the critical path of inference. What changes is what can be asserted with the resulting data. ## What cannot be promised to the auditor A dashboard built on this pipeline answers statistical questions well: what is the p95 of time to first token for the model group, which team consumes the most, what proportion of requests fails, how cost per request evolves. That is what it is designed for, and that is what it is good for. It does not answer completeness questions well, which are the ones an ENS auditor or an ISO/IEC 42001 management system auditor asks. Faced with "show me that I have a record of every interaction with the AI system during March", the honest answer is that the traces are a best-effort record with four documented points of silent dropping, with no dead-letter queue on the last hop and no durability guarantee on any of them. The clean separation is the one already proposed in [technical controls ENS, 42001 and the AI Act](https://blog.lo0.es/en/posts/technical-controls-ens-iso-42001-eu-ai-act-cross-mapping/index.md): the regulatory audit record is a different artefact, with a different write path, durability guarantees and retention of its own. In the pair this article deals with, the `LiteLLM_SpendLogs` table in Postgres is a much better candidate than Langfuse for that function, because it is a transactional write against a relational database, and because it already contains `request_id`, `api_key` in hashed form, `team_id`, `organization_id`, `end_user`, `model`, tokens, `startTime`, `endTime`, `status` and `requester_ip_address`. Writing the audit event from the same place and in the same transaction as the spend is the sensible route; delegating it to the tracing pipeline is not. Here an operational tension appears that has to be resolved deliberately. LiteLLM's performance recommendation for production includes `disable_error_logs: true` and, in very loaded deployments, `disable_spend_logs: true`, because provider errors inflate the table and concurrent writes against the same rows produce locks and exhaust the Postgres connections. Disabling `spend_logs` to gain performance and at the same time leaning on that table for the audit record are incompatible decisions. Above a thousand requests per second or more than ten instances, the way out is `use_redis_transaction_buffer: true`, which batches the spend updates in Redis before pushing them down to Postgres, watching the `litellm_in_memory_spend_update_queue_size` and `litellm_redis_spend_update_queue_size` metrics. ## PII: switching off the content before it leaves An LLM trace contains, by definition, the user's prompt. In an on-premise deployment for a regulated customer, that content may be exactly what cannot leave the application's domain, not even towards the observability backend in the same cluster. The controls on the proxy side, from most global to most fine-grained: - `litellm_settings.turn_off_message_logging: true` cuts messages and responses in every callback, and keeps the metadata and the spend. The redaction replaces the content with the `REDACTED_BY_LITELLM` sentinel in messages, `choices`, `reasoning_content`, streaming deltas, tool call arguments and in the standard logging object. - The `x-litellm-enable-message-redaction: true` header enables redaction per request, and `LiteLLM-Disable-Message-Redaction: true` disables it. The second is checked first and wins over everything else, including the global configuration, which makes it a control to block at the ingress if the policy is that content is never logged. - `redact_user_api_key_info: true` removes the hashed token, the `user_id` and the `team_id` from the log. - Per request and per callback, `metadata: {"mask_input": true, "mask_output": true}`. - The body parameter `"no-log": true` disables logging of that call. It is a body parameter, not a header, and it can be disabled globally with `global_disable_no_log_param: true` so a client cannot make itself invisible. On the OTel v2 route the policy is inverted towards the safe side: content is off by default and is opened with `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT`, with values `no_content`, `span_only`, `event_only` and `span_and_event`. The documentation underlines that the gate is applied centrally and that a user's request cannot force their prompt towards the backend while capture is disabled. It is also the stance the OpenTelemetry specification recommends, whose preferred pattern for production is to store the content in external storage and leave only the reference in the span. One specific version to check in the inventory: below LiteLLM 1.98.0, with per-team Langfuse credentials, the team's own credentials ended up inside its traces. That version changed the origin of the metadata emitted to the `StandardLoggingPayload` to cut it off. The same change removed a couple of dozen metadata fields (`model_group`, `deployment`, `queue_time_seconds`, `attempted_retries` and others), so it breaks saved dashboards and alerts that filter on them. It is an upgrade with work attached, and it has to be done anyway. The same applies to the general hardening of the stack, covered in [hardening and secrets of the sovereign stack](https://blog.lo0.es/en/posts/hardening-secrets-sovereign-llm-stack-defence-in-depth/index.md). ## The price in latency LiteLLM publishes two overhead figures that differ by a factor of thirty, and both are theirs. The benchmarks page, with Locust, a thousand users with think time and a fake OpenAI endpoint, gives for four instances an `x-litellm-overhead-duration-ms` of median 2 ms, p95 8 ms and p99 13 ms, at 1,170 requests per second. For two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures hold with around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little's law. The AIGatewayBench benchmark, of July 2026, measures the Python proxy at **257.7 ms of added p99** with 329.5 MB of peak memory, against 0.7 ms for the Rust variant in beta. That measurement has no callbacks, no spend tracking and no persistence. The operational reading is that the first figure is the best case and the second the order of magnitude to plan for agentic load, which is the one that arrives in a closed loop against fast responses. To watch it there are two always-on headers: `x-litellm-overhead-duration-ms` and `x-litellm-callback-duration-ms`, and the documentation sets the useful threshold of the second at 100 ms, above which the diagnosis is that the payloads are too large. Three production settings that appear in LiteLLM's documentation and that are wrong by default: **`request_timeout` is 6,000 seconds.** A hundred minutes holding a connection against a hung upstream. The recommendation is to lower it to 600. **`LITELLM_LOG=DEBUG` serialises the payload with `json.dumps(indent=4)` synchronously**, and the documentation puts the cost of that serialisation at 2 to 5 seconds with payloads over 2 MB. It is the first cause of latency their troubleshooting guide mentions. **Background jobs register per uvicorn worker, not per pod.** With `--num_workers 4` and ten replicas that is forty copies of every periodic job. The `LITELLM_JOB_ROLE` variable separates the pods that serve traffic from the replica that runs jobs. And one failure mode with an open issue that deserves to be in the runbook: under sustained 429s from the upstream, pods have been reported unable to answer readiness probes during startup, with the ensuing restart loop. It fits with what was covered in [incident response runbooks](https://blog.lo0.es/en/posts/incident-response-runbooks-llm-inference-kafka-keep/index.md). ## Start-up checklist Ten checks before signing off the installation of the pair: 1. `callbacks: ["langfuse_otel"]` instead of `success_callback: ["langfuse"]`, unless you depend on prompt management, which the OTLP preset does not support. 2. `LANGFUSE_HOST` defined explicitly. The preset's default value points to the US region. 3. LiteLLM at 1.98.0 or higher, because of the team credential leak into the traces, and at 1.95.0 or higher because of the v4 ingestion header. 4. A price registered for every self-hosted model, per token and in USD, verified with `/spend/calculate` before opening traffic. 5. A test request with a known `traceparent`, checking that the same identifier appears in Langfuse. 6. An OpenTelemetry Collector between vLLM and Langfuse, with `OTEL_SERVICE_NAME` per engine and renaming of the old token attributes. 7. `request_timeout: 600` and `LITELLM_LOG` out of `DEBUG`. 8. Tail sampling in the Collector, with a latency and status code policy, to keep 100 % of what is slow and what failed. 9. A written decision about prompt content, and blocking of the `LiteLLM-Disable-Message-Redaction` header at the ingress if the policy is not to log. 10. A dashboard with `x-litellm-overhead-duration-ms`, `x-litellm-callback-duration-ms` and Langfuse's ingestion queue depth (`langfuse.queue.ingestion.depth`), which is the autoscaling signal for its workers. ## Traps and things that are not what they look like **The trace arrives, but ten minutes late.** The `x-langfuse-ingestion-version: 4` header is missing, which LiteLLM sends from 1.95.0. It is not a ClickHouse capacity problem. **The metadata works in development and disappears in production.** Nginx drops headers with underscores by default, and all of this integration's have one. **Session cost comes out as zero and generation cost does not.** Langfuse adds up only the ingested cost in the session totals, not the one it computes itself. **Enabling OTel v2 breaks nothing visible and ungroups the conversations.** The v2 mapper does not read `session_id` or `trace_user_id` from the request. **A team's budget is never exhausted.** Its spend is zero because the model has no price, and the calculation failure is only logged at debug level. **The alerts stop firing after the migration to v2.** The success status becomes `UNSET`; no rule on `OK` is satisfied. **Forwarding the `traceparent` is enabled and it does not reach the engine.** The generic `otel` callback is missing, without which the function that applies it returns before looking at the flag. **All the engines' spans appear under `unknown_service`.** vLLM emits no service name; `OTEL_SERVICE_NAME` has to be set per deployment. ## Closing The gateway and the tracing backend are the two pieces of the inference stack that take the least time to install and the longest to operate well. The installation fits in a ConfigMap. The operation consists of four decisions no getting-started guide raises: which integration route to take knowing that one expires in November, where the price per token of a model nobody invoices comes from, what to do with the `traceparent` at each hop, and what to tell an auditor about data the system drops by design when things get tight. The answer to the fourth is the one that orders the other three. This pipeline is operational instrumentation, with statistical precision and best-effort guarantees, and separating it from day one from the regulatory audit record avoids the awkward conversation two years from now. With that separation made, the LiteLLM and Langfuse pair is the pair of pieces that turns a fleet of GPUs into a platform with an owner, a known cost and explainable latency. ## See also - [Langfuse from the inside: the sorting centre](https://blog.lo0.es/en/posts/langfuse-inside-sorting-centre-bottleneck/index.md) — the six-service architecture and the ten backend knobs behind the receiving end of this integration. That post opens the box; this one connects the cable. - [The LLM inference router: the L7 switchboard](https://blog.lo0.es/en/posts/llm-inference-router-l7-switchboard/index.md) — the router's four functions and the minimal LiteLLM Proxy manifest on the generic cluster. - [Choosing the OSS gateway for LLM inference](https://blog.lo0.es/en/posts/choosing-oss-gateway-llm-inference/index.md) — the decision prior to this article, with the small print on licences and what is gated in LiteLLM. - [FinOps and GPU multi-tenancy with LiteLLM](https://blog.lo0.es/en/posts/finops-multi-tenancy-gpu-cluster-who-pays/index.md) — where the numbers that go into `input_cost_per_token` and `output_cost_per_token` come from, and why utilisation is the lever. - [From GPU-hour to cost per token and per request](https://blog.lo0.es/en/posts/gpu-hour-to-cost-per-token-on-prem-vs-cloud/index.md) — the full arithmetic of the conversion summarised here in one line. - [Instrumenting vLLM with OpenTelemetry](https://blog.lo0.es/en/posts/instrumenting-vllm-otel-measuring-optimisations/index.md) — the other end of the correlation, with the detail of what each `gen_ai.latency.*` measures. - [LLM tracing with OpenTelemetry GenAI](https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/index.md) — the `SDK → Collector → backend` pipeline and the two-layer sampling this post takes as given. - [Technical controls: ENS, ISO 42001 and the EU AI Act](https://blog.lo0.es/en/posts/technical-controls-ens-iso-42001-eu-ai-act-cross-mapping/index.md) — why the regulatory audit record cannot lean on a best-effort pipeline. - [Incident response runbooks](https://blog.lo0.es/en/posts/incident-response-runbooks-llm-inference-kafka-keep/index.md) — where the diagnostic headers and the readiness probe failure mode under sustained 429s fit. - [Anatomy of an LLM request in production](https://blog.lo0.es/en/posts/anatomy-llm-request-production-tour-six-stages/index.md) — the complete journey of a request through every layer, of which this post details two. - [LiteLLM on day 2: high availability](https://blog.lo0.es/en/posts/litellm-day-2-high-availability/index.md) — the process model, the probes, the retries and the 6,000-second timeout of the proxy that emits these traces. - [Virtual keys, budgets and limits](https://blog.lo0.es/en/posts/litellm-virtual-keys-budgets-limits/index.md) — who can call and with what cap, and the audit table that does work as evidence when the traces do not. - [The gateway does not live alone](https://blog.lo0.es/en/posts/gateway-jwt-audience-trace-identity-seams/index.md) — the gateway's other four seams, including the identity that never reaches the trace. - [Langfuse v4: what goes into a trace](https://blog.lo0.es/en/posts/langfuse-v4-what-goes-into-a-trace/index.md) — the new data model at the receiving end, and the 16 November date with its small print. ## Sources - LiteLLM, *Langfuse integration* (rutas de integración, metadata aceptada, cabeceras `langfuse_*`): . - LiteLLM, *OpenTelemetry v2* y *Migración a OTel v2* (preset, captura de contenido, cambios de semántica): · . - LiteLLM, *Logging* y *Config settings* (redacción, `turn_off_message_logging`, variables de entorno): · . - LiteLLM, *Custom pricing* y *Spend tracking* (claves de precio, SpendLogs, endpoints de gasto): · . - LiteLLM, *Production best practices* (topología, `request_timeout`, buffer de transacciones en Redis, `LITELLM_JOB_ROLE`): . - LiteLLM, *Benchmarks* y *Latency overhead* (cifras de sobrecarga y cabeceras de diagnóstico): · . - LiteLLM, *Rust AI Gateway benchmarks* (la cifra de 257,7 ms de p99 del proxy Python): . - LiteLLM, notas de versión v1.95.0, v1.98.0 y v1.100.0: · · . - Langfuse, *OpenTelemetry* (endpoint, autenticación, tabla de mapeo de atributos, ausencia de gRPC): . - Langfuse, *Compatibility* y *Migrate v3 to v4* (la fecha del 16 de noviembre de 2026 y los modos de escritura): · . - Langfuse, *Token and cost tracking* (prioridad del coste ingerido sobre el inferido): . - Langfuse, *Scaling* y *Sampling* (dimensionado, señal de autoescalado, muestreo de cabecera): · . - Langfuse, issue #13468 — descarte de filas en `ClickhouseWriter` sin cola de mensajes muertos: . Issue #15109 — coste de sesión a cero: . Issue #5173 — Nginx y cabeceras con guión bajo: . - vLLM, `vllm/tracing/utils.py` y `vllm/tracing/otel.py` en la etiqueta v0.28.0 (atributos emitidos, protocolo por defecto, `insecure=True`): . - Langfuse, *vLLM integration* (la advertencia de que vLLM solo exporta contadores y latencias): . - LiteLLM, `litellm/constants.py` y `litellm/litellm_core_utils/logging_worker.py` (capacidades de las colas y política de descarte): . - OpenTelemetry, *Semantic Conventions for Generative AI* y el repositorio `semantic-conventions-genai` (estado de desarrollo, renombrados, captura de contenido): . - OpenTelemetry Collector, `tailsamplingprocessor` (políticas de muestreo por cola y la restricción de instancia única por traza): . - llm-d, *End-to-end and fine-grained tracing in llm-d* (`OTEL_SERVICE_NAME` y los spans huérfanos tras Envoy): .