LLM tracing with OpenTelemetry GenAI: the aircraft black box the field stabilised in 2026

This post complements the one on Prompt versioning with Langfuse and MLflow, where prompt_id@version appeared as a span attribute without the whole pipeline being explained, and the one on Evals for LLMs, which distinguishes tracing from eval. Here we go inside tracing itself.

TL;DR

OpenTelemetry GenAI, the gen_ai.* and mcp.* semantic conventions, have been stable since late 2025, and as of May 2026 they are the substrate on which the whole ecosystem (Langfuse, Phoenix, LangSmith, Braintrust, Arize, Honeycomb, Datadog, New Relic) builds its LLM observability. That means your team instruments once, against OTel, and picks a backend afterwards; not the other way round. The canonical pipeline is application → OTel SDK (OpenLLMetry or openinference) → OTel Collector → backend(s), with two-layer sampling (head-based at 1-5 % to keep volume viable, tail-based at 100 % over errors and high latencies). This post takes apart the exact attributes you have to fill in on each span (gen_ai.system, request.model, usage.input_tokens, and so on), the operational difference between the three usual SDKs, and the anatomy of a real chat-with-RAG-and-tool trace with all its child spans.

You are here: OBSERVE

The analogy: the aircraft black box

An aircraft black box does three things at once:

1. It records everything without anyone having to ask for it explicitly. Every flight parameter, every cockpit conversation, every action by the pilots is logged with a precise timestamp and a standardised format.
2. It uses a format everybody knows. Any air accident investigator anywhere in the world can read the black box of any manufacturer, because the schema is agreed between all of them. There is no “Boeing black box” and “Airbus black box”: there is one single black box.
3. It survives the event: it is designed on the assumption that the aircraft may go down. It does not depend on the aircraft itself to be stored or to be read.

LLM tracing with OTel GenAI does exactly those three things:

1. It records everything automatically. Every model call, every agent step, every tool call, every RAG retrieval ends up as a span, a unit of work with a start, an end, attributes and a parent/child relationship with other spans.
2. It uses a common format. The attributes gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.response.finish_reasons and the rest are agreed by the OpenTelemetry GenAI working group and every backend respects them.
3. It survives the backend. Spans do not travel straight to Langfuse: they travel to an intermediate OTel Collector that handles buffering, sampling and fan-out to 1, 2 or N backends. If Langfuse goes down, the Collector holds the spans.

The analogy is not decorative: it is the only way to understand why OTel GenAI beat the ecosystem of proprietary SDKs that every vendor shipped in 2023-2024. The standardised black box beats the proprietary recorder, every time.

Why OTel GenAI exists (and why it won)

The state of things at the end of 2024 was broken: every vendor shipped its own SDK with its own attribute names. LangSmith called it model_name, OpenLLMetry called it llm.model, Helicone called it model. If you wanted to change observability provider, you re-instrumented the whole application. For serious teams, that was unacceptable.

OpenTelemetry already had a framework to solve this: the semantic conventions. A working group (the OpenTelemetry GenAI SIG) proposed a schema. In 2025 it reached “Stable” for the attributes of the basic modality (chat completions + tool calls), and in 2026 it also covers multimodal, embeddings and reasoning modalities.

What changes in practice:

• Before (proprietary SDKs): you move from Langfuse to Phoenix → you re-instrument, re-deploy everything, lose the history.
• Now (OTel): you move from Langfuse to Phoenix → you edit the exporter in otel-collector-config.yaml, reload the Collector, and the new spans go to the new destination without touching a line of application code.

The canonical gen_ai.* attributes

As of May 2026 the stable attributes, that is, the ones that are not going to change, are those in the table. There are more in “experimental” state (multimodal, reasoning) and others in “deprecated” (kept for compatibility).

And for tool calls (a function invoked by the model), the gen_ai.tool.* attributes:

And for tool calls over MCP (Model Context Protocol), the mcp.* attributes add a specific layer (mcp.server.name, mcp.method, mcp.session.id and others) that coexists with the gen_ai.* ones.

The practical rule: if your instrumentation fills in these attributes on every span, the Langfuse dashboard, the Tempo graph and the Prometheus metrics come out “for free”, because all three know how to read the convention.

The canonical pipeline: SDK → Collector → backends

Three layers, each with its own responsibility:

1 · The SDK

Three dominant options in 2026:

OpenLLMetry (Traceloop) — the most widespread. It automatically instruments the OpenAI Python SDK, the Anthropic SDK, LangChain, LlamaIndex, Haystack, Cohere, Mistral, vLLM and almost any library in the ecosystem. A single line (Traceloop.init()) instruments everything. Apache 2.0 licence. It maintains its own fork with extensions that are not yet in OTel core, but it exports in line with the convention.

openinference (Arize) — a direct competitor, also Apache 2.0. Closer to Phoenix (same vendor) but it exports standard OTel. Better instrumentation for LangChain/LlamaIndex/DSPy in some versions; worse for vLLM directly.

Langfuse SDK — proprietary in form (the API is Langfuse’s), but underneath it emits OTel spans. The natural choice if Langfuse is the main backend. It has the best support for “session” and “user” linking, concepts that are not yet in OTel core but that Langfuse maps to gen_ai.conversation.id for compatibility.

Practical recommendation for May 2026: OpenLLMetry if you want to stay backend-agnostic (you will change provider); the Langfuse SDK if Langfuse is already your bet (you save yourself a mapping). Both produce valid spans that any Collector consumes.

2 · The OTel Collector

The Collector is the most important piece of the pipeline and the least talked about. Three responsibilities:

Buffering: if Langfuse or Tempo have a problem, the Collector holds the spans in memory (or on disk with persistence enabled) and re-emits them when the backend returns.

Sampling: it applies the two sampling layers (next section) without the application knowing anything about it.

Routing: with the exporters config it points at one or several backends at once. The usual practice is to send to Langfuse + Tempo + Prometheus simultaneously:

• Langfuse for the LLM-first UI (prompts, evals, user sessions).
• Tempo (or Jaeger) for the full distributed context (an LLM span inside an HTTP request that has touched 12 microservices).
• Prometheus for aggregated metrics (P95 latency per model, tokens per second, error rate).

A representative production fragment of otel-collector-config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow_traces
        type: latency
        latency: { threshold_ms: 5000 }
      - name: head_5pct
        type: probabilistic
        probabilistic: { sampling_percentage: 5 }

exporters:
  otlphttp/langfuse:
    endpoint: https://langfuse.internal:3000/api/public/otel/v1
    headers: { Authorization: "Basic ${LANGFUSE_AUTH}" }
  otlp/tempo:
    endpoint: tempo:4317
    tls: { insecure: true }
  prometheus:
    endpoint: 0.0.0.0:8889

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/langfuse, otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]

3 · The backends

Langfuse is the dominant option for OSS LLM observability (MIT licence, complete suite: tracing + evals + prompts + datasets). Its model assumes OTel trace_id and span_id and draws them in the UI with the LLM context expanded (formatted messages, counted tokens, costs calculated from a model→price mapping).

Phoenix (Arize, ELv2 licence) has a more eval-first focus. Useful if your team comes from classic NLP and wants UMAP embedding visualisation by default.

LangSmith (LangChain) is proprietary, not OSS. Excellent integration with LangChain/LangGraph; less relevant if you do not use that stack.

Tempo / Jaeger show the LLM traces inside the context of the distributed request. It is what you need when a customer reports that “the page takes 8 seconds” and you want to see whether those 8 seconds are the model, the retrieval or the downstream microservice.

Sampling: the two layers that coexist

Without sampling, a system at 10 req/s generates about 26 million spans a month. The backend drowns, the cost explodes and most of the spans are redundant (200 similar requests tell you no more than 20).

The canonical pattern in 2026 is two combined layers:

Head-based sampling (on the way in)

You decide before the span exists whether you are going to trace it or not, based on a probabilistic decision (1-5 % typically) or on a deterministic rule (always trace if the user is premium, if the model is the new one in canary, and so on).

Pros: cheap (the span is not generated at all), deterministic.

Cons: you can miss infrequent errors. If an error happens once in every 1,000 and you trace 1 %, you will see it once in every 100,000, which is no longer debuggable.

Tail-based sampling (on the way out)

The complete span is generated and at the end you decide whether to keep it or throw it away. The decision can be:

• Keep 100 % of errors.
• Keep 100 % of traces with latency > X ms.
• Keep 100 % of traces that triggered a guardrail.
• Keep a random sample of the rest (1-5 %).

Pros: it guarantees that you see the errors and the latency outliers. Without it, the most interesting problems are the ones most likely to be left out.

Cons: you need a buffer, because until the trace finishes you cannot decide. The Collector’s tail_sampling processor keeps the spans in memory for 5-30 seconds until it takes the decision.

The practical rule in 2026 is to combine both: head-based at 5 % + tail-based capturing 100 % of errors and latencies > 5s. That gives you around 5 % of baseline traffic plus 100 % of what is interesting.

Anatomy of a real trace

We are going to take apart the trace of a single user question to a typical chat-with-RAG-and-tool system. The question is “what is the balance of my main account?”.

Seven spans, one single trace. The information you can exploit:

• Latency broken down: 38 ms guardrail + 218 ms retrieval + 2104 ms LLM + 60 ms tool + 28 ms guardrail = 2,448 ms (the rest is orchestration overhead). The bottleneck is the LLM (74 % of the time). If the user complains about slowness, you know where to look.
• Tokens and cost: 1,247 input + 412 output. With Llama 3.1 70B on-prem at about 12 W/token of equivalent energy cost, roughly 3.5 c€ per response. Multiplied by volume, that gives you the inference bill.
• prompt.id+version travelling along: if in a month’s time you notice that quality has dropped, you filter by prompt.version and see whether it coincides with a change to the prompt.
• Guardrail decisions traced: if a guardrail blocked the response, it stays in the trace with gen_ai.guardrail.decision=block and a reason. ENS audit satisfied.
• Correlated tool calls: tool.get_balance is a child span of gen_ai.chat. If the tool failed, you see the error directly in its span; it does not turn up only in separate logs.

Implications for on-premise hardware

Tracing is not free, but its overhead, properly sized, is negligible.

Cost per span

At the SDK level, an LLM span adds around 50-200 µs of overhead (creation, attributes, serialisation). For an LLM call lasting 2 seconds, that is < 0.01 % of the latency.

At the network level, a typical compressed OTLP span weighs around 1-3 KB. For a system at 100 req/s with 7 spans per request, that is 700 spans/s × 2 KB ≈ 1.4 MB/s to the Collector. Trivial.

Storage

This is where the money goes. Without sampling, a month at 100 req/s is about 18 GB of spans. With head-based at 5 % + tail-based on errors, about 1-2 GB a month. Manageable in any self-hosted ClickHouse or Loki.

On an RTX 4090 (24 GB) + Collector and Langfuse on-prem

For a demo service or a small single-tenant: Langfuse + Postgres + Clickhouse + Collector on the same host as the model. It uses around 8 GB of RAM and < 1 % of CPU continuously. The 4090 serves the model, the rest lives on the CPU. Minimum viable setup for a startup or a small team.

On a generic 4×H100 SXM cluster + central observability

A dedicated pod for the OTel Collector (a DaemonSet on each node + a central Gateway). Langfuse + ClickHouse + Tempo + Prometheus + Grafana in an observability namespace separate from the serving namespace. This is the canonical architecture of any serious ENS/NIS2 production deployment: observability does not share resources with the model.

What we have not covered (upcoming articles)

• LLM metrics (Prometheus): the canonical counters and histograms — gen_ai_client_request_duration_seconds, gen_ai_client_input_tokens_total and the rest. How Grafana dashboards are built on that base.
• Guardrails as spans: how to model Llama Guard 4, NeMo Guardrails and LLM Guard as child spans of the LLM span, and what attributes they carry. Covered in the post on guardrails — attributes gen_ai.guardrail.line, gen_ai.guardrail.detector, gen_ai.guardrail.category, gen_ai.guardrail.score, gen_ai.guardrail.action.
• Distributed tracing with MCP — traceparent propagation from the LLM client to the MCP server, mcp.* attributes, known problems.
• eBPF for automatic tracing without an SDK — Tetragon and Hubble extracting LLM traces without explicit instrumentation, for cases where the code cannot be modified.

See also

• Prompt versioning with Langfuse and MLflow — the prompt_id@version that travels as gen_ai.prompt.id in the spans described here.
• Evals for LLMs: the layer after tracing — the “the judge runs over a sample of the traces” pattern rests on this pipeline.
• LLM-as-judge: the exam marker — the judge consumes traces to evaluate quality continuously. The traces kept by tail-sampling are its input.
• Continuous fine-tuning in production — traces are the origin of the fine-tuning dataset. When a user regenerates, that span with gen_ai.response.finish_reasons=["stop"] rejected enters the DPO pipeline.
• The six-stage LLMOps pipeline — the master map where OBSERVE is stage 5.
• Guardrails and safety in LLMs — the online safety layer whose decisions are modelled as child spans of the LLM span with gen_ai.guardrail.* attributes. The traceability of every block / redact / flag is the basis for ENS / NIS2 / EU AI Act audits.
• LLM Guard: anatomy and integration with Langfuse — the concrete tool whose OTel HTTP export described in this post closes the circle: gen_ai.guardrail.* spans for each scanner executed, ingestable directly by Langfuse via /api/public/otel.
• ISO/IEC 42001: the operations manual for the AI system — the OTel + Langfuse tracing described here materialises control A.8 (information to interested parties) of Annex A: per-request traceability, forensic reporting capability and incident notification to users and regulators.
• EU AI Act: the technical file article by article — the OTel traceability described here is the canonical materialisation of Arts. 12 + 19 (record-keeping and automatic logs) of the Regulation; a minimum retention of 6 months + WORM + redacted PII are the explicit legal requirements.
• Technical controls ENS × 42001 × EU AI Act — OTel gen_ai.* spans with the right metadata satisfy op.exp.8 + .10 ENS + A.8.2 ISO 42001 + Arts. 12 + 19 AI Act in a single piece when they are labelled with a common vocabulary.
• GPU observability for LLM inference — the other half of LLM observability: the DCGM (hardware) and vLLM (engine) metrics complement the semantic tracing of gen_ai.* to cover the questions “is the GPU healthy?” and “is it meeting the SLO?”.
• Inside Langfuse: v3 architecture and the 10 backend knobs — here Langfuse is the destination of the pipeline; there the box is opened: the six services (Web, Worker, Postgres, ClickHouse, Redis, S3), asynchronous ingestion and self-hosted tuning. The two-layer sampling in this post is the “knob 0” that multiplies the ten in that one.
• Langfuse v4: what goes into a trace — the precedence chain with which a real backend resolves these attributes, and the scores that do not travel over OTLP.

References

• OpenTelemetry GenAI Semantic Conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/.
• OpenLLMetry (Traceloop): https://github.com/traceloop/openllmetry.
• openinference (Arize): https://github.com/Arize-ai/openinference.
• Langfuse OTel integration: https://langfuse.com/docs/opentelemetry/get-started.
• OTel Collector tail-sampling processor: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor.
• Phoenix (Arize) docs: https://docs.arize.com/phoenix.
• Model Context Protocol observability (mcp.* semconv): https://modelcontextprotocol.io/docs/observability.