Instrumenting vLLM with OTel: measuring what the optimisations really do
Contents
TL;DR
vLLM exposes two independent observability signals: Prometheus metrics (pull, aggregated) and OTel traces (push, per request). To measure whether chunked prefill, prefix caching, speculative decoding, FP8 KV cache and concurrency are really working, you need both. The metrics tell you what is happening in the system; the traces tell you why one specific request was slow. This article configures the complete pipeline and maps each optimisation to its diagnostic metric.
The analogy
A Formula 1 driver and his telemetry engineers. The driver feels that the car “goes odd” in turn 3, but without the sensor data he does not know whether it is the rear tyre, the differential or the fuel. The engineers see exactly what happened in that turn, temperature per sensor, lateral load per millisecond.
vLLM without OTel is the driver on his own: you notice that the TTFT “looks high” but you do not know whether it is a long prefill, a prefix cache miss, or a KV cache preemption. With OTel you have the full picture: the metrics are the race summary (aggregated), the traces are the lap-by-lap telemetry (per-request).
Architecture of the two signals
vLLM deliberately separates its two observability channels:
┌─────────────────────────────┐
│ vLLM │
│ │
requests ────────────►│ SchedulerStats │
│ │ │
│ ├─► Prometheus /metrics │◄── scrape (pull)
│ │ (aggregated, ~15s) │
│ │ │
│ └─► OTLP exporter │──► push (per request)
│ (spans, immediate) │
└─────────────────────────────┘
│ OTLP gRPC/HTTP
▼
┌─────────────────────┐
│ OTel Collector │
│ │
│ receivers: │
│ otlp (traces) │
│ prometheus │
│ exporters: │
│ langfuse │
│ prometheus remote│
│ loki (logs) │
└─────────────────────┘
Prometheus pull exposes metrics with the vllm: prefix at :8000/metrics. They are histograms, gauges and counters updated on every scheduler iteration. Good for dashboards and alerts about the system as a whole.
OTLP push sends one span per request, immediately on completion. It contains attributes of the specific request: prompt tokens, generated tokens, TTFT, model. Good for debugging anomalous requests and for Langfuse.
Installation and basic configuration
# vLLM with OTel support
pip install "vllm[otel]"
# Installs: opentelemetry-sdk, opentelemetry-api,
# opentelemetry-exporter-otlp, opentelemetry-semantic-conventions-ai
# Start vLLM with OTel enabled
export OTEL_SERVICE_NAME="vllm-production"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://otel-collector:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_TRACES_INSECURE="true" # on an internal network without TLS
vllm serve Qwen/Qwen2.5-7B-Instruct \
--otlp-traces-endpoint http://otel-collector:4317 \
--enable-chunked-prefill \
--enable-prefix-caching \
--kv-cache-dtype fp8 \
--speculative-model Qwen/Qwen2.5-0.5B-Instruct \
--num-speculative-tokens 5
The Prometheus metrics need no extra configuration: they are always at :8000/metrics.
OTel Collector: minimum configuration
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
prometheus:
config:
scrape_configs:
- job_name: vllm
scrape_interval: 15s
static_configs:
- targets: ["vllm:8000"]
processors:
batch:
timeout: 5s
resource:
attributes:
- key: deployment.environment
value: "production"
action: upsert
exporters:
otlphttp/langfuse:
endpoint: "https://cloud.langfuse.com/api/public/otel"
headers:
Authorization: "Basic <base64(pk:sk)>"
prometheusremotewrite:
endpoint: "http://prometheus:9090/api/v1/write"
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, resource]
exporters: [otlphttp/langfuse]
metrics:
receivers: [prometheus]
processors: [batch]
exporters: [prometheusremotewrite]
The five metrics that matter
Every optimisation has a primary diagnostic signal. If the metric does not move as expected after enabling the flag, there is a configuration or load problem.
1. Chunked prefill → vllm:time_to_first_token_seconds
Chunked prefill should reduce the variance of the TTFT, not necessarily the median. Its main goal is for the high percentiles (p99) to come down even if the p50 rises slightly.
# TTFT p50 and p99 — expect p99 to drop with chunked prefill enabled
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))
Sign that it works: the p99/p50 ratio approaches 1. Without chunked prefill, one request’s long prefill blocks all the others and the p99 rises disproportionately.
Sign of a problem: p50 and p99 both rise. A --max-num-batched-tokens that is too low makes the chunks so small that the prefill takes many steps to complete even though the other requests are not blocked. Raise the budget.
It is also useful to look at the llm.usage.prompt_tokens attribute per span in the OTel traces: requests with many prompt tokens should have a proportional TTFT, not a blocking one.
2. Prefix caching → vllm:gpu_prefix_cache_hit_rate
# Prefix cache hit rate on the GPU (0.0–1.0)
vllm:gpu_prefix_cache_hit_rate
# Evolution over a 5-minute window
rate(vllm:gpu_prefix_cache_hit_rate[5m])
Sign that it works: a sustained hit rate > 0.5 in workloads with a shared system prompt. With a hit rate of 0.8, 80% of requests skip the prefix prefill; the TTFT of those requests falls to the cost of the variable suffix alone.
Sign of a problem: a hit rate close to zero despite system prompts that “look” identical. The usual causes:
# ❌ This breaks the prefix caching hash:
system_prompt = f"Today is {datetime.now()}. You are an assistant..."
# ^^^ a different timestamp on every request
# ✅ The system prompt has to be identical byte for byte:
system_prompt = "You are an assistant specialising in infrastructure..."
Any variation in the system prompt, timestamps, session IDs, interpolated prompt versions, produces a different hash and a cache miss. The OTel traces do not expose the hit/miss per request directly in the current implementation; use them to correlate a high llm.usage.prompt_tokens with a high TTFT in the same request.
3. Speculative decoding → vllm:spec_decode_draft_acceptance_rate
# Acceptance rate of the draft model (0.0–1.0)
vllm:spec_decode_draft_acceptance_rate
# Estimated effective speedup (with k=5 proposed tokens)
# speedup ≈ (1 + α·k) / (1 + overhead_draft)
# Simplified: if α=0.75 and k=5 → speedup ≈ 1 + 0.75×5×(1 - cost_ratio)
Sign that it works: a sustained acceptance rate > 0.70. Below 0.60, the draft model’s overhead outweighs the gain from the accepted tokens and speculative decoding is counterproductive.
Sign of a problem: an acceptance rate < 0.50. The usual causes:
- A drafter from a different family than the verifier (for instance, Mistral 0.5B as the draft for Qwen 7B).
- A high generation temperature (>0.9): the higher the temperature, the more the draft’s distribution diverges from the verifier’s.
- A very large batch: at high concurrency, the draft can fall outside the domain of the current requests.
# Alert: inefficient speculative decoding
ALERT SpecDecodeInefficient
IF vllm:spec_decode_draft_acceptance_rate < 0.60
FOR 5m
LABELS { severity = "warning" }
ANNOTATIONS { summary = "Draft acceptance rate low: disable spec decode or change the drafter" }
In the OTel traces, the complete span of the request includes the total decode time. Without an acceptance rate per span, the way to detect that spec decode is working is to compare the total decode time divided by the tokens generated: if it is significantly lower than the baseline without spec decode, it is helping.
4. FP8 KV cache and concurrency → vllm:gpu_cache_usage_perc + vllm:num_preemptions_total
These two metrics are the two sides of KV cache management:
# KV cache utilisation (0.0–1.0)
# With FP8 enabled, the same hardware supports more requests before saturating
vllm:gpu_cache_usage_perc
# Cumulative preemptions (counter)
# Rises when vLLM cannot fit more requests and pauses one
rate(vllm:num_preemptions_total[5m])
Sign that FP8 works: with --kv-cache-dtype fp8 enabled, gpu_cache_usage_perc should saturate at concurrency levels around 2× higher than the BF16 baseline before num_preemptions_total starts to grow.
Sign of a problem: num_preemptions_total growing at rates > 1/minute with gpu_cache_usage_perc below 0.90. It indicates that max-num-seqs is too high for the available KV cache: requests enter the system but there are no free blocks to assign to them. Lower max-num-seqs or reduce max-model-len.
# Alert: KV cache saturated with preemptions
ALERT KVCacheSaturated
IF rate(vllm:num_preemptions_total[2m]) > 0.5
AND vllm:gpu_cache_usage_perc > 0.85
FOR 3m
LABELS { severity = "critical" }
ANNOTATIONS { summary = "KV cache saturated: lower max-num-seqs or max-model-len" }
The impact of FP8 on capacity can be quantified:
$$\Delta\text{capacity} = \frac{\text{max tokens FP8}}{\text{max tokens BF16}} \approx 2\times$$Measure before and after enabling --kv-cache-dtype fp8: the level of gpu_cache_usage_perc for a given concurrency should fall by half.
5. Effective concurrency → vllm:num_running_seqs + vllm:num_waiting_seqs
# Requests active in the engine (decode + prefill in progress)
vllm:num_running_seqs
# Requests queued waiting for a slot
vllm:num_waiting_seqs
# Waiting ratio: if > 0.2 sustained, there is a concurrency bottleneck
vllm:num_waiting_seqs / (vllm:num_running_seqs + vllm:num_waiting_seqs)
Healthy sign: num_running_seqs stable close to the configured --max-num-seqs value, num_waiting_seqs low (< 10% of running).
Sign of a problem: a high num_waiting_seqs with a low gpu_cache_usage_perc. It indicates that the scheduler is not filling the available slots because max-num-batched-tokens is too low: the token budget per step does not allow the pending prefills to be processed fast enough. Raise max-num-batched-tokens.
Reference dashboard: the 5 metrics in Grafana
{
"panels": [
{
"title": "TTFT p50 / p99 (chunked prefill)",
"targets": [
{"expr": "histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p50"},
{"expr": "histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p99"}
]
},
{
"title": "Prefix cache hit rate",
"targets": [{"expr": "vllm:gpu_prefix_cache_hit_rate", "legendFormat": "GPU hit rate"}]
},
{
"title": "Spec decode acceptance rate",
"targets": [{"expr": "vllm:spec_decode_draft_acceptance_rate", "legendFormat": "acceptance rate"}]
},
{
"title": "KV cache usage + preemptions",
"targets": [
{"expr": "vllm:gpu_cache_usage_perc", "legendFormat": "cache usage"},
{"expr": "rate(vllm:num_preemptions_total[2m]) * 60", "legendFormat": "preemptions/min"}
]
},
{
"title": "Effective concurrency",
"targets": [
{"expr": "vllm:num_running_seqs", "legendFormat": "running"},
{"expr": "vllm:num_waiting_seqs", "legendFormat": "waiting"}
]
}
]
}
Connecting traces to Langfuse
vLLM’s OTel traces are GenAI semconv compatible spans. Langfuse accepts them directly via OTLP:
# In the OTel Collector (already configured above)
# The otlphttp/langfuse exporter sends traces to Langfuse Cloud or self-hosted
# For self-hosted Langfuse (ENS/sovereign):
exporters:
otlphttp/langfuse:
endpoint: "http://langfuse-internal:3000/api/public/otel"
headers:
Authorization: "Basic <base64(pk_xxx:sk_xxx)>"
In Langfuse, each vLLM request appears as a trace with:
gen_ai.system: the model servedgen_ai.usage.input_tokens: prompt tokensgen_ai.usage.output_tokens: generated tokens- Span duration: end-to-end latency
What does not appear directly in the span: the speculative decoding acceptance rate, the prefix cache hit/miss, or the number of preemptions. That data is only in Prometheus. The correct workflow is:
- Langfuse identifies an anomalous request by latency.
- Prometheus/Grafana shows whether in that interval there were high preemptions, low spec decode, or a prefix cache miss.
- The two are correlated by timestamp.
Quick diagnostic matrix
| Observable symptom | Prometheus metric | Probable cause | Action |
|---|---|---|---|
| TTFT p99 very high | ttft p99/p50 >> 2 | Long blocking prefills | Raise --max-num-batched-tokens |
| TTFT p50 high, p99 the same | ttft p50 > 500ms | Prefix cache not working | Check the system prompt hash |
| Slow decode with no improvement | spec_decode_acceptance < 0.60 | Incompatible drafter | Change the drafter or disable it |
| Sporadic OOM / crash | gpu_cache_usage_perc = 1.0 + preemptions | KV cache full | Lower max-num-seqs or enable FP8 |
| High queue with free cache | waiting >> 0 + cache < 0.70 | Low token budget | Raise --max-num-batched-tokens |
Implications for sovereign on-premise inference
In an ENS deployment where you cannot use Langfuse Cloud or DataDog, the complete self-hosted stack is:
# docker-compose.yml (or equivalent K8s manifests)
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:latest
volumes: [./otel-config.yaml:/etc/otel/config.yaml]
langfuse:
image: langfuse/langfuse:latest
environment:
DATABASE_URL: postgres://...
prometheus:
image: prom/prometheus:latest
grafana:
image: grafana/grafana:latest
The whole pipeline runs on-premise. The traces never leave the perimeter. ENS compliance does not depend on which observability you choose: it depends on the inference data not going out to third parties. With a local stack, both conditions are met.
See also
- the fundamentals of OTel GenAI semconv: what spans are, the standard attributes and how they flow from the SDK to the collector
- the prefill optimisations this article instruments: chunked prefill, prefix caching, FP8 KV
- the decode optimisations: speculative decoding, gpu-memory-utilization, max-num-seqs
- DCGM for the GPU metrics underneath vLLM: SM utilization, memory bandwidth, temperature; the hardware layer below the application metrics
- how to correlate GPU metrics (DCGM) with application metrics (vLLM Prometheus) for a complete diagnosis
- the scheduler that produces the num_running_seqs and num_waiting_seqs metrics; without understanding the scheduler, the concurrency metrics have no context