Optimization on lo0 — Tech Blog

Batch sizing in vLLM: the two-hour grid search worth weeks of hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

max-num-seqs and max-num-batched-tokens are the two dials that control how much work vLLM processes on each scheduler iteration. Their default values are calibrated to be safe on any hardware, not to maximise throughput on yours. A systematic grid search over 25 configurations, runnable in two hours, identifies the combination that, for your specific workload and hardware, can double throughput without changing a single line of the model or adding a GPU. The OTel metrics that confirm you found the optimum are vllm:num_waiting_seqs, vllm:num_preemptions_total and vllm:time_per_output_token_seconds.

The analogy

An industrial kitchen with one chef and ten stoves. If the maître d’ only sends one order at a time, the chef works at 10% of capacity. If he sends a hundred simultaneous orders but there are ingredients for only twenty, the chef spends half his time waiting for restocking. The optimum is the point where every stove is lit and the resupply never runs dry.

max-num-seqs is how many orders the chef can have in preparation at once. max-num-batched-tokens is how many ingredients he can process in a single movement of the wok. Getting either of them wrong leaves stoves empty.

The problem: the defaults are not for your hardware

In vLLM V1 (≥ 0.6), the defaults are:

max-num-seqs = 1024 (V1) / 256 (V0)
max-num-batched-tokens = 8192

These values guarantee that vLLM starts on any GPU without OOM. They do not guarantee optimal throughput. The reason: the optimal point depends on three variables vLLM does not know at start-up:

The length distribution of your real workload — a RAG system with 2K-token prompts needs a different budget from a chat with 50-token messages.
VRAM available for KV cache — determined by the model, the quantisation and --gpu-memory-utilization.
Real expected concurrency — how many simultaneous users arrive at the 95th percentile.

The interaction between these three factors makes it impossible for a universal default to be optimal in specific cases.

The arithmetic that matters

On each iteration, vLLM’s scheduler decides which tokens to process. The total budget available per step is max-num-batched-tokens. That budget is split between:

Decode tokens: 1 for each active request in the generation phase. With 64 requests in decode, 64 tokens of budget are consumed.
Prefill tokens (in chunks): the rest of the budget goes to processing new prompts.

$$\text{prefill tokens per step} = \text{max num batched tokens} - \text{num requests decode}$$

If max-num-batched-tokens = 8192 and you have 512 requests in decode, each step can only process 8192 - 512 = 7680 prefill tokens. With 2000-token prompts, that is ~3.8 new prompts per iteration.

The problem appears when max-num-seqs is very high relative to the available KV cache. Each active request in decode occupies KV cache blocks. If the blocks run out, vLLM performs preemption: it pauses a request, frees its KV cache and re-queues it. Each preemption costs additional latency for the paused request and complexity for the scheduler.

$$\text{KV budget} = \frac{\text{free VRAM} \times \text{gpu memory utilization}}{\text{bytes per token} \times \text{max model len}}$$

For a Qwen2.5-14B on an RTX 4090 with Q4_K_M (9 GB of model, 15 GB free):

$$\text{KV budget} = \frac{15 \times 0.92 \times 10^9}{40\,000} \approx 345\,000 \text{ tokens}$$

With max-model-len = 8192, the maximum number of simultaneous requests with a full context is:

$$\text{real max seqs} = \frac{345\,000}{8192} \approx 42 \text{ requests}$$

Setting max-num-seqs = 1024 with those numbers guarantees constant preemptions. The optimum is at 40-50.

The grid search: methodology

Step 1: measure the real workload

Before hunting for the optimum, you need to know your traffic’s percentiles. From Langfuse or the vLLM logs:

# Extract the length distribution from Langfuse
import langfuse
client = langfuse.Langfuse()

traces = client.fetch_traces(limit=1000).data
prompt_lens = [t.input_tokens for t in traces if t.input_tokens]
output_lens = [t.output_tokens for t in traces if t.output_tokens]

import numpy as np
print(f"Prompt p50={np.percentile(prompt_lens,50):.0f} p95={np.percentile(prompt_lens,95):.0f} p99={np.percentile(prompt_lens,99):.0f}")
print(f"Output p50={np.percentile(output_lens,50):.0f} p95={np.percentile(output_lens,95):.0f} p99={np.percentile(output_lens,99):.0f}")

Step 2: work out the KV budget

Run once with --dry-run, or read vLLM’s start-up log:

INFO: # GPU blocks: 4521, # CPU blocks: 512

Each block is 16 tokens. 4521 × 16 = 72,336 tokens of total KV budget.

Step 3: the grid

With the KV budget known and the p95 of prompt/output length:

# grid_search_batch.py
import subprocess, json, time

MODEL = "Qwen/Qwen2.5-14B-Instruct-AWQ"
PROMPT_LEN = 512 # p50 of your workload
OUTPUT_LEN = 256
CONCURRENCY = 32 # simultaneous users expected at peak

seqs_values = [32, 64, 128, 256, 512]
tokens_values = [4096, 8192, 16384, 32768, 65536]

results = []
for seqs in seqs_values:
 for tokens in tokens_values:
 cmd = [
 "python", "-m", "vllm.entrypoints.benchmark_throughput",
 "--model", MODEL,
 "--max-num-seqs", str(seqs),
 "--max-num-batched-tokens", str(tokens),
 "--num-prompts", "200",
 "--input-len", str(PROMPT_LEN),
 "--output-len", str(OUTPUT_LEN),
 ]
 out = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
 # Parse throughput from the output
 for line in out.stdout.splitlines():
 if "Throughput" in line:
 tps = float(line.split(":")[1].strip().split()[0])
 results.append({"seqs": seqs, "tokens": tokens, "tps": tps})
 print(f"seqs={seqs} tokens={tokens} → {tps:.1f} tok/s")

# Save for analysis
with open("grid_results.json", "w") as f:
 json.dump(results, f, indent=2)

25 configurations × ~5 min = ~2 hours. Real execution time, not waiting time.

Step 4: read the surface

The result is a 5×5 throughput matrix. The typical shape:

max-num-batched-tokens → 4K 8K 16K 32K 64K
max-num-seqs ↓
32 180 310 380 390 385 ← max-num-seqs too low
64 185 350 480 510 508 ← optimal point for this workload
128 182 340 450 480 475
256 178 320 400 410 402 ← KV cache runs out, preemptions
512 170 290 360 370 368 ← high preemptions

The optimum in this example: max-num-seqs=64, max-num-batched-tokens=32768. Above that, preemptions cancel out the concurrency gain.

Confirmation with OTel

Once the optimal configuration is deployed, three Prometheus metrics confirm it is well calibrated:

# 1. Queued requests — should stay close to 0
# If it grows steadily: max-num-seqs too low or max-num-batched-tokens insufficient
vllm:num_waiting_seqs

# 2. Preemptions — should be 0 or very occasional (<1/min)
# If it grows: max-num-seqs too high for the available KV cache
rate(vllm:num_preemptions_total[5m]) * 60

# 3. ITL (inter-token latency) — should be stable, with no spikes
# Bimodality = badly calibrated batch size (some requests outside the CUDA graph bucket)
histogram_quantile(0.99, rate(vllm:time_per_output_token_seconds_bucket[5m]))

The optimal configuration produces:

num_waiting_seqs ≈ 0 under normal operation
num_preemptions_total stable (not growing)
time_per_output_token unimodal

If num_waiting_seqs is high with gpu_cache_usage_perc low: raise max-num-batched-tokens to process prefills faster. If num_preemptions_total grows: lower max-num-seqs or enable FP8 KV cache to free up blocks.

Reference configurations by profile

Based on the grid search for mid-range hardware (generic 4×H100, 14B-70B model):

Profile	Prompt p50	Output p50	max-num-seqs	max-num-batched-tokens
Conversational chat	150 tok	300 tok	256	16384
Enterprise RAG	1500 tok	200 tok	64	32768
Coding (completion)	800 tok	500 tok	128	32768
Summarisation	2500 tok	400 tok	32	65536
Batch processing	4000 tok	800 tok	16	65536

None of these is universal. They are starting points for the grid search on your real hardware and workload.

When not to touch the defaults

If your system sits below 50% KV cache utilisation (vllm:gpu_cache_usage_perc < 0.50) under real demand and with no num_waiting_seqs, the defaults are enough for your current load. The grid search contributes more when you are close to maximum capacity or when you want to extract the full performance of fixed hardware.

References

Instrumenting vLLM with OTel: measuring what the optimisations really do

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM exposes two independent observability signals: Prometheus metrics (pull, aggregated) and OTel traces (push, per request). To measure whether chunked prefill, prefix caching, speculative decoding, FP8 KV cache and concurrency are really working, you need both. The metrics tell you what is happening in the system; the traces tell you why one specific request was slow. This article configures the complete pipeline and maps each optimisation to its diagnostic metric.

The analogy

A Formula 1 driver and his telemetry engineers. The driver feels that the car “goes odd” in turn 3, but without the sensor data he does not know whether it is the rear tyre, the differential or the fuel. The engineers see exactly what happened in that turn, temperature per sensor, lateral load per millisecond.

vLLM without OTel is the driver on his own: you notice that the TTFT “looks high” but you do not know whether it is a long prefill, a prefix cache miss, or a KV cache preemption. With OTel you have the full picture: the metrics are the race summary (aggregated), the traces are the lap-by-lap telemetry (per-request).

Architecture of the two signals

vLLM deliberately separates its two observability channels:

 ┌─────────────────────────────┐
│ vLLM │
│ │
requests ────────────►│ SchedulerStats │
│ │ │
│ ├─► Prometheus /metrics │◄── scrape (pull)
│ │ (aggregated, ~15s) │
│ │ │
│ └─► OTLP exporter │──► push (per request)
│ (spans, immediate) │
└─────────────────────────────┘
│ OTLP gRPC/HTTP
▼
┌─────────────────────┐
│ OTel Collector │
│ │
│ receivers: │
│ otlp (traces) │
│ prometheus │
│ exporters: │
│ langfuse │
│ prometheus remote│
│ loki (logs) │
└─────────────────────┘

Prometheus pull exposes metrics with the vllm: prefix at :8000/metrics. They are histograms, gauges and counters updated on every scheduler iteration. Good for dashboards and alerts about the system as a whole.

OTLP push sends one span per request, immediately on completion. It contains attributes of the specific request: prompt tokens, generated tokens, TTFT, model. Good for debugging anomalous requests and for Langfuse.

Installation and basic configuration

# vLLM with OTel support
pip install "vllm[otel]"
# Installs: opentelemetry-sdk, opentelemetry-api,
# opentelemetry-exporter-otlp, opentelemetry-semantic-conventions-ai

# Start vLLM with OTel enabled
export OTEL_SERVICE_NAME="vllm-production"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://otel-collector:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
export OTEL_EXPORTER_OTLP_TRACES_INSECURE="true" # on an internal network without TLS

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --otlp-traces-endpoint http://otel-collector:4317 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5

The Prometheus metrics need no extra configuration: they are always at :8000/metrics.

OTel Collector: minimum configuration

# otel-collector-config.yaml
receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

 prometheus:
 config:
 scrape_configs:
 - job_name: vllm
 scrape_interval: 15s
 static_configs:
 - targets: ["vllm:8000"]

processors:
 batch:
 timeout: 5s
 resource:
 attributes:
 - key: deployment.environment
 value: "production"
 action: upsert

exporters:
 otlphttp/langfuse:
 endpoint: "https://cloud.langfuse.com/api/public/otel"
 headers:
 Authorization: "Basic <base64(pk:sk)>"

 prometheusremotewrite:
 endpoint: "http://prometheus:9090/api/v1/write"

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [batch, resource]
 exporters: [otlphttp/langfuse]
 metrics:
 receivers: [prometheus]
 processors: [batch]
 exporters: [prometheusremotewrite]

The five metrics that matter

Every optimisation has a primary diagnostic signal. If the metric does not move as expected after enabling the flag, there is a configuration or load problem.

1. Chunked prefill → `vllm:time_to_first_token_seconds`

Chunked prefill should reduce the variance of the TTFT, not necessarily the median. Its main goal is for the high percentiles (p99) to come down even if the p50 rises slightly.

# TTFT p50 and p99 — expect p99 to drop with chunked prefill enabled
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))

Sign that it works: the p99/p50 ratio approaches 1. Without chunked prefill, one request’s long prefill blocks all the others and the p99 rises disproportionately.

Sign of a problem: p50 and p99 both rise. A --max-num-batched-tokens that is too low makes the chunks so small that the prefill takes many steps to complete even though the other requests are not blocked. Raise the budget.

It is also useful to look at the llm.usage.prompt_tokens attribute per span in the OTel traces: requests with many prompt tokens should have a proportional TTFT, not a blocking one.

2. Prefix caching → `vllm:gpu_prefix_cache_hit_rate`

# Prefix cache hit rate on the GPU (0.0–1.0)
vllm:gpu_prefix_cache_hit_rate

# Evolution over a 5-minute window
rate(vllm:gpu_prefix_cache_hit_rate[5m])

Sign that it works: a sustained hit rate > 0.5 in workloads with a shared system prompt. With a hit rate of 0.8, 80% of requests skip the prefix prefill; the TTFT of those requests falls to the cost of the variable suffix alone.

Sign of a problem: a hit rate close to zero despite system prompts that “look” identical. The usual causes:

# ❌ This breaks the prefix caching hash:
system_prompt = f"Today is {datetime.now()}. You are an assistant..."
# ^^^ a different timestamp on every request

# ✅ The system prompt has to be identical byte for byte:
system_prompt = "You are an assistant specialising in infrastructure..."

Any variation in the system prompt, timestamps, session IDs, interpolated prompt versions, produces a different hash and a cache miss. The OTel traces do not expose the hit/miss per request directly in the current implementation; use them to correlate a high llm.usage.prompt_tokens with a high TTFT in the same request.

3. Speculative decoding → `vllm:spec_decode_draft_acceptance_rate`

# Acceptance rate of the draft model (0.0–1.0)
vllm:spec_decode_draft_acceptance_rate

# Estimated effective speedup (with k=5 proposed tokens)
# speedup ≈ (1 + α·k) / (1 + overhead_draft)
# Simplified: if α=0.75 and k=5 → speedup ≈ 1 + 0.75×5×(1 - cost_ratio)

Sign that it works: a sustained acceptance rate > 0.70. Below 0.60, the draft model’s overhead outweighs the gain from the accepted tokens and speculative decoding is counterproductive.

Sign of a problem: an acceptance rate < 0.50. The usual causes:

A drafter from a different family than the verifier (for instance, Mistral 0.5B as the draft for Qwen 7B).
A high generation temperature (>0.9): the higher the temperature, the more the draft’s distribution diverges from the verifier’s.
A very large batch: at high concurrency, the draft can fall outside the domain of the current requests.

# Alert: inefficient speculative decoding
ALERT SpecDecodeInefficient
 IF vllm:spec_decode_draft_acceptance_rate < 0.60
 FOR 5m
 LABELS { severity = "warning" }
 ANNOTATIONS { summary = "Draft acceptance rate low: disable spec decode or change the drafter" }

In the OTel traces, the complete span of the request includes the total decode time. Without an acceptance rate per span, the way to detect that spec decode is working is to compare the total decode time divided by the tokens generated: if it is significantly lower than the baseline without spec decode, it is helping.

4. FP8 KV cache and concurrency → `vllm:gpu_cache_usage_perc` + `vllm:num_preemptions_total`

These two metrics are the two sides of KV cache management:

# KV cache utilisation (0.0–1.0)
# With FP8 enabled, the same hardware supports more requests before saturating
vllm:gpu_cache_usage_perc

# Cumulative preemptions (counter)
# Rises when vLLM cannot fit more requests and pauses one
rate(vllm:num_preemptions_total[5m])

Sign that FP8 works: with --kv-cache-dtype fp8 enabled, gpu_cache_usage_perc should saturate at concurrency levels around 2× higher than the BF16 baseline before num_preemptions_total starts to grow.

Sign of a problem: num_preemptions_total growing at rates > 1/minute with gpu_cache_usage_perc below 0.90. It indicates that max-num-seqs is too high for the available KV cache: requests enter the system but there are no free blocks to assign to them. Lower max-num-seqs or reduce max-model-len.

# Alert: KV cache saturated with preemptions
ALERT KVCacheSaturated
 IF rate(vllm:num_preemptions_total[2m]) > 0.5
 AND vllm:gpu_cache_usage_perc > 0.85
 FOR 3m
 LABELS { severity = "critical" }
 ANNOTATIONS { summary = "KV cache saturated: lower max-num-seqs or max-model-len" }

The impact of FP8 on capacity can be quantified:

$$\Delta\text{capacity} = \frac{\text{max tokens FP8}}{\text{max tokens BF16}} \approx 2\times$$

Measure before and after enabling --kv-cache-dtype fp8: the level of gpu_cache_usage_perc for a given concurrency should fall by half.

5. Effective concurrency → `vllm:num_running_seqs` + `vllm:num_waiting_seqs`

# Requests active in the engine (decode + prefill in progress)
vllm:num_running_seqs

# Requests queued waiting for a slot
vllm:num_waiting_seqs

# Waiting ratio: if > 0.2 sustained, there is a concurrency bottleneck
vllm:num_waiting_seqs / (vllm:num_running_seqs + vllm:num_waiting_seqs)

Healthy sign: num_running_seqs stable close to the configured --max-num-seqs value, num_waiting_seqs low (< 10% of running).

Sign of a problem: a high num_waiting_seqs with a low gpu_cache_usage_perc. It indicates that the scheduler is not filling the available slots because max-num-batched-tokens is too low: the token budget per step does not allow the pending prefills to be processed fast enough. Raise max-num-batched-tokens.

Reference dashboard: the 5 metrics in Grafana

{
 "panels": [
 {
 "title": "TTFT p50 / p99 (chunked prefill)",
 "targets": [
 {"expr": "histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p50"},
 {"expr": "histogram_quantile(0.99, rate(vllm:time_to_first_token_seconds_bucket[5m]))", "legendFormat": "p99"}
 ]
 },
 {
 "title": "Prefix cache hit rate",
 "targets": [{"expr": "vllm:gpu_prefix_cache_hit_rate", "legendFormat": "GPU hit rate"}]
 },
 {
 "title": "Spec decode acceptance rate",
 "targets": [{"expr": "vllm:spec_decode_draft_acceptance_rate", "legendFormat": "acceptance rate"}]
 },
 {
 "title": "KV cache usage + preemptions",
 "targets": [
 {"expr": "vllm:gpu_cache_usage_perc", "legendFormat": "cache usage"},
 {"expr": "rate(vllm:num_preemptions_total[2m]) * 60", "legendFormat": "preemptions/min"}
 ]
 },
 {
 "title": "Effective concurrency",
 "targets": [
 {"expr": "vllm:num_running_seqs", "legendFormat": "running"},
 {"expr": "vllm:num_waiting_seqs", "legendFormat": "waiting"}
 ]
 }
 ]
}

Connecting traces to Langfuse

vLLM’s OTel traces are GenAI semconv compatible spans. Langfuse accepts them directly via OTLP:

# In the OTel Collector (already configured above)
# The otlphttp/langfuse exporter sends traces to Langfuse Cloud or self-hosted

# For self-hosted Langfuse (ENS/sovereign):
exporters:
 otlphttp/langfuse:
 endpoint: "http://langfuse-internal:3000/api/public/otel"
 headers:
 Authorization: "Basic <base64(pk_xxx:sk_xxx)>"

In Langfuse, each vLLM request appears as a trace with:

gen_ai.system: the model served
gen_ai.usage.input_tokens: prompt tokens
gen_ai.usage.output_tokens: generated tokens
Span duration: end-to-end latency

What does not appear directly in the span: the speculative decoding acceptance rate, the prefix cache hit/miss, or the number of preemptions. That data is only in Prometheus. The correct workflow is:

Langfuse identifies an anomalous request by latency.
Prometheus/Grafana shows whether in that interval there were high preemptions, low spec decode, or a prefix cache miss.
The two are correlated by timestamp.

Quick diagnostic matrix

Observable symptom	Prometheus metric	Probable cause	Action
TTFT p99 very high	`ttft p99/p50 >> 2`	Long blocking prefills	Raise `--max-num-batched-tokens`
TTFT p50 high, p99 the same	`ttft p50 > 500ms`	Prefix cache not working	Check the system prompt hash
Slow decode with no improvement	`spec_decode_acceptance < 0.60`	Incompatible drafter	Change the drafter or disable it
Sporadic OOM / crash	`gpu_cache_usage_perc = 1.0` + preemptions	KV cache full	Lower `max-num-seqs` or enable FP8
High queue with free cache	`waiting >> 0` + `cache < 0.70`	Low token budget	Raise `--max-num-batched-tokens`

Implications for sovereign on-premise inference

In an ENS deployment where you cannot use Langfuse Cloud or DataDog, the complete self-hosted stack is:

# docker-compose.yml (or equivalent K8s manifests)
services:
 otel-collector:
 image: otel/opentelemetry-collector-contrib:latest
 volumes: [./otel-config.yaml:/etc/otel/config.yaml]

 langfuse:
 image: langfuse/langfuse:latest
 environment:
 DATABASE_URL: postgres://...

 prometheus:
 image: prom/prometheus:latest

 grafana:
 image: grafana/grafana:latest

The whole pipeline runs on-premise. The traces never leave the perimeter. ENS compliance does not depend on which observability you choose: it depends on the inference data not going out to third parties. With a local stack, both conditions are met.

References

Knowledge Distillation: teaching a small model to think like a big one

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Knowledge Distillation is the technique of training a small model (student) using the output probabilities of a large model (teacher) as supervision, instead of using only the hard labels from the training dataset. The result is a small model that reasons better than its size suggests, because it learns the teacher’s uncertainty distributions rather than memorising binary answers. It is the reason Phi-4 (14B) beats most 70B models at reasoning, and why the Gemma 3 family models are surprisingly capable for their size. It is not a compression technique for an existing model: it is a training process that produces a smaller model from scratch or from a different starting point.

The analogy

A master surgeon with thirty years of experience and a first-year resident. If the resident only learns from the anatomy manual (binary correct answers: “cut here, not here”) it will take years to develop the master’s clinical judgement. But if they operate alongside them, watching their micro-decisions, their hesitations, the ambiguous cases where the master knows that two options are almost equally valid, they learn something the manual cannot teach: the structure of uncertainty.

Knowledge distillation is exactly that. The “anatomy manual” is the hard labels (the correct answer). The “master surgeon” is the teacher LLM. The probability distributions over the vocabulary are the materialisation of that uncertainty the student absorbs.

What it really is

When an LLM generates text, it does not produce a single word: it produces a probability distribution over its whole vocabulary at each position. For the next token, the model might say:

"Paris": 42%
"Lyon": 8%
"Marseille": 6%
"the city": 5%
...rest of the vocabulary: 39%

This distribution is dense information. It tells you not only what the correct answer is, but also which other answers were plausible and to what degree. A student trained only with the label “Paris” (probability 1.0 on the correct token, 0.0 on the rest) never sees this richness.

Distillation uses the teacher’s full distribution as the student’s training target. The loss function has two terms:

$$\mathcal{L}_{total} = (1 - \alpha) \cdot \mathcal{L}_{CE}(y, \hat{y}_S) + \alpha \cdot \mathcal{L}_{KD}(p_T, p_S, T)$$

Where:

$\mathcal{L}_{CE}$ is the standard cross-entropy with the hard labels (classic supervision).
$\mathcal{L}_{KD}$ is the KL divergence between the teacher’s and the student’s distributions.
$\alpha$ controls the relative weight of each term (typically 0.5–0.9 in favour of KD).
$T$ is the temperature, a parameter that softens the distributions to make the KD signal more informative.

The role of temperature

If the teacher assigns 99% to “Paris” and 0.001% to every other word, the distribution is almost as informative as a hard label. A temperature $T > 1$ softens that distribution:

$$p_T(k) = \frac{\exp(z_k / T)}{\sum_j \exp(z_j / T)}$$

With $T = 4$ and the original logits, the distribution that was previously [99%, 0.001%, 0.001%…] becomes something like [42%, 8%, 6%…]. The student sees the teacher’s real probability neighbourhood, not just its point answer.

Numerical example with temperature:

Teacher logits for “The capital of France is _____”:

Paris: 8.5
Lyon: 3.2
Europe: 2.1
a: 1.8

With T=1 (standard softmax):

$$p(\text{Paris}) = \frac{e^{8.5}}{e^{8.5} + e^{3.2} + e^{2.1} + e^{1.8}} \approx 99.3\%$$

With T=4:

$$p(\text{Paris}) = \frac{e^{8.5/4}}{e^{8.5/4} + e^{3.2/4} + e^{2.1/4} + e^{1.8/4}} = \frac{e^{2.125}}{e^{2.125} + e^{0.8} + e^{0.525} + e^{0.45}} \approx 54\%$$

The signal at T=4 is far more informative for the student: it learns that Lyon is more plausible than Europe, that Europe is more plausible than “a”, and so on.

The three modes of distillation

Offline (or “black-box”)

The teacher generates a synthetic dataset of answers before training. The student is trained on that dataset as if it were ordinary hard labels.

teacher → generates 100M (prompt, completion) pairs → dataset
student → is trained on that dataset

It is the cheapest way to scale: the teacher runs only once, the student is trained on the generated data with conventional hardware. Most open source instruction models (Alpaca, Vicuna, WizardLM in their early versions) used this strategy: GPT-4 as teacher, saved data, Llama-7B as student.

Limitation: the student never sees the teacher’s probability distributions, only its answers. It is distillation of “behaviour”, not of “knowledge” in the strict sense. If the teacher makes a mistake (and GPT-4 does make mistakes), the error is crystallised into the dataset.

Online (or “white-box”)

Teacher and student run together during training. The student processes each batch, the teacher processes the same batch in parallel, and the KD loss is computed in real time with the full probability distributions.

for batch in dataset:
logits_teacher = teacher(batch) # teacher forward pass
logits_student = student(batch) # student forward pass
loss = KL(softmax(logits_teacher/T), softmax(logits_student/T))
loss.backward() # only updates student

The teacher has gradients disabled (torch.no_grad()). The learning signal is richer than in offline, but the cost is high: you need to keep the teacher in VRAM throughout training. To distil a 405B teacher into an 8B student, you would need several H100s for the teacher alone.

On-policy

A recent variant (2024–2026) that combines the best of both: the teacher generates answers dynamically during training, but the student evaluates them with its own distribution. The cycle is:

The student generates a proposed answer (rollout).
The teacher scores that proposal with its probability distribution.
The student updates with the teacher’s signal.

This stops the student from learning from distributions outside its own domain (the distribution shift problem in offline). It is the basis of algorithms such as SimCT (2026) that use teachers from different families (Qwen, Phi, Gemma) to generate a cross-tokeniser signal.

Why the best small models use distillation

Phi-4 (Microsoft, 14B), Gemma 3 (Google, 9B/27B), and the compact Qwen3 family models are the clearest examples. Their benchmarks are anomalous with respect to their size: Phi-4-14B beats LLaMA-3-70B on MATH and GPQA-Diamond, two mathematical and scientific reasoning benchmarks where size is usually decisive.

Why? The key is what supervises the training:

A model trained on internet data learns the distribution of human text, which includes a lot of low-quality text, errors and ambiguities.
A student that learns from a frontier teacher (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro) absorbs a distribution filtered towards high-quality text and correct reasoning.

The student with 14B parameters does not “know more” than an undistilled one of the same size, but it has learned to use them better because its training gradients were never contaminated by low-quality text.

Empirical data point: Phi-4 (14B distilled) vs LLaMA-3-70B (not distilled) on the MATH benchmark (2025):

Phi-4: 80.4%
LLaMA-3-70B: 68.0%

A model 5× smaller beats the large one because the training signal is better, not because it has more parameters.

Reasoning distillation: the case of thinking models

Reasoning models (DeepSeek-R1, Qwen3-thinking, QwQ) generate internal chains of thought before giving the final answer. Distilling reasoning is more complex because you do not only want to transfer the answer: you want to transfer the way of thinking.

The current strategy (2025–2026) is reasoning trace distillation:

The teacher (a large thinking model) generates answers with its full internal chain of thought.
The dataset includes those chains of thought as part of the output.
The student learns to imitate both the chain and the final answer.

This explains why Qwen3-7B-thinking can reason formally about mathematics while being 10× smaller than the models that preceded it without distillation: it learned the process, not just the result.

When to use distillation vs. the alternatives

Technique	What it does	Requires retraining	Result
Quantisation	Reduces weight precision	No	Same model, smaller
Pruning	Removes irrelevant weights	No (PTQ)	Same model, sparser
Distillation	Trains a new model	Yes	Different model, smaller

Distillation does not compress an existing model: it produces a new one. That is why it is complementary, not a substitute: you can distil a 405B into an 8B, and then quantise that 8B to INT4 to reduce its inference cost.

When it is the right option:

You need a model 5–10× smaller than the best available one.
You have access (API or local) to a quality teacher.
You have training data or the ability to generate it.
Latency or inference cost is a hard constraint.

When it is not:

You want to compress an existing model quickly: use quantisation + pruning.
You have no training budget (online distillation takes weeks of GPU).
The teacher is not significantly better than the base student: the KD signal will be weak.

Implications for on-premise inference

In a sovereign deployment, the teacher can be a large model running locally (no external API needed). The flow is:

generic 4×H100:
teacher: Llama-3.3-70B-Instruct (on the 4×H100, full load)
→ generates a dataset of 10M (prompt, completion with logits) pairs
→ 3-4 weeks of generation at batch 32
After the dataset:
student: Qwen2.5-7B (fine-tuned with KD loss over the dataset)
→ 2-3 days of training on the same H100s
→ result: a 7B that reasons like the 70B in the specific domain
Production:
RTX 4090: serves the 7B student quantised to INT4 (4 GB)

The teacher is only needed to generate the data. The student is what goes into production. The investment in training compute pays for itself in months of cheaper inference.

For ENS/NIS2: this flow is 100% on-premise, zero dependency on external APIs, and the resulting model is yours in every sense.

References

Distilling the Knowledge in a Neural Network — Hinton, Vinyals & Dean, 2015 (paper fundacional)
Phi-4 Technical Report — Microsoft Research, 2024
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek, 2025 (destilación de razonamiento)
Awesome LLM On-Policy Distillation — colección de papers de destilación en-policy, 2025–2026
Knowledge Distillation for LLMs: Survey — ICLR 2025

Optimising decode in vLLM: squeezing every token out of small hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Decode is the phase in which vLLM generates output tokens one at a time. It is memory-bound, not compute-bound: the GPU spends more time waiting for the weights to arrive from VRAM than doing arithmetic. On small hardware, an RTX 4090 (24 GB) or an L40 (48 GB), a badly configured decode wastes half the card’s capacity. Five vLLM parameters change the equation: gpu-memory-utilization, max-num-seqs, speculative decoding, FP8 KV cache and a swap-space correctly set to zero. Well calibrated, the difference is real: from 15 tokens/s to 35–50 tokens/s on the same hardware.

The analogy

An assembly-line worker putting cars together. Every car needs exactly the same process: walk to the store for the part, come back, bolt it on, repeat. The travel time to the store, the VRAM latency, is fixed and cannot be eliminated. But there are ways to make it less painful:

Have several cars in parallel on the line (more concurrency, the same travel time amortised).
Have an assistant prefabricate common parts (speculative decoding: the draft model proposes, the verifier confirms).
Keep only the most-used parts in the workshop (quantised KV cache: more contexts fit in the same space).

Those three strategies are exactly the three axes of decode optimisation in vLLM.

Why decode is memory-bound

During prefill, the GPU processes N tokens in parallel: the attention operation is a large matmul and the compute units are busy. During decode, it processes 1 token per step: the matmul becomes a vector-matrix product, an operation that underuses the tensor cores.

The typical compute utilisation ratio during decode on an RTX 4090:

$$\text{MFU}_{decode} \approx 5–15\% \quad \text{(vs 40–60\% in prefill)}$$

The bottleneck is not compute power, it is bandwidth. To generate each token, the model has to read its complete weights from VRAM:

$$\text{time per token} \approx \frac{\text{weight size in bytes}}{\text{VRAM bandwidth}}$$

For Qwen2.5-7B in BF16 (14 GB of weights) on an RTX 4090 (1,008 GB/s):

$$t \approx \frac{14 \times 10^9}{1.008 \times 10^{12}} \approx 13.9 \text{ ms/token} \approx 72 \text{ tokens/s theoretical maximum}$$

The real value is lower (~30–50 tok/s) because of scheduler overhead, attention over the growing KV cache and other latencies. But the theoretical limit sets the ceiling.

With Q4_K_M (weights ~4 GB):

$$t \approx \frac{4 \times 10^9}{1.008 \times 10^{12}} \approx 3.97 \text{ ms/token} \approx 252 \text{ tokens/s theoretical}$$

Quantising the model is the most direct way to improve decode throughput on memory-bound hardware. Everything else optimises on top of that ceiling.

The five levers

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

--gpu-memory-utilization (short form --gpu-mem-util) defines the fraction of available VRAM that vLLM may use for the KV cache, once the model weights are loaded. The rest is reserved for activations during the forward pass and for the CUDA context.

vllm serve my-model \
 --gpu-memory-utilization 0.92

The default value is 0.90. On bare metal where no other process uses the GPU, 0.92–0.95 is safe. Do not go above 0.95: vLLM needs headroom for activations during batch peaks, and running out of VRAM in the middle of an inference results in a process crash, not a clean error.

Why it matters: more KV cache available = more simultaneous requests in flight = better GPU utilisation during decode. PagedAttention allocates the KV cache in fixed-size blocks (16 tokens per block by default), and vLLM manages them like virtual memory pages. The more blocks available, the more requests it can serve without any of them waiting for space.

RTX 4090, Qwen2.5-7B-BF16 (14 GB weights):
Total VRAM: 24 GB
Weights: 14 GB
Available for KV cache: 10 GB
gpu-memory-utilization 0.90 → 0.90 × 10 GB = 9 GB for KV cache
gpu-memory-utilization 0.94 → 0.94 × 10 GB = 9.4 GB → ~4% more tokens in flight

The impact is modest with models that fit comfortably, but it is amplified with models that push VRAM to its limit.

2. Real concurrency — `--max-num-seqs`

--max-num-seqs is the maximum number of requests vLLM can have in process simultaneously (counting prefill and decode together). It is the parameter that controls the system’s effective concurrency.

vllm serve my-model \
 --max-num-seqs 128

The effect is direct: more requests in simultaneous decode = better amortisation of the fixed cost of reading the weights. When the decode batch grows from 1 to 8, the time to generate 8 tokens is almost the same as generating 1 (the weights are read once for all of them). Aggregate throughput scales almost linearly until the KV cache or the activation VRAM becomes the bottleneck.

$$\text{aggregate throughput}(B) \approx B \times \text{throughput}(1) \quad \text{for } B \ll B_{max}$$

Common mistake: raising --max-num-seqs without making sure there is enough KV cache in VRAM for all the requests. If vLLM cannot hold the KV caches of 128 simultaneous requests, it does preemption (pauses some request and frees its KV cache) at a latency cost. Monitor vllm:num_preemptions_total.

Interaction with --max-num-batched-tokens: the vLLM scheduler processes up to max-num-batched-tokens tokens per step. If you have 128 requests in decode generating 1 token each, that is 128 decode tokens. The decode budget consumes 128 tokens of the total budget; the rest goes to prefill in chunks. Tune both values together.

# For an RTX 4090 serving ~50 concurrent users with responses of up to 512 tokens
vllm serve my-model \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192
# 64 decode tokens per step + up to 8128 chunked prefill tokens

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

Speculative decoding is the highest-impact change for decode on small hardware. The idea is simple: a small draft model proposes several tokens at once, and the verifier model validates or rejects them in a single forward pass.

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1

Why it works: the 7B verifier has to read 14 GB of weights per step. With 5 proposed tokens, if the acceptance rate is 80%, an average of 4–5 tokens are generated per verifier step instead of 1. Effective throughput goes up without the GPU working any harder.

The acceptance rate (α) depends on how well the draft predicts the verifier’s distribution. Within the same domain, models from the same family usually have α > 0.75:

$$\text{speedup} \approx \frac{1 + \alpha \cdot k}{1 + \alpha \cdot k / \text{cost ratio}}$$

Where $k$ is the number of proposed tokens and cost_ratio is the draft/verifier cost ratio. For a 0.5B draft and a 7B verifier (ratio ~14×):

$$\text{speedup} \approx 1 + 0.8 \times 5 \approx 5 \text{ (theoretical maximum, not reachable)}$$

In practice, with α = 0.75 and k = 5 on hardware without NVLink: 1.8–2.5× more tokens/s compared with decode alone.

EAGLE-3 in 2026: the best drafters today are not small versions of the same model, but networks specialised in predicting the verifier’s distribution. EAGLE-3 reports a 3–6.5× speedup over vanilla decode in public benchmarks. In production with mixed batches the real speedup is more conservative (1.5–3×). vLLM supports EAGLE/EAGLE-2 via --speculative-model:

# With an EAGLE drafter (requires a drafter trained specifically for the base model)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --speculative-model yuhuili/EAGLE3-LLaMA3.1-Instruct-8B \
 --num-speculative-tokens 6

When speculative decoding does NOT help:

Very large batches (>32 requests): the acceptance rate varies between requests and the batch spends more time re-drafting than accepting.
High-entropy tasks (brainstorming, very creative code): the draft predicts worse, α falls below 0.5 and the draft overhead outweighs the gain.
If the draft model does not fit in the available VRAM alongside the verifier.

On an RTX 4090 with a 7B verifier and a 0.5B draft (BF16): 14 + 1 GB = 15 GB. That leaves 9 GB for the KV cache. It works.

4. Quantised KV cache — `--kv-cache-dtype fp8`

This was already covered in the prefill article for its effect on context capacity. From the decode point of view, the benefit is different: more tokens fit in the KV cache → more simultaneous requests without preemption → better aggregate throughput.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Precision warning in decode: the KV cache is read on every attention step of the decode. Quantisation introduces noise into the attention activations. For long texts (>4K tokens of context) it can accumulate. In quality benchmarks (MMLU, HellaSwag) the degradation with FP8 KV and --calculate-kv-scales is <0.5% on modern models. Without --calculate-kv-scales, the degradation can be larger because the scales are fixed statically.

Optimal combination for RTX 4090:

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
 --quantization awq \  # weights in INT4: 4 GB model
 --kv-cache-dtype fp8 \  # KV cache at half the size
 --calculate-kv-scales \
 --gpu-memory-utilization 0.94

# Available VRAM: 24 - 4 = 20 GB for KV cache
# With FP8: ~40 KB/token (vs 80 KB BF16) → 20 GB / 40 KB = 500,000 tokens of total context
# With max-num-seqs 64 and a 4K ctx: 64 × 4096 × 40KB = 10 GB → fits with room to spare

5. Remove the swap — `--swap-space 0`

--swap-space defines how much system RAM (not VRAM) vLLM may use to preempt KV caches. When vLLM has more active requests than fit in VRAM, it can “pause” some by moving their KV cache to RAM and reactivating them later.

The problem: moving a 4K-token KV cache from VRAM to RAM and back has a latency of tens of milliseconds over PCIe. For a system where you want predictable latency, the swap introduces unacceptable jitter.

vllm serve my-model \
 --swap-space 0

With --swap-space 0, when vLLM cannot hold any more requests in VRAM, it simply queues them instead of doing preemption. The queue adds waiting latency, but it is predictable and it does not interrupt the requests already in flight.

When should you have swap? If your workload has short demand peaks and you can tolerate occasional jitter in exchange for not rejecting requests, a swap of 4–8 GB can be useful. In ENS deployments where latency is a contractual SLA, --swap-space 0 is the right option.

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1 \
 --dtype bfloat16

Expected throughput: 35–55 tokens/s per user, up to 64 simultaneous, TTFT <500ms for prompts <1K tokens.

L40 (48 GB) — 14B model, multi-user

vllm serve Qwen/Qwen2.5-14B-Instruct \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --max-num-seqs 128 \
 --max-num-batched-tokens 16384 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-1.5B-Instruct \
 --num-speculative-tokens 5 \
 --dtype bfloat16

Expected throughput: 25–40 tokens/s per user, up to 128 simultaneous with speculative decoding active, TTFT <800ms for prompts <2K tokens.

How to measure that decode is optimised

# Key metrics at vllm:8000/metrics

vllm:generation_tokens_total # total tokens generated → trend
vllm:e2e_request_latency_seconds_* # end-to-end latency per percentile
vllm:time_per_output_token_seconds_* # ITL (inter-token latency)
vllm:num_preemptions_total # if it rises, the KV cache is filling up
vllm:spec_decode_draft_acceptance_rate # speculative decoding hit rate

If spec_decode_draft_acceptance_rate < 0.6, the drafter is not helping: turn speculative decoding off or find a drafter better trained for your model or domain.

If num_preemptions_total grows, you have too many simultaneous requests for the available KV cache. Options: lower max-num-seqs, enable FP8 KV cache, lower max-model-len, or quantise the model further.

Implications for on-premise inference

In a sovereign deployment with fixed hardware, you cannot buy more GPUs at will. Every well-calibrated tenth of gpu-memory-utilization, every point of speculative decoding acceptance rate and every MB of KV cache freed by FP8 is real capacity you do not have to provision with another node.

The combination of quantised weights (AWQ/GPTQ), FP8 KV cache and speculative decoding lets a 14B serve on one L40 what without optimisations would need two L40s in tensor parallel. That is the economic argument for investing time in these parameters.

Decode cannot be accelerated indefinitely on memory-bound hardware: the theoretical limit is set by VRAM bandwidth. But the difference between the minimum and the maximum achievable on that hardware can be 3–4× with the right levers.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Li et al., 2024
Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2022
vLLM Optimization and Tuning — documentación oficial
vLLM Speculative Decoding
FP8 KV Cache en vLLM

Optimising prefill in vLLM: the knobs your TTFT will not forgive

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Prefill is the phase in which vLLM processes your input prompt and produces the initial KV cache. It is compute-bound (unlike decode, which is memory-bound), it takes longer the longer the prompt is, and it blocks the decode of every other queued request. There are four levers in vLLM that radically change its behaviour: chunked prefill, prefix caching, FP8 KV cache and the per-batch token budget. With modest hardware, a 24 GB RTX 4090 or a 48 GB L40, the difference between ignoring them and using them well can be a 3× lower TTFT and 40% more aggregate throughput.

The analogy

Picture a printing shop from the early twentieth century. Setting the lead type (preparing the forme) is slow and blocks the press. Printing the pages already set is fast, but it needs the forme ready before it can start.

Prefill is setting the type. Decode is printing. A press that can only do one thing at a time, either set or print, leaves the machinery idle half the time. The historical solution was to have one worker setting the next page while the previous one was already on the press. That is, exactly, chunked prefill.

What prefill really is

When a request arrives at vLLM, the engine has to process every token of the prompt at once before it can emit the first token of the response. During that processing it computes, for each prompt token, its attention Key and Value vectors. The result, the initial KV cache, is stored in VRAM and used throughout the subsequent decode.

Unlike decode, where the model processes one new token per step, in prefill it processes N tokens in one go. That makes it far more efficient in FLOPs/token (GPUs are good at large matmuls), but it carries a quadratic cost in attention:

$$\text{prefill attention FLOPs} \approx 4 \cdot N^2 \cdot d_{model}$$

With a prompt of 1,000 tokens and $d_{model} = 4096$ (Qwen2.5-7B): $4 \cdot 10^6 \cdot 4096 \approx 16 \times 10^9$ FLOPs in attention alone. With 4,000 tokens, 256× more, because of the quadratic nature.

Prefill (compute-bound):
prompt tokens → [attention O(N²)] → [FFN] → initial KV cache
Decode (memory-bound):
1 new token → [cross-attention over KV cache] → next token

Why prefill is a problem on small hardware

On an H100 with 3.35 TB/s of bandwidth, a long prefill amortises quickly. On an RTX 4090 (1.008 TB/s) or an L40 (864 GB/s), the bottleneck shows up sooner and has concrete consequences:

The head-of-line blocking problem. By default, vLLM processes a complete prefill before doing any decode. If you have 10 requests queued, 9 in decode and 1 with an 8,000-token prompt, those 9 requests stall while the GPU chews through the long prefill. Their users see the streaming freeze. This is called head-of-line blocking and it is the number one enemy of TTFT in production.

The four levers

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

Chunked prefill splits a long prefill into pieces (chunks) and interleaves them with decode steps in the same batch. In vLLM V1 (≥ 0.6) it is on by default.

vllm serve my-model \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096

--max-num-batched-tokens is the total token budget vLLM can process in a single engine step, counting prefill and decode together. It is the most important parameter for controlling the trade-off:

`max-num-batched-tokens`	Effect
Low (512–2048)	More decode steps per cycle → better ITL, worse TTFT
High (8192–32768)	Large prefill chunks → better TTFT and throughput, worse ITL

For an RTX 4090 serving 7B–13B models with mixed contexts (256–4096 tokens):

--max-num-batched-tokens 8192 # a reasonable balance point

For an L40 (48 GB) with larger models and longer prompts:

--max-num-batched-tokens 16384

How it works internally: with a budget of 4,096 tokens and a prefill of 10,000, vLLM splits it into 3 chunks (4,096 + 4,096 + 1,808). Between chunks, it processes the pending decode steps. The requests in decode keep advancing; the long prefill takes longer to finish, but it freezes nothing.

Without chunked prefill:
t=0 [prefill 10k tokens]─────────────────────────────┐
t=1 └─[decode r1,r2...r9]
With chunked prefill (budget 4096):
t=0 [prefill chunk 4096][decode r1..r9]
t=1 [prefill chunk 4096][decode r1..r9]
t=2 [prefill chunk 1808][decode r1..r9]
t=3 [decode all, including the new one]

The TTFT of the long request rises slightly (3 steps instead of 1), but the ITL of the other 9 requests is not interrupted.

2. Prefix caching — `--enable-prefix-caching`

If several requests share the same prefix, a system prompt, a few-shot block, a context document, vLLM can compute the KV cache of that prefix once and reuse it.

vllm serve my-model \
 --enable-prefix-caching

This is called Automatic Prefix Caching (APC). Internally, vLLM splits the KV cache into fixed-size blocks (16 tokens per block by default) and assigns them a content-based SHA hash. When a new request arrives, it checks whether any initial block is already in the cache. On a hit, that prefill is skipped.

The numerical impact: suppose a 512-token system prompt that appears in 80% of your requests, and a rate of 100 req/min:

Without APC: 80 req/min × 512 tokens × prefill_cost = 41,000 tokens/min of redundant prefill
With APC (80% hit rate): 20 req/min × 512 = 10,240 tokens/min of prefill → a 75% reduction

The TTFT of those 80 requests falls to whatever it costs to process only the new suffix.

Limitation with chunked prefill: when chunked prefill is active, only the first chunk of the prefill benefits from APC in the current vLLM implementation. For workloads where the cache hit rate is very high and the suffixes are short, consider lowering --max-num-batched-tokens so that the first chunk covers more of the shared prefix.

# Configuration optimised for a high prefix cache hit rate
vllm serve my-model \
 --enable-prefix-caching \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096 # smaller chunks = prefix fits in chunk 1

3. FP8 KV cache — `--kv-cache-dtype fp8`

The KV cache takes up VRAM. The more VRAM it consumes, the fewer concurrent requests you can keep in flight. On a 24 GB RTX 4090, the Qwen2.5-14B model in BF16 already takes ~28 GB, so it does not fit. In Q4 it takes ~9 GB, leaving ~14 GB for the KV cache.

How many context tokens fit in 14 GB of BF16 KV cache for a 14B with GQA?

$$\text{KV size per token} = 2 \cdot n_{kv\_heads} \cdot d_{head} \cdot n_{layers} \cdot 2 \text{ bytes}$$

For Qwen2.5-14B: $n_{kv\_heads}=8$, $d_{head}=128$, $n_{layers}=40$, BF16 → $2 \cdot 8 \cdot 128 \cdot 40 \cdot 2 = 163{,}840$ bytes ≈ 160 KB/token.

14 GB / 160 KB ≈ 87,500 tokens of total context. With 8 users in parallel and 4,096 tokens of context each: 32,768 tokens used out of 87,500. There is headroom, but it is finite.

Moving to FP8 (1 byte instead of 2):

$$\text{KV FP8} = 80 \text{ KB/token} \implies 14 \text{ GB} / 80 \text{ KB} = 175{,}000 \text{ tokens}$$

Twice the context capacity with the same VRAM. That buys either more concurrency or longer contexts.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # calibrates the scales dynamically; without this there is degradation

Warning for RTX 4090 and L40: Ada Lovelace has FP8 instructions at the CUDA level but without Hopper’s (H100’s) dedicated scaling hardware. The memory reduction is real; the compute speedup is smaller than on an H100. Do not expect the same speedup as in a Hopper datacenter. On the L40S (the variant with optimised FP8 tensor cores) the benefit is larger than on an RTX 4090.

4. Context budget — `--max-model-len`

--max-model-len defines the maximum number of tokens vLLM can handle in a single request (prompt plus generation). It is the hard limit that determines how much VRAM is reserved for the KV cache in the worst case.

On small hardware, lowering it frees VRAM for more concurrency:

# 7B model on RTX 4090, typical context of 4K but the model supports 128K
vllm serve my-model \
 --max-model-len 8192 \  # instead of 131072
 --gpu-memory-utilization 0.92

With the context cut to 8,192 tokens, vLLM does not reserve KV cache for 131,072 potential tokens and can fit more simultaneous requests. The risk is obvious: requests above 8,192 tokens fail with an error. Tune it to the P99 of your real length distribution.

Interaction between parameters

The four parameters are not independent. A common mistake is enabling prefix caching without tuning the block size, or raising max-num-batched-tokens without checking that max-num-seqs allows it to be filled:

max-num-batched-tokens = 8192
max-num-seqs = 4
average prompt = 512 tokens → 4 × 512 = 2048 prefill tokens < 8192
The 8192 budget is never filled because max-num-seqs caps it first.
Fix: raise max-num-seqs or lower max-num-batched-tokens.

A balanced configuration for RTX 4090 + a 7B model:

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 8192 \
 --max-num-seqs 64 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Configuration for an L40 (48 GB) + a 14B model:

vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 16384 \
 --max-num-seqs 128 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

How to measure that it is working

The metrics that confirm prefill is optimised:

# In vLLM's Prometheus metrics (port 8000/metrics):
vllm:time_to_first_token_seconds_bucket → TTFT distribution
vllm:gpu_cache_usage_perc → KV cache utilisation
vllm:prefix_cache_hit_rate → APC hit rate (if enabled)
vllm:num_running_seqs → simultaneous in-flight requests

A prefix_cache_hit_rate below 30% on workloads with a fixed system prompt means something in the hash is not working (a system prompt that varies by timestamp, a date format in the prompt, and so on).

Implications for on-premise inference

Chunked prefill and prefix caching are zero cost: they are enabled with flags and require no extra hardware. FP8 KV cache requires the model to be compatible (almost every modern transformer is) and that you are on Ada Lovelace or newer.

For sovereign ENS deployments where the hardware is fixed and you cannot scale horizontally on demand, a well-configured prefill is the difference between needing 4 nodes and needing 2 for the same load.

The second article in this series covers the decode optimisations: speculative decoding, KV cache tuning to maximise concurrency and how to configure gpu-memory-utilization without vLLM running out of VRAM at midnight.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
Sarathi-Serve: Chunked Prefill and Stall-Free Scheduling — Agrawal et al., 2024
vLLM Optimization and Tuning — documentación oficial
Quantized KV Cache en vLLM

Prefix cache: hit rate engineering to go from 15% to 75%

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

vLLM’s prefix cache stores the KV cache blocks of shared prefixes and reuses them in later requests. A hit avoids recomputing that prefix: TTFT drops to the cost of the variable suffix only. In enterprise workloads with fixed system prompts (RAG, domain chatbots, assistants with long instructions) the hit rate should be >70%. In practice it is 10-20% for entirely avoidable reasons. This article identifies them, fixes them, and gives the OTel queries to confirm the result.

The analogy

A simultaneous conference interpreter who has to translate the speeches of twenty speakers. All of them open with the same two-page protocol preamble: the conference declaration, the rules of conduct, the day’s programme. An interpreter with no memory rereads the two pages for each speaker before starting on their specific speech. An interpreter with good notes reads them once, files them, and when the second speaker starts goes straight to the speech.

The prefix cache is that file. The prefix hash is the reference that lets you skip to the new part. But if the preamble changes by even one word, because someone drops in the day’s date, the interpreter has to reread everything from the beginning.

How the prefix cache hash works

vLLM splits the KV cache into blocks of 16 tokens. Each block has a hash computed over its exact content. When a new request arrives, vLLM checks whether any initial block of the prompt is already in cache by comparing hashes.

The hash is computed over the byte-by-byte content of the tokens. Any difference, a space, a different character, one extra token, produces a completely different hash. There is no partial matching inside a block.

Direct consequence: if your system prompt has 512 tokens and token number 3 changes between requests (because you interpolate a date, an ID, a version number), no block hits even though 99% of the text is identical.

Block 0 (tokens 0-15): hash = a3f7... ← in cache?
Block 1 (tokens 16-31): hash = 9d2c... ← in cache?
...
Block 31 (tokens 496-511): hash = 7e1a... ← in cache?

If block 0 does not hit (because its content changed), blocks 1-31 are not even checked although they are identical. The prefix cache is sequential.

Audit: why your real hit rate is low

Before changing anything, you need to know what is breaking the hash. The most direct method: pull the last 1000 production prompts and work out what fraction of the prefix varies.

# audit_prefix_cache.py
import langfuse, hashlib, collections
from transformers import AutoTokenizer

client = langfuse.Langfuse()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")

traces = client.fetch_traces(limit=1000).data
prompts = [t.input for t in traces if t.input]

# Tokenise and extract the first 512 tokens (the typical system prompt)
prefixes = []
for prompt in prompts:
 tokens = tokenizer.encode(prompt, add_special_tokens=False)
 prefix_tokens = tuple(tokens[:512])
 prefixes.append(prefix_tokens)

# How many unique prefixes are there?
unique = len(set(prefixes))
total = len(prefixes)
print(f"Unique prefixes: {unique}/{total} ({unique/total*100:.1f}%)")
print(f"Theoretical hit rate if they were all equal: {(1 - unique/total)*100:.1f}%")

# Find which token differs between the most common prefix and the rest
from collections import Counter
most_common_prefix = Counter(prefixes).most_common(1)[0][0]

divergence_positions = []
for prefix in prefixes:
 if prefix == most_common_prefix:
 continue
 for i, (a, b) in enumerate(zip(most_common_prefix, prefix)):
 if a != b:
 divergence_positions.append(i)
 break

if divergence_positions:
 pos = Counter(divergence_positions).most_common(1)[0][0]
 token_text = tokenizer.decode([most_common_prefix[pos]])
 print(f"\nMost frequent divergence at position {pos}: '{token_text}'")
 print("→ The token at that position varies between requests")

The most common culprits, in order of frequency:

1. Timestamps and dates:

# ❌ Breaks the hash on every request
system = f"Current date: {datetime.now().strftime('%Y-%m-%d %H:%M')}. You are an assistant..."

# ✅ Take the date out of the system prompt
system = "You are an assistant specialising in cloud infrastructure."
# Pass the date as part of the user message if it is needed

2. Session and user IDs:

# ❌
system = f"User ID: {user_id}. Preferences: {user_prefs}. You are an assistant..."

# ✅ Separate the static from the contextual
system = "You are a specialised assistant." # always the same
# Add the user context as the first message of the history

3. Interpolated prompt versions:

# ❌
system = f"[v{PROMPT_VERSION}] You are an assistant..." # changes with every deploy

# ✅ Do not version in the text, version in the prompt name in Langfuse
system = "You are an assistant..."

4. Dynamic few-shots:

# ❌ Examples retrieved at random from a pool
examples = random.sample(example_pool, k=3)
system = f"Examples:\n{format_examples(examples)}\n\nYou are an assistant..."

# ✅ Fixed few-shots always in the same order
FIXED_EXAMPLES = [example_pool[0], example_pool[1], example_pool[2]]
system = f"Examples:\n{format_examples(FIXED_EXAMPLES)}\n\nYou are an assistant..."

Template engineering: the structure that maximises hits

The principle is simple: everything static goes first, everything dynamic goes after. The prefix cache is sequential. Once a block fails to hit, the rest are not looked up either.

OPTIMAL STRUCTURE for maximising prefix cache:
┌──────────────────────────────────────────────┐
│ STATIC BLOCK (tokens 0-511) │ ← hit rate ~100%
│ Invariant system prompt │
│ Fixed instructions │
│ Few-shots always in the same order │
├──────────────────────────────────────────────┤
│ SEMI-STATIC BLOCK (tokens 512-1023) │ ← hit rate ~60-80%
│ RAG documents for this session │
│ Conversation history so far │
├──────────────────────────────────────────────┤
│ DYNAMIC BLOCK (tokens 1024+) │ ← hit rate ~0% (expected)
│ Current user message │
│ Context specific to this request │
└──────────────────────────────────────────────┘

For RAG specifically: if the retrieved documents are the same across a set of similar queries (very frequent in RAG over fixed corporate documents), ordering them always in the same order (by ID, by a fixed score, not by a variable score) multiplies the hit rate of the semi-static block.

Prefix-aware routing: the next level

With a single vLLM instance, the prefix cache works automatically. The problem shows up with multiple replicas: the load balancer distributes requests round-robin, and the prefix cached on replica A is no use at all when the request lands on replica B.

The solution is prefix-aware routing: send requests with the same prefix to the same node.

With Ray Serve (native integration):

# ray_serve_prefix_router.py
from ray import serve
from ray.serve.llm import LLMConfig, build_llm_deployment

@serve.deployment
class PrefixAwareRouter:
 def __init__(self, replicas):
 self.replicas = replicas # list of vLLM handles

 async def __call__(self, request):
 body = await request.json()
 messages = body.get("messages", [])

 # Compute the hash of the system prompt (static prefix)
 system_content = ""
 for msg in messages:
 if msg["role"] == "system":
 system_content = msg["content"]
 break

 prefix_hash = hash(system_content)
 # Deterministic routing: same hash → same node
 replica_idx = prefix_hash % len(self.replicas)
 return await self.replicas[replica_idx].remote(request)

With an L7 gateway (Nginx/Traefik):

# nginx.conf — routing by X-Prefix-Hash header
upstream vllm_backends {
 hash $http_x_prefix_hash consistent;
 server vllm-0:8000;
 server vllm-1:8000;
 server vllm-2:8000;
 server vllm-3:8000;
}

The client computes the hash of the static prefix and includes it as a header:

import hashlib, requests

def llm_request(messages, base_url):
 system_msg = next((m["content"] for m in messages if m["role"] == "system"), "")
 prefix_hash = hashlib.sha256(system_msg.encode()).hexdigest()[:16]

 return requests.post(
 f"{base_url}/v1/chat/completions",
 json={"messages": messages, "model": "my-model"},
 headers={"X-Prefix-Hash": prefix_hash}
 )

Measuring the impact with OTel

# Current hit rate (0.0 to 1.0) — target > 0.70 with enterprise workloads
vllm:gpu_prefix_cache_hit_rate

# TTFT by percentile — should fall when the hit rate rises
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse correlation between hit rate and TTFT is the proof that the cache is working. If the hit rate goes from 15% to 70% and the p50 TTFT does not change, there is a configuration problem: the cache may be disabled, or the routing may not be sending requests to the right node.

Correlation query in Grafana (two-axis panel):

# Left Y axis: hit rate
vllm:gpu_prefix_cache_hit_rate

# Right Y axis: TTFT p50 (inverted)
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse slope should be visible: when the hit rate drops (a spike of requests with new prompts), TTFT rises. When the hit rate stabilises (users repeating the same flow), TTFT falls.

The impact in numbers

For a system with 100 req/min, a 512-token system prompt and a before/after hit rate:

Metric	Hit rate 15%	Hit rate 75%	Difference
Prefill tokens per minute	5,100	12,800 — 50% cached → 6,400 effective	−37% load
TTFT p50 (512 prompt + 100 suffix)	~820 ms	~180 ms (suffix only)	−78%
Prefill capacity freed	—	+1,200 tok/min	available for more requests

The 75% hit rate in this example is equivalent to being able to serve 37% more requests on the same hardware, because the prefill work of 3 out of every 4 requests is already done.

When the prefix cache does not help

The prefix cache is ineffective in workloads where every request has a completely unique prompt: translating a different document every time, code analysis with always-different context, creative generation with no system prompt. In these cases the hit rate structurally cannot exceed 5-10% and the template engineering effort does not pay off.

The signal: if your p99 input length is greater than the p50, you have high prompt variance and the prefix cache contributes little. If the p50 and the p99 are similar (consistent prompts), the prefix cache is the cheapest lever available.

References

Pruning LLM models: cutting without amputating

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

A 7B-parameter model has tens of billions of neural connections. Many of them contribute so little that you could remove them without any reasonable benchmark noticing. That is pruning: identifying the irrelevant weights and deleting them to obtain a model that is smaller, faster or lighter on memory. Modern techniques (SparseGPT, Wanda, 2:4 structured sparsity) do this without retraining, in a few GPU hours, and with less than 1 point of perplexity penalty. It does not replace quantisation; it combines with it.

The analogy

An oak tree with a hundred branches. When winter comes, the tree prunes its weak branches: it redirects resources towards the main trunks. An expert pruner does not cut at random, they look at which branches have little foliage, which are dry, which grow in the wrong direction, and cut only those.

A language model is that tree. Its “branches” are the weights connecting neurons. After training, many of those connections are vestiges of the optimisation process: they existed so the gradient could descend smoothly, but in production they barely change the output. The pruner that removes them precisely is SparseGPT or Wanda. The one that cuts at random is magnitude pruning without calibration. Both give you a smaller tree; only the expert gives you one that still bears the same fruit.

What pruning really is

A transformer language model stores its knowledge in weight matrices. An attention layer has four matrices: $W_Q, W_K, W_V, W_O$. An FFN layer has at least two ($W_{up}, W_{down}$, plus $W_{gate}$ in SwiGLU). For a 7B model with 32 layers, the number of individual parameters exceeds 7 billion.

Pruning is the process of setting a subset of those parameters to zero so that:

The resulting model takes up less memory (if stored in a sparse format) or computes fewer operations.
Answer quality does not drop appreciably.

There are two classification dimensions that matter:

Granularity: which unit gets removed.

Unstructured pruning: individual weights, scattered across the whole matrix. High compression, hard to accelerate on conventional hardware.
Structured pruning: whole attention heads, entire FFN neurons, or complete layers. Less compression, but the resulting model is dense and compatible with any hardware.
Semi-structured N:M: for each group of M consecutive weights, exactly N are zero. The 2:4 case (2 zeros out of every 4) is the one supported by the Tensor Cores in NVIDIA Ampere and later.

Timing: when the removal happens.

Post-training (weight PTQ): requires no gradient, the standard in large LLMs.
During training (gradual/iterative): more precise, incompatible with 70B+ models on cost grounds.

Why so many redundant weights exist

The answer lies in how the models are trained. Stochastic gradient descent with millions of steps and a decaying learning rate produces networks that are over-parameterised by design: the extra parameters do not represent additional knowledge, but room for manoeuvre so that optimisation converges more easily.

The Lottery Ticket Hypothesis (Frankle & Carlin, ICLR 2019) formalised this intuition: inside any trained dense network there is a subnetwork that, trained from scratch in isolation, reaches the same quality. The original network is that subnetwork wrapped in parametric noise generated by the training process.

For LLMs, the empirical evidence is consistent: 7B–70B models tolerate up to 50% unstructured sparsity with no observable degradation on conversational tasks. In larger models, the tolerance threshold rises.

The maths that matter

Which weights are safe to remove?

Magnitude pruning: the naive criterion

$$\text{importance}(w_{ij}) = |w_{ij}|$$

The weights with the smallest absolute value are removed. Intuitive, but incomplete: a small weight connected to a very large activation still contributes significantly to the output.

Wanda: magnitude × activation

$$\text{importance}(w_{ij}) = |w_{ij}| \cdot \|x_j\|_2$$

Where $x_j$ is the input activation vector corresponding to weight $j$, computed over a calibration dataset of ~128 samples. The product captures both dimensions: a weight is safe to remove only if it is small and its input neuron is barely active.

Numerical example:

Weight A: $|w| = 0.001$, $\|x\|_2 = 500$ → importance = 0.5
Weight B: $|w| = 0.01$, $\|x\|_2 = 10$ → importance = 0.1

Magnitude pruning would remove A (smaller absolute value). Wanda removes B (lower importance). B is the safer one to delete.

Wanda requires neither gradients nor Hessian matrix inverses. It runs in minutes on a 70B model on a single GPU. On WikiText-2 perplexity benchmarks at 50% unstructured sparsity, Wanda reaches results comparable to SparseGPT at 10–100× less computational cost.

SparseGPT: Hessian compensation

SparseGPT applies the same mathematical framework as GPTQ (layer-by-layer quantisation), but for pruning. When it removes a weight $w_p$, it computes a correction $\delta w$ over the remaining weights in the same row to minimise the change in the layer’s output:

$$\min_{\delta w} \|W x - (W + \delta W) x\|_2^2 \quad \text{s.t.} \quad w_p + \delta w_p = 0$$

The solution uses the inverse of the second-order Hessian matrix $H = X X^T$. The extra cost is justified by the higher precision when target sparsity is high (>70%) or the model is small (<7B, where redundancy is lower).

Method	Criterion	Cost	50% sparsity (7B, WikiText-2 ppl)
Magnitude	\|w\|	Instant	+2–5 points
Wanda	\|w\| · \|x\|	Minutes	~+0.5 points
SparseGPT	Hessian	1–4h GPU	~+0.4 points

2:4 Structured Sparsity: NVIDIA’s special case

NVIDIA Ampere (A100) and later (H100, Ada Lovelace) include dedicated hardware for the 2:4 pattern: exactly 2 out of every 4 consecutive weights are zero. This lets the hardware skip the multiplications by zero efficiently, obtaining up to 2× speedup in matmul on models with 2:4 weights.

The constraint is that the sparsity has to be exactly 2:4, not an arbitrary pattern. NVIDIA’s tools (APEX Sparse, cuSPARSELt) and frameworks such as PyTorch 2.x support this natively:

from torch.sparse import to_sparse_semi_structured, SparseSemiStructuredTensor

# Convert dense weights to 2:4 sparse
sparse_weight = to_sparse_semi_structured(dense_weight)
# Forward pass automatically uses sparse tensor cores
output = F.linear(input, sparse_weight)

What to expect in practice with 2:4:

RTX 4090 (Ada Lovelace): supports 2:4 sparse tensor cores for FP16/BF16. Theoretical speedup 2×, real 1.3–1.6× depending on batch and sequence size.
H100 (Hopper): the same, with additional improvements in FP8 + 2:4 combined.
A100: supported, without FP8.
Consumer GPUs earlier than Ada (3090, etc.): no hardware support. 2:4 sparsity gives a smaller model on disk but does not accelerate inference.

Structured pruning: removing whole heads and layers

Attention head pruning

A 32-layer transformer with 32 heads per layer has 1,024 attention heads. Systematic studies on Llama-2 and Qwen models show that between 20–40% of the heads have marginal influence on the final output: their output can be set to zero without the benchmark changing within the margin of error.

The most used metric is Taylor importance: the product of the gradient of the loss with respect to the head’s output and the value of that output, summed over a calibration dataset:

$$\text{I}_{head} = \left| \sum_t \frac{\partial \mathcal{L}}{\partial o_t} \cdot o_t \right|$$

The heads with the lowest $I_{head}$ are removed first. After removing 25% of the heads in Llama-3-8B, MMLU degradation is <1% and attention inference time falls ~20% because the attention matmuls are smaller.

Layer dropping: the most aggressive shortcut

Removing a whole transformer layer deletes its attention block and its FFN. The most robust criterion is Block Influence (BI), introduced in ShortGPT (2024):

$$\text{BI}(l) = 1 - \cos(\text{input}_l, \text{output}_l)$$

A layer whose output is almost identical to its input (cosine close to 1, BI close to 0) acts as an identity function: removing it does not change the flow of information. The layers in the middle of the transformer tend to have lower BI than the initial and final layers.

Numerical example on LLaMA-2-70B:

Layers 0–5 (early): BI > 0.3 → do not remove
Layers 20–45 (mid): BI < 0.05 → candidates for removal
Layers 76–80 (final): BI > 0.2 → do not remove

Removing 8 layers out of 80 (10%): the model goes from ~140 GB to ~126 GB in BF16. Inference speedup: ~10% (proportional to the number of layers removed). Degradation on reasoning benchmarks: 1–3%.

Implications for on-premise inference

Unstructured pruning (50% sparsity) produces models with the same number of parameters but half of them at zero. Without specialised sparse kernels, that gives no speedup: the GPU still runs the multiplications, it just multiplies by zero very efficiently. The real benefit is storage and transfer (the model takes up less on disk and in system RAM).

With 2:4 structured sparsity on Ada/Hopper hardware, the speedup is real but moderate (1.3–1.7×) and requires additional tooling (cuSPARSELt or PyTorch sparse).

Structured pruning (heads, layers) does accelerate on any hardware because it reduces the model’s real size. It is the right option if the goal is throughput on hardware without sparse tensor cores.

Combining with quantisation: pruning and quantisation are orthogonal. A 50% sparse model at INT4 takes up roughly an eighth of the original in FP32. It is the end point of many aggressive compression pipelines for edge inference.

Applied to generic on-premise hardware

RTX 4090 (24 GB, Ada Lovelace)

Supports 2:4 sparse tensor cores for FP16/BF16. With Wanda + 2:4 sparsity on a Qwen2.5-14B:

# Pruning pipeline: Wanda 2:4 + INT4 quantisation
# 1. Run Wanda with calibration over 128 samples
python wanda/main.py \
 --model Qwen/Qwen2.5-14B \
 --sparsity_ratio 0.5 \
 --sparsity_type 2:4 \
 --save pruned_model/

# 2. Quantise the pruned model (optional but complementary)
python -m awq.entry --model_path pruned_model/ \
 --w_bit 4 --output_path pruned_awq_model/

Expected result: ~13 GB BF16 → ~6.5 GB after 2:4 pruning in sparse format → ~3.2 GB with AWQ INT4. The 14B model will fit in the RTX 4090 with room for KV cache.

4× H100 SXM (320 GB total, Hopper)

On this hardware structured pruning (layer dropping) makes more sense than 2:4 for high-concurrency inference: you reduce the number of FLOPs per token proportionally, which benefits throughput at large batch where the bottleneck is compute, not memory.

# Apply layer dropping with the ShortGPT BI metric
from shortgpt import compute_block_influence, drop_layers

bi_scores = compute_block_influence(model, calibration_data)
# Remove the 15% of layers with the lowest BI
model = drop_layers(model, bi_scores, drop_ratio=0.15)

A Llama-3-70B pruned by 15% of its layers fits in 3 H100s instead of 4, freeing a GPU for another task.

References

SparseGPT: Massive Language Models Can be Accurately Pruned in One Shot — Frantar & Alistarh, 2023
A Simple and Effective Pruning Approach for Large Language Models (Wanda) — Sun et al., ICLR 2024
The Lottery Ticket Hypothesis — Frankle & Carlin, ICLR 2019
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect — Men et al., 2024
NVIDIA 2:4 Sparsity in PyTorch — PyTorch Blog
SparseForge: Efficient Semi-Structured LLM Sparsification — 2025

Semantic cache in RAG: the receptionist with a photographic memory

Thu, 04 Jun 2026 08:00:00 +0200

TL;DR

In a RAG system with real traffic, 30–70% of queries are semantically equivalent to some earlier one even when the text differs. The semantic cache intercepts those queries ahead of the retriever and the LLM, returning the answer already computed if the cosine similarity with a previous query exceeds a threshold θ (typically 0.92–0.95). At 10,000 requests/day and a 45% hit rate, that amounts to not running 4,500 LLM generations: roughly 0.62 GPU hours saved every day on a cluster running Llama-3.1-70B. The fundamental trade-off is that a high θ gives more accurate answers but less saving; a low θ maximises saving but can return incorrect answers for subtly different queries.

The analogy: the receptionist with a notebook

Picture the front desk of a 400-room hotel. Over the course of the day, the receptionist takes hundreds of questions. But if you look through the log book, you will see that 60% of those questions are variants of the same ten:

“Where is the gym?”
“What time is breakfast?”
“Do you have parking?”
“How do I connect to the WiFi?”

By the third day, the receptionist has built a mental notebook of answers. When someone asks “where can I go to exercise?”, they do not call the concierge (retrieval) or open the hotel’s 300-page internal manual (LLM): they look at the notebook, work out that the question is the same as “where is the gym?”, and answer in two seconds.

But when someone asks “what time does the gym close today?”, the receptionist knows the notebook cannot be trusted: the schedule may have changed because of a private event. They have to call the concierge.

That is exactly the mechanism of a semantic cache:

The notebook is the cache store (Redis with a vector index, or a Qdrant collection).
Working out that “exercise” ≈ “gym” is the cosine similarity search with threshold θ.
Calling the concierge is retrieval over the corpus.
Consulting the manual is LLM generation.
“Today” is the temporal query signal that invalidates the cache.

The threshold θ is exactly what separates “where is it” (semantically the same) from “what time is it open today” (semantically different). It is not magic: it is vector arithmetic over learned representations.

The problem in production

A typical RAG pipeline has three layers of latency and compute: embedding the query, vector search over the corpus, and generation with the LLM. In development, that cost is irrelevant. In production with 50 concurrent users, each of those layers scales linearly with the number of requests.

The problem is that users ask the same questions over and over, with slightly different wording:

Original query	Equivalent query
“How do I configure the agent?”	“What is the process for configuring the agent?”
“error installing the dependency”	“the dependency installation fails”
“what is an embedding?”	“explain to me what embeddings are”

Running the full pipeline for each of these variants is pure waste. Empirical studies in technical support and corporate Q&A systems report that between 30% and 70% of a day’s queries are semantically redundant with respect to earlier queries from the same week.

The query distribution in real systems follows a power law similar to the Zipf distribution: the 100 most frequent topics account for roughly 60% of total traffic. A well-calibrated cache captures exactly that concentration.

How the semantic cache works

The full flow is shown in the diagram below. Let us describe it in prose first.

When a new query $q$ arrives:

Embedding the query: $q$ is embedded with the same model used to index the corpus. This is critical: if the corpus was indexed with text-embedding-3-large and the cache uses a different embedder, the vector spaces are not comparable.
Search in the cache store: an ANN (Approximate Nearest Neighbor) search is run over the vectors of previously cached queries. The most similar query $q^*$ and its cosine similarity $s$ are retrieved.
Threshold decision:
$$ \text{answer} = \begin{cases} r^* & \text{if } s(q, q^*) \geq \theta \\ \text{pipeline}(q) & \text{if } s(q, q^*) < \theta \end{cases} $$
where $r^*$ is the cached answer associated with $q^*$.
On a miss: the full pipeline runs (retrieval + LLM). The generated answer is stored in the cache with a configurable TTL for future similar queries.

The cache store is not an ordinary key-value database. It is a vector index over the query embeddings, with the values being the generated answers. Each entry has the structure:

{vector: embed(q), response: r, ttl: T, metadata: {...}}

Flow diagram

The semantic cache flow as middleware between the gateway and the retriever. Cache hits skip the vector search over the corpus and the LLM generation entirely.

The threshold θ and its trade-off

The threshold θ is the most sensitive parameter in the system. It works exactly like the receptionist’s recognition threshold: if it is too demanding, it will only identify textually identical questions and the notebook will not be much use. If it is too lax, it will return the answer to “where is the gym?” to someone who asked “what time does the gym close?”.

The cosine similarity between two vectors $\mathbf{a}$ and $\mathbf{b}$ is:

$$ s(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \cdot \|\mathbf{b}\|} $$

For prose text (Spanish or English), modern embedders such as text-embedding-3-large or nomic-embed-text assign cosine similarities of around 0.90–0.96 to semantically equivalent paraphrases and similarities of 0.75–0.88 to related but non-equivalent queries.

The cache quality metric is not just hit rate: it is precision@cache, defined as the fraction of cached answers that remain correct for the new query. A cached answer is “correct” if an evaluator (another LLM, or metrics such as BERTScore) judges it equivalent to the one the full pipeline would have generated for that specific query.

θ	Estimated hit rate	Estimated precision@cache	Effective saving
0.85	~65%	~72%	~47%
0.90	~55%	~85%	~47%
0.92	~48%	~91%	~44%
0.93	~45%	~94%	~42%
0.95	~35%	~98%	~34%
0.97	~18%	~99.5%	~18%

The effective saving is defined as $\text{hit rate} \times \text{precision@cache}$, since a hit with an incorrect answer is not a saving: it is an error that can cost more in lost trust than what was saved in GPU.

The empirical sweet spot for most corporate Q&A applications in Spanish or English lies between θ = 0.92 and θ = 0.95. In highly specialised domains where small nuances change the answer (medicine, law, finance), θ ≥ 0.95 is advisable.

The arithmetic of the saving

Let us put concrete numbers on a real system.

Base configuration:

10,000 requests/day
Technical corpus of 1 million chunks in a Qdrant index
LLM: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)
Average answer: 200 output tokens
LLM throughput on this hardware: ~400 tokens/s/GPU with batching (continuous batching enabled, see continuous-batching-fundamentos)

Cost of a request without cache:

Embedding the query takes ~2 ms on a GPU. The ANN vector search over 1 M chunks in Qdrant takes ~5 ms (measured empirically with HNSW, ef=128). Generating 200 tokens at 400 tok/s in total (4 GPUs) comes to:

$$ t_{\text{LLM}} = \frac{200 \text{ tokens}}{400 \text{ tok/s}} = 0.5 \text{ s per request} $$

If 10,000 requests reach the LLM in a day, the total GPU time spent on generation is:

$$ T_{\text{GPU}} = 10{,}000 \times 0.5 \text{ s} = 5{,}000 \text{ s} \approx 1.38 \text{ GPU hours per day} $$

With a semantic cache at θ = 0.93 and a hit rate of ~45%:

Only 5,500 requests (55%) reach the LLM:

$$ T_{\text{GPU,cache}} = 5{,}500 \times 0.5 \text{ s} = 2{,}750 \text{ s} \approx 0.76 \text{ GPU hours per day} $$

Saving:

$$ \Delta T_{\text{GPU}} = 1.38 - 0.76 = 0.62 \text{ GPU hours/day} $$

In inference compute, this amounts roughly to being able to serve 45% more users without adding hardware, or cutting inference costs by 45% if you work with external APIs billed per token.

The cost of the semantic cache itself (embedding the query + ANN search over the cache store) is ~7 ms per request, negligible against the 500 ms of generation avoided on hits.

Zipf distribution of topics:

The reason it works is the Zipf distribution of traffic. If we number topics by frequency (topic 1 = most frequent), the frequency of topic $k$ is proportional to $1/k$. With 1,000 distinct topics:

$$ \text{fraction of traffic covered by top-}N = \frac{\sum_{k=1}^{N} 1/k}{\sum_{k=1}^{1000} 1/k} \approx \frac{\ln N}{\ln 1000} = \frac{\ln N}{6.9} $$

For the top-100 topics: $\ln(100)/6.9 \approx 4.6/6.9 \approx 67\%$ of traffic. The cache does not need to cover every topic: it captures 67% of traffic while covering only 10% of the topics.

OSS stack 2026

GPTCache

GPTCache is the reference library for a standalone semantic cache. Its architecture is modular:

Embedder: ONNX Runtime with converted models (onnx/all-MiniLM-L6-v2 by default), with no GPU dependency for the cache layer.
Vector store: Faiss (local), Milvus, or Qdrant.
Scalar store: SQLite (development) or Redis (production) for metadata, TTL, and answers.
Similarity evaluation: cosine by default, configurable.

Minimal configuration in Python:

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

onnx = Onnx()
data_manager = get_data_manager(
 CacheBase("redis", url="redis://localhost:6379"),
 VectorBase("qdrant", host="localhost", collection_name="query_cache")
)
cache.init(
 embedding_func=onnx.to_embeddings,
 data_manager=data_manager,
 similarity_evaluation=SearchDistanceEvaluation(),
)
cache.set_openai_key()

GPTCache intercepts calls to the OpenAI API (or to compatible proxies) transparently. The TTL is configured at the data_manager level.

MeanCache

MeanCache (2024) extends GPTCache to multi-turn conversations. The problem with standard GPTCache is that in dialogue the “relevant” query is not just the last message but the whole context window. MeanCache computes the query embedding as the weighted mean of the embeddings of the last $k$ turns:

$$ \mathbf{e}_{\text{query}} = \frac{\sum_{i=1}^{k} w_i \cdot \mathbf{e}_{q_i}}{\sum_{i=1}^{k} w_i} $$

where $w_i$ decreases with the age of the turn. This reduces false positives in dialogues where the topic keeps shifting.

Qdrant as a dual cache store

If the RAG corpus is already in Qdrant, you can use the same instance with a separate collection for the cache. The advantages are operational: a single service to manage, the same backup and monitoring infrastructure.

The cache collection uses payload filters to implement TTL:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, Range
import time

client = QdrantClient("localhost", port=6333)

# Search the cache with a TTL filter
hits = client.search(
 collection_name="semantic_cache",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[FieldCondition(
 key="expires_at",
 range=Range(gt=time.time())
 )]
 ),
 limit=1,
 score_threshold=0.93
)

You have to run a periodic cleanup of expired entries, since Qdrant has no native TTL (unlike Redis).

Langfuse for traceability

Langfuse is the OSS standard for LLM pipeline observability (see tracing-llm-otel-genai). Every request should be tagged with whether it was a cache hit or a miss:

from langfuse import Langfuse
from langfuse.decorators import observe

langfuse = Langfuse()

@observe()
def process_query(query: str) -> dict:
 cache_result = semantic_cache.lookup(query)
 if cache_result:
 langfuse.update_current_observation(
 metadata={"cache_hit": True, "cache_score": cache_result.score}
 )
 return cache_result.response
 # full pipeline...
 langfuse.update_current_observation(
 metadata={"cache_hit": False}
 )

With this metadata, Langfuse lets you compute the real hit rate, the distribution of similarity scores, and detect whether the threshold θ needs adjusting.

On-premise hardware: a reference configuration

For an on-premise deployment with this stack, a suitable configuration for RAG with a semantic cache is:

Inference node: 4×H100 SXM (320 GB NVLink in total) for Llama-3.1-70B in FP8. Throughput ~400 tok/s in generation with continuous batching (vLLM or TGI).

Vector services node: CPU with 256 GB RAM. Qdrant for the corpus (1–10 M chunks) and for the cache store (up to 500K entries in memory). Redis 7.x for metadata and an exact-match cache as a first layer.

Embedding node: CPU or a mid-range GPU (A10G). The cache embedder can run on ONNX Runtime on CPU with no perceptible impact on latency (~2 ms per embedding).

Separating the cache store from the corpus matters: the corpus holds millions of chunks with large HNSW indexes; the cache store holds at most tens of thousands of queries with a much smaller index and search times of 1–2 ms.

Cases where the cache fails

The receptionist with a notebook fails in three well-defined scenarios:

1. Queries with a temporal context

“What is the current status of the incident?” or “What changed in the latest release?” are questions whose correct answer changes over time. A cache with a 24-hour TTL could return stale information.

The solution is to detect temporal markers in the query (regular expressions over “today”, “now”, “current”, “latest”, “yesterday”, and their Spanish equivalents) and force a cache miss for these queries, whatever the similarity score.

2. Personalised queries over private data

If the RAG has access to user data (account history, private documents), two different users asking the same question must get different answers. A shared cache that ignores user context is a privacy risk.

The solution is a cache partitioned by user_id or tenant_id. This lowers the hit rate (each user’s cache is smaller) but it is the only safe option in multi-tenant architectures.

3. TTL and a stale corpus

When the corpus is updated (new documents are ingested, errors are corrected), cached answers can go out of date. A fixed TTL (24–48 hours) mitigates the problem but does not remove it.

For corpora with frequent updates, the solution is an active invalidation mechanism: when the corpus is updated in Qdrant, a job runs that identifies which cache entries might be affected (by semantic overlap with the updated chunks) and deletes them. This is the “selective cache invalidation” mentioned in the section on topics not covered.

Integration into the pipeline as middleware

The semantic cache is implemented as middleware between the API gateway and the retriever. It does not change the API contract: the client still sends queries and receives answers in the same format.

class SemanticCacheMiddleware:
 def __init__(self, cache_store, retriever, llm, threshold=0.93):
 self.cache = cache_store
 self.retriever = retriever
 self.llm = llm
 self.threshold = threshold

 async def process(self, query: str, context: dict) -> dict:
 # First layer: exact-match cache (Redis GET, O(1))
 exact = await self.cache.exact_lookup(query)
 if exact:
 return {**exact, "cache_type": "exact"}

 # Second layer: semantic cache (ANN search)
 query_embedding = await self.embed(query)
 semantic = await self.cache.semantic_lookup(
 query_embedding, threshold=self.threshold
 )
 if semantic:
 return {**semantic, "cache_type": "semantic"}

 # Miss: full pipeline
 chunks = await self.retriever.retrieve(query_embedding)
 response = await self.llm.generate(query, chunks)

 # Store for future queries
 await self.cache.store(
 embedding=query_embedding,
 query=query,
 response=response,
 ttl=context.get("ttl", 86400)
 )
 return {**response, "cache_type": "miss"}

The first exact-match layer (Redis GET) is an extra optimisation: for textually identical queries, the embedding is not even computed. The cost is a Redis operation taking microseconds. Only if there is no exact match does it fall through to the semantic lookup.

What we have not covered

Selective cache invalidation: when a subset of the corpus is updated (for example, the documents for a specific product are reindexed), you would need to identify which cache entries overlap semantically with the updated chunks and mark them stale. The mechanism involves computing similarity between the embeddings of the updated chunks and the embeddings of the cached queries, which is expensive at scale.
Multi-tenant cache: isolation vs sharing: in a SaaS product with multiple customers, a shared cache maximises the hit rate but can expose one tenant’s answers to another if filtering is not done correctly. A cache partitioned by tenant is safe but has much lower hit rates. The middle ground is a shared cache with ACL filtering applied over Qdrant’s payload filters.
Semantic cache for streaming responses: when the LLM emits tokens in streaming mode (SSE), the cache cannot easily intercept the complete answer. The options are: cache on the first miss and return the whole answer at once on hits (breaking the streaming experience), or implement a “fake streaming” that emits the tokens of the cached answer at a controlled rate.
Exact-match cache as a first layer: ahead of the semantic cache, an O(1) lookup in Redis with the query as the key can catch textually identical queries at negligible cost. The code in the previous section already shows this two-layer architecture.

References

Bang, J. et al. (2024). MeanCache: User-Centric Semantic Cache for Large Language Model Based Web Applications. arXiv:2403.02694.
Zilliz. (2023). GPTCache: A Library for Creating Semantic Cache for LLM Queries. GitHub: zilliztech/GPTCache.
Qdrant Team. (2024). Qdrant Documentation: Filtering with payload. qdrant.tech/documentation.
Manning, C. D., Raghavan, P., Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press. Cap. 19: Web search (distribución Zipf).
Langfuse. (2024). Observability for LLM Applications. langfuse.com/docs.
Meta AI. (2024). Llama 3.1 Model Card. ai.meta.com.
Guo, Y. et al. (2023). Evaluating the Factual Consistency of Large Language Models Through Summarization. Referencia para BERTScore como métrica de evaluación de respuestas cacheadas.

Optimization on lo0 — Tech Blog

Batch sizing in vLLM: the two-hour grid search worth weeks of hardware

TL;DR

The analogy

The problem: the defaults are not for your hardware

The arithmetic that matters

The grid search: methodology

Step 1: measure the real workload

Step 2: work out the KV budget

Step 3: the grid

Step 4: read the surface

Confirmation with OTel

Reference configurations by profile

When not to touch the defaults

See also

In this same series

References

Instrumenting vLLM with OTel: measuring what the optimisations really do

TL;DR

The analogy

Architecture of the two signals

Installation and basic configuration

OTel Collector: minimum configuration

The five metrics that matter

1. Chunked prefill → vllm:time_to_first_token_seconds

2. Prefix caching → vllm:gpu_prefix_cache_hit_rate

3. Speculative decoding → vllm:spec_decode_draft_acceptance_rate

4. FP8 KV cache and concurrency → vllm:gpu_cache_usage_perc + vllm:num_preemptions_total

5. Effective concurrency → vllm:num_running_seqs + vllm:num_waiting_seqs

Reference dashboard: the 5 metrics in Grafana

Connecting traces to Langfuse

Quick diagnostic matrix

Implications for sovereign on-premise inference

See also

References

Knowledge Distillation: teaching a small model to think like a big one

TL;DR

The analogy

What it really is

The role of temperature

The three modes of distillation

Offline (or “black-box”)

Online (or “white-box”)

On-policy

Why the best small models use distillation

Reasoning distillation: the case of thinking models

When to use distillation vs. the alternatives

Implications for on-premise inference

See also

References

Optimising decode in vLLM: squeezing every token out of small hardware

TL;DR

The analogy

Why decode is memory-bound

The five levers

1. Give vLLM all the VRAM you can — --gpu-memory-utilization

2. Real concurrency — --max-num-seqs

3. Speculative decoding — --speculative-model + --num-speculative-tokens

4. Quantised KV cache — --kv-cache-dtype fp8

5. Remove the swap — --swap-space 0

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

L40 (48 GB) — 14B model, multi-user

How to measure that decode is optimised

Implications for on-premise inference

See also

References

Optimising prefill in vLLM: the knobs your TTFT will not forgive

TL;DR

The analogy

What prefill really is

Why prefill is a problem on small hardware

The four levers

1. Chunked prefill — --enable-chunked-prefill + --max-num-batched-tokens

2. Prefix caching — --enable-prefix-caching

3. FP8 KV cache — --kv-cache-dtype fp8

4. Context budget — --max-model-len

Interaction between parameters

How to measure that it is working

Implications for on-premise inference

1. Chunked prefill → `vllm:time_to_first_token_seconds`

2. Prefix caching → `vllm:gpu_prefix_cache_hit_rate`

3. Speculative decoding → `vllm:spec_decode_draft_acceptance_rate`

4. FP8 KV cache and concurrency → `vllm:gpu_cache_usage_perc` + `vllm:num_preemptions_total`

5. Effective concurrency → `vllm:num_running_seqs` + `vllm:num_waiting_seqs`

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

2. Real concurrency — `--max-num-seqs`

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

4. Quantised KV cache — `--kv-cache-dtype fp8`

5. Remove the swap — `--swap-space 0`

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

2. Prefix caching — `--enable-prefix-caching`

3. FP8 KV cache — `--kv-cache-dtype fp8`

4. Context budget — `--max-model-len`