Prefix cache: hit rate engineering to go from 15% to 75%

TL;DR

vLLM’s prefix cache stores the KV cache blocks of shared prefixes and reuses them in later requests. A hit avoids recomputing that prefix: TTFT drops to the cost of the variable suffix only. In enterprise workloads with fixed system prompts (RAG, domain chatbots, assistants with long instructions) the hit rate should be >70%. In practice it is 10-20% for entirely avoidable reasons. This article identifies them, fixes them, and gives the OTel queries to confirm the result.

The analogy

A simultaneous conference interpreter who has to translate the speeches of twenty speakers. All of them open with the same two-page protocol preamble: the conference declaration, the rules of conduct, the day’s programme. An interpreter with no memory rereads the two pages for each speaker before starting on their specific speech. An interpreter with good notes reads them once, files them, and when the second speaker starts goes straight to the speech.

The prefix cache is that file. The prefix hash is the reference that lets you skip to the new part. But if the preamble changes by even one word, because someone drops in the day’s date, the interpreter has to reread everything from the beginning.

How the prefix cache hash works

vLLM splits the KV cache into blocks of 16 tokens. Each block has a hash computed over its exact content. When a new request arrives, vLLM checks whether any initial block of the prompt is already in cache by comparing hashes.

The hash is computed over the byte-by-byte content of the tokens. Any difference, a space, a different character, one extra token, produces a completely different hash. There is no partial matching inside a block.

Direct consequence: if your system prompt has 512 tokens and token number 3 changes between requests (because you interpolate a date, an ID, a version number), no block hits even though 99% of the text is identical.

Block 0 (tokens 0-15):  hash = a3f7...  ← in cache?
Block 1 (tokens 16-31): hash = 9d2c...  ← in cache?
...
Block 31 (tokens 496-511): hash = 7e1a... ← in cache?

If block 0 does not hit (because its content changed), blocks 1-31 are not even checked although they are identical. The prefix cache is sequential.

Audit: why your real hit rate is low

Before changing anything, you need to know what is breaking the hash. The most direct method: pull the last 1000 production prompts and work out what fraction of the prefix varies.

# audit_prefix_cache.py
import langfuse, hashlib, collections
from transformers import AutoTokenizer

client = langfuse.Langfuse()
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-14B-Instruct")

traces = client.fetch_traces(limit=1000).data
prompts = [t.input for t in traces if t.input]

# Tokenise and extract the first 512 tokens (the typical system prompt)
prefixes = []
for prompt in prompts:
    tokens = tokenizer.encode(prompt, add_special_tokens=False)
    prefix_tokens = tuple(tokens[:512])
    prefixes.append(prefix_tokens)

# How many unique prefixes are there?
unique = len(set(prefixes))
total  = len(prefixes)
print(f"Unique prefixes: {unique}/{total} ({unique/total*100:.1f}%)")
print(f"Theoretical hit rate if they were all equal: {(1 - unique/total)*100:.1f}%")

# Find which token differs between the most common prefix and the rest
from collections import Counter
most_common_prefix = Counter(prefixes).most_common(1)[0][0]

divergence_positions = []
for prefix in prefixes:
    if prefix == most_common_prefix:
        continue
    for i, (a, b) in enumerate(zip(most_common_prefix, prefix)):
        if a != b:
            divergence_positions.append(i)
            break

if divergence_positions:
    pos = Counter(divergence_positions).most_common(1)[0][0]
    token_text = tokenizer.decode([most_common_prefix[pos]])
    print(f"\nMost frequent divergence at position {pos}: '{token_text}'")
    print("→ The token at that position varies between requests")

The most common culprits, in order of frequency:

1. Timestamps and dates:

# ❌ Breaks the hash on every request
system = f"Current date: {datetime.now().strftime('%Y-%m-%d %H:%M')}. You are an assistant..."

# ✅ Take the date out of the system prompt
system = "You are an assistant specialising in cloud infrastructure."
# Pass the date as part of the user message if it is needed

2. Session and user IDs:

# ❌
system = f"User ID: {user_id}. Preferences: {user_prefs}. You are an assistant..."

# ✅ Separate the static from the contextual
system = "You are a specialised assistant."  # always the same
# Add the user context as the first message of the history

3. Interpolated prompt versions:

# ❌
system = f"[v{PROMPT_VERSION}] You are an assistant..."  # changes with every deploy

# ✅ Do not version in the text, version in the prompt name in Langfuse
system = "You are an assistant..."

4. Dynamic few-shots:

# ❌ Examples retrieved at random from a pool
examples = random.sample(example_pool, k=3)
system = f"Examples:\n{format_examples(examples)}\n\nYou are an assistant..."

# ✅ Fixed few-shots always in the same order
FIXED_EXAMPLES = [example_pool[0], example_pool[1], example_pool[2]]
system = f"Examples:\n{format_examples(FIXED_EXAMPLES)}\n\nYou are an assistant..."

Template engineering: the structure that maximises hits

The principle is simple: everything static goes first, everything dynamic goes after. The prefix cache is sequential. Once a block fails to hit, the rest are not looked up either.

OPTIMAL STRUCTURE for maximising prefix cache:
┌──────────────────────────────────────────────┐
│  STATIC BLOCK (tokens 0-511)                 │ ← hit rate ~100%
│  Invariant system prompt                     │
│  Fixed instructions                          │
│  Few-shots always in the same order          │
├──────────────────────────────────────────────┤
│  SEMI-STATIC BLOCK (tokens 512-1023)         │ ← hit rate ~60-80%
│  RAG documents for this session              │
│  Conversation history so far                 │
├──────────────────────────────────────────────┤
│  DYNAMIC BLOCK (tokens 1024+)                │ ← hit rate ~0% (expected)
│  Current user message                        │
│  Context specific to this request            │
└──────────────────────────────────────────────┘

For RAG specifically: if the retrieved documents are the same across a set of similar queries (very frequent in RAG over fixed corporate documents), ordering them always in the same order (by ID, by a fixed score, not by a variable score) multiplies the hit rate of the semi-static block.

Prefix-aware routing: the next level

With a single vLLM instance, the prefix cache works automatically. The problem shows up with multiple replicas: the load balancer distributes requests round-robin, and the prefix cached on replica A is no use at all when the request lands on replica B.

The solution is prefix-aware routing: send requests with the same prefix to the same node.

With Ray Serve (native integration):

# ray_serve_prefix_router.py
from ray import serve
from ray.serve.llm import LLMConfig, build_llm_deployment

@serve.deployment
class PrefixAwareRouter:
    def __init__(self, replicas):
        self.replicas = replicas  # list of vLLM handles
    
    async def __call__(self, request):
        body = await request.json()
        messages = body.get("messages", [])
        
        # Compute the hash of the system prompt (static prefix)
        system_content = ""
        for msg in messages:
            if msg["role"] == "system":
                system_content = msg["content"]
                break
        
        prefix_hash = hash(system_content)
        # Deterministic routing: same hash → same node
        replica_idx = prefix_hash % len(self.replicas)
        return await self.replicas[replica_idx].remote(request)

With an L7 gateway (Nginx/Traefik):

# nginx.conf — routing by X-Prefix-Hash header
upstream vllm_backends {
    hash $http_x_prefix_hash consistent;
    server vllm-0:8000;
    server vllm-1:8000;
    server vllm-2:8000;
    server vllm-3:8000;
}

The client computes the hash of the static prefix and includes it as a header:

import hashlib, requests

def llm_request(messages, base_url):
    system_msg = next((m["content"] for m in messages if m["role"] == "system"), "")
    prefix_hash = hashlib.sha256(system_msg.encode()).hexdigest()[:16]
    
    return requests.post(
        f"{base_url}/v1/chat/completions",
        json={"messages": messages, "model": "my-model"},
        headers={"X-Prefix-Hash": prefix_hash}
    )

Measuring the impact with OTel

# Current hit rate (0.0 to 1.0) — target > 0.70 with enterprise workloads
vllm:gpu_prefix_cache_hit_rate

# TTFT by percentile — should fall when the hit rate rises
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))
histogram_quantile(0.95, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse correlation between hit rate and TTFT is the proof that the cache is working. If the hit rate goes from 15% to 70% and the p50 TTFT does not change, there is a configuration problem: the cache may be disabled, or the routing may not be sending requests to the right node.

Correlation query in Grafana (two-axis panel):

# Left Y axis: hit rate
vllm:gpu_prefix_cache_hit_rate

# Right Y axis: TTFT p50 (inverted)
histogram_quantile(0.50, rate(vllm:time_to_first_token_seconds_bucket[5m]))

The inverse slope should be visible: when the hit rate drops (a spike of requests with new prompts), TTFT rises. When the hit rate stabilises (users repeating the same flow), TTFT falls.

The impact in numbers

For a system with 100 req/min, a 512-token system prompt and a before/after hit rate:

The 75% hit rate in this example is equivalent to being able to serve 37% more requests on the same hardware, because the prefill work of 3 out of every 4 requests is already done.

When the prefix cache does not help

The prefix cache is ineffective in workloads where every request has a completely unique prompt: translating a different document every time, code analysis with always-different context, creative generation with no system prompt. In these cases the hit rate structurally cannot exceed 5-10% and the template engineering effort does not pay off.

The signal: if your p99 input length is greater than the p50, you have high prompt variance and the prefix cache contributes little. If the p50 and the p99 are similar (consistent prompts), the prefix cache is the cheapest lever available.

See also

• FlashInfer’s cascade attention: the compute side of the shared prefix that the prefix cache solves in memory — attending the common prefix once instead of R times

• --enable-prefix-caching and the interaction with chunked prefill: only the first chunk benefits from the cache, which affects the optimal budget for max-num-batched-tokens

• the block structure the prefix cache operates on: why the 16-token granularity matters for template design

• the grid search that determines the optimal max-num-seqs, which interacts with the number of blocks available for the cache

• the L7 gateway where prefix-aware routing via header is implemented

• how to configure gpu_prefix_cache_hit_rate on the Grafana dashboard and the alert for when it falls below the target threshold

• Prefix routing: what LiteLLM does not do — the real state of KV-aware routers in September 2026, and how much they genuinely improve.

In this same series

• the first optimisation in the series: the grid search of max-num-seqs × max-num-batched-tokens
• FP8 in weights and KV cache: doubling the VRAM available for cache and measuring the quality degradation before going to production
• TP=4×1 vs TP=2×2: the per-session routing that complements the prefix-aware routing in this article

References

• vLLM Automatic Prefix Caching — official documentation
• Prefix-aware routing — Ray Serve
• KV-Cache Wins You Can See — llm-d blog
• The Inference Tax: Prefix-Aware Routing — DigitalOcean
• vLLM issue #24394: Improve Prefix Cache Hit Rate