Semantic cache in RAG: the receptionist with a photographic memory

TL;DR

In a RAG system with real traffic, 30–70% of queries are semantically equivalent to some earlier one even when the text differs. The semantic cache intercepts those queries ahead of the retriever and the LLM, returning the answer already computed if the cosine similarity with a previous query exceeds a threshold θ (typically 0.92–0.95). At 10,000 requests/day and a 45% hit rate, that amounts to not running 4,500 LLM generations: roughly 0.62 GPU hours saved every day on a cluster running Llama-3.1-70B. The fundamental trade-off is that a high θ gives more accurate answers but less saving; a low θ maximises saving but can return incorrect answers for subtly different queries.

The analogy: the receptionist with a notebook

Picture the front desk of a 400-room hotel. Over the course of the day, the receptionist takes hundreds of questions. But if you look through the log book, you will see that 60% of those questions are variants of the same ten:

• “Where is the gym?”
• “What time is breakfast?”
• “Do you have parking?”
• “How do I connect to the WiFi?”

By the third day, the receptionist has built a mental notebook of answers. When someone asks “where can I go to exercise?”, they do not call the concierge (retrieval) or open the hotel’s 300-page internal manual (LLM): they look at the notebook, work out that the question is the same as “where is the gym?”, and answer in two seconds.

But when someone asks “what time does the gym close today?”, the receptionist knows the notebook cannot be trusted: the schedule may have changed because of a private event. They have to call the concierge.

That is exactly the mechanism of a semantic cache:

• The notebook is the cache store (Redis with a vector index, or a Qdrant collection).
• Working out that “exercise” ≈ “gym” is the cosine similarity search with threshold θ.
• Calling the concierge is retrieval over the corpus.
• Consulting the manual is LLM generation.
• “Today” is the temporal query signal that invalidates the cache.

The threshold θ is exactly what separates “where is it” (semantically the same) from “what time is it open today” (semantically different). It is not magic: it is vector arithmetic over learned representations.

The problem in production

A typical RAG pipeline has three layers of latency and compute: embedding the query, vector search over the corpus, and generation with the LLM. In development, that cost is irrelevant. In production with 50 concurrent users, each of those layers scales linearly with the number of requests.

The problem is that users ask the same questions over and over, with slightly different wording:

Running the full pipeline for each of these variants is pure waste. Empirical studies in technical support and corporate Q&A systems report that between 30% and 70% of a day’s queries are semantically redundant with respect to earlier queries from the same week.

The query distribution in real systems follows a power law similar to the Zipf distribution: the 100 most frequent topics account for roughly 60% of total traffic. A well-calibrated cache captures exactly that concentration.

How the semantic cache works

The full flow is shown in the diagram below. Let us describe it in prose first.

When a new query $q$ arrives:

1. Embedding the query: $q$ is embedded with the same model used to index the corpus. This is critical: if the corpus was indexed with text-embedding-3-large and the cache uses a different embedder, the vector spaces are not comparable.

2. Search in the cache store: an ANN (Approximate Nearest Neighbor) search is run over the vectors of previously cached queries. The most similar query $q^*$ and its cosine similarity $s$ are retrieved.

3. Threshold decision:

   $$ \text{answer} = \begin{cases} r^* & \text{if } s(q, q^*) \geq \theta \\ \text{pipeline}(q) & \text{if } s(q, q^*) < \theta \end{cases} $$

   where $r^*$ is the cached answer associated with $q^*$.

4. On a miss: the full pipeline runs (retrieval + LLM). The generated answer is stored in the cache with a configurable TTL for future similar queries.

The cache store is not an ordinary key-value database. It is a vector index over the query embeddings, with the values being the generated answers. Each entry has the structure:

{vector: embed(q), response: r, ttl: T, metadata: {...}}

Flow diagram

The threshold θ and its trade-off

The threshold θ is the most sensitive parameter in the system. It works exactly like the receptionist’s recognition threshold: if it is too demanding, it will only identify textually identical questions and the notebook will not be much use. If it is too lax, it will return the answer to “where is the gym?” to someone who asked “what time does the gym close?”.

The cosine similarity between two vectors $\mathbf{a}$ and $\mathbf{b}$ is:

For prose text (Spanish or English), modern embedders such as text-embedding-3-large or nomic-embed-text assign cosine similarities of around 0.90–0.96 to semantically equivalent paraphrases and similarities of 0.75–0.88 to related but non-equivalent queries.

The cache quality metric is not just hit rate: it is precision@cache, defined as the fraction of cached answers that remain correct for the new query. A cached answer is “correct” if an evaluator (another LLM, or metrics such as BERTScore) judges it equivalent to the one the full pipeline would have generated for that specific query.

The effective saving is defined as $\text{hit rate} \times \text{precision@cache}$, since a hit with an incorrect answer is not a saving: it is an error that can cost more in lost trust than what was saved in GPU.

The empirical sweet spot for most corporate Q&A applications in Spanish or English lies between θ = 0.92 and θ = 0.95. In highly specialised domains where small nuances change the answer (medicine, law, finance), θ ≥ 0.95 is advisable.

The arithmetic of the saving

Let us put concrete numbers on a real system.

Base configuration:

• 10,000 requests/day
• Technical corpus of 1 million chunks in a Qdrant index
• LLM: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)
• Average answer: 200 output tokens
• LLM throughput on this hardware: ~400 tokens/s/GPU with batching (continuous batching enabled, see continuous-batching-fundamentos)

Cost of a request without cache:

Embedding the query takes ~2 ms on a GPU. The ANN vector search over 1 M chunks in Qdrant takes ~5 ms (measured empirically with HNSW, ef=128). Generating 200 tokens at 400 tok/s in total (4 GPUs) comes to:

If 10,000 requests reach the LLM in a day, the total GPU time spent on generation is:

With a semantic cache at θ = 0.93 and a hit rate of ~45%:

Only 5,500 requests (55%) reach the LLM:

Saving:

In inference compute, this amounts roughly to being able to serve 45% more users without adding hardware, or cutting inference costs by 45% if you work with external APIs billed per token.

The cost of the semantic cache itself (embedding the query + ANN search over the cache store) is ~7 ms per request, negligible against the 500 ms of generation avoided on hits.

Zipf distribution of topics:

The reason it works is the Zipf distribution of traffic. If we number topics by frequency (topic 1 = most frequent), the frequency of topic $k$ is proportional to $1/k$. With 1,000 distinct topics:

For the top-100 topics: $\ln(100)/6.9 \approx 4.6/6.9 \approx 67\%$ of traffic. The cache does not need to cover every topic: it captures 67% of traffic while covering only 10% of the topics.

OSS stack 2026

GPTCache

GPTCache is the reference library for a standalone semantic cache. Its architecture is modular:

• Embedder: ONNX Runtime with converted models (onnx/all-MiniLM-L6-v2 by default), with no GPU dependency for the cache layer.
• Vector store: Faiss (local), Milvus, or Qdrant.
• Scalar store: SQLite (development) or Redis (production) for metadata, TTL, and answers.
• Similarity evaluation: cosine by default, configurable.

Minimal configuration in Python:

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

onnx = Onnx()
data_manager = get_data_manager(
    CacheBase("redis", url="redis://localhost:6379"),
    VectorBase("qdrant", host="localhost", collection_name="query_cache")
)
cache.init(
    embedding_func=onnx.to_embeddings,
    data_manager=data_manager,
    similarity_evaluation=SearchDistanceEvaluation(),
)
cache.set_openai_key()

GPTCache intercepts calls to the OpenAI API (or to compatible proxies) transparently. The TTL is configured at the data_manager level.

MeanCache

MeanCache (2024) extends GPTCache to multi-turn conversations. The problem with standard GPTCache is that in dialogue the “relevant” query is not just the last message but the whole context window. MeanCache computes the query embedding as the weighted mean of the embeddings of the last $k$ turns:

where $w_i$ decreases with the age of the turn. This reduces false positives in dialogues where the topic keeps shifting.

Qdrant as a dual cache store

If the RAG corpus is already in Qdrant, you can use the same instance with a separate collection for the cache. The advantages are operational: a single service to manage, the same backup and monitoring infrastructure.

The cache collection uses payload filters to implement TTL:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, Range
import time

client = QdrantClient("localhost", port=6333)

# Search the cache with a TTL filter
hits = client.search(
    collection_name="semantic_cache",
    query_vector=query_embedding,
    query_filter=Filter(
        must=[FieldCondition(
            key="expires_at",
            range=Range(gt=time.time())
        )]
    ),
    limit=1,
    score_threshold=0.93
)

You have to run a periodic cleanup of expired entries, since Qdrant has no native TTL (unlike Redis).

Langfuse for traceability

Langfuse is the OSS standard for LLM pipeline observability (see tracing-llm-otel-genai). Every request should be tagged with whether it was a cache hit or a miss:

from langfuse import Langfuse
from langfuse.decorators import observe

langfuse = Langfuse()

@observe()
def process_query(query: str) -> dict:
    cache_result = semantic_cache.lookup(query)
    if cache_result:
        langfuse.update_current_observation(
            metadata={"cache_hit": True, "cache_score": cache_result.score}
        )
        return cache_result.response
    # full pipeline...
    langfuse.update_current_observation(
        metadata={"cache_hit": False}
    )

With this metadata, Langfuse lets you compute the real hit rate, the distribution of similarity scores, and detect whether the threshold θ needs adjusting.

On-premise hardware: a reference configuration

For an on-premise deployment with this stack, a suitable configuration for RAG with a semantic cache is:

Inference node: 4×H100 SXM (320 GB NVLink in total) for Llama-3.1-70B in FP8. Throughput ~400 tok/s in generation with continuous batching (vLLM or TGI).

Vector services node: CPU with 256 GB RAM. Qdrant for the corpus (1–10 M chunks) and for the cache store (up to 500K entries in memory). Redis 7.x for metadata and an exact-match cache as a first layer.

Embedding node: CPU or a mid-range GPU (A10G). The cache embedder can run on ONNX Runtime on CPU with no perceptible impact on latency (~2 ms per embedding).

Separating the cache store from the corpus matters: the corpus holds millions of chunks with large HNSW indexes; the cache store holds at most tens of thousands of queries with a much smaller index and search times of 1–2 ms.

Cases where the cache fails

The receptionist with a notebook fails in three well-defined scenarios:

1. Queries with a temporal context

“What is the current status of the incident?” or “What changed in the latest release?” are questions whose correct answer changes over time. A cache with a 24-hour TTL could return stale information.

The solution is to detect temporal markers in the query (regular expressions over “today”, “now”, “current”, “latest”, “yesterday”, and their Spanish equivalents) and force a cache miss for these queries, whatever the similarity score.

2. Personalised queries over private data

If the RAG has access to user data (account history, private documents), two different users asking the same question must get different answers. A shared cache that ignores user context is a privacy risk.

The solution is a cache partitioned by user_id or tenant_id. This lowers the hit rate (each user’s cache is smaller) but it is the only safe option in multi-tenant architectures.

3. TTL and a stale corpus

When the corpus is updated (new documents are ingested, errors are corrected), cached answers can go out of date. A fixed TTL (24–48 hours) mitigates the problem but does not remove it.

For corpora with frequent updates, the solution is an active invalidation mechanism: when the corpus is updated in Qdrant, a job runs that identifies which cache entries might be affected (by semantic overlap with the updated chunks) and deletes them. This is the “selective cache invalidation” mentioned in the section on topics not covered.

Integration into the pipeline as middleware

The semantic cache is implemented as middleware between the API gateway and the retriever. It does not change the API contract: the client still sends queries and receives answers in the same format.

class SemanticCacheMiddleware:
    def __init__(self, cache_store, retriever, llm, threshold=0.93):
        self.cache = cache_store
        self.retriever = retriever
        self.llm = llm
        self.threshold = threshold

    async def process(self, query: str, context: dict) -> dict:
        # First layer: exact-match cache (Redis GET, O(1))
        exact = await self.cache.exact_lookup(query)
        if exact:
            return {**exact, "cache_type": "exact"}

        # Second layer: semantic cache (ANN search)
        query_embedding = await self.embed(query)
        semantic = await self.cache.semantic_lookup(
            query_embedding, threshold=self.threshold
        )
        if semantic:
            return {**semantic, "cache_type": "semantic"}

        # Miss: full pipeline
        chunks = await self.retriever.retrieve(query_embedding)
        response = await self.llm.generate(query, chunks)

        # Store for future queries
        await self.cache.store(
            embedding=query_embedding,
            query=query,
            response=response,
            ttl=context.get("ttl", 86400)
        )
        return {**response, "cache_type": "miss"}

The first exact-match layer (Redis GET) is an extra optimisation: for textually identical queries, the embedding is not even computed. The cost is a Redis operation taking microseconds. Only if there is no exact match does it fall through to the semantic lookup.

What we have not covered

• Selective cache invalidation: when a subset of the corpus is updated (for example, the documents for a specific product are reindexed), you would need to identify which cache entries overlap semantically with the updated chunks and mark them stale. The mechanism involves computing similarity between the embeddings of the updated chunks and the embeddings of the cached queries, which is expensive at scale.

• Multi-tenant cache: isolation vs sharing: in a SaaS product with multiple customers, a shared cache maximises the hit rate but can expose one tenant’s answers to another if filtering is not done correctly. A cache partitioned by tenant is safe but has much lower hit rates. The middle ground is a shared cache with ACL filtering applied over Qdrant’s payload filters.

• Semantic cache for streaming responses: when the LLM emits tokens in streaming mode (SSE), the cache cannot easily intercept the complete answer. The options are: cache on the first miss and return the whole answer at once on hits (breaking the streaming experience), or implement a “fake streaming” that emits the tokens of the cached answer at a controlled rate.

• Exact-match cache as a first layer: ahead of the semantic cache, an O(1) lookup in Redis with the query as the key can catch textually identical queries at negligible cost. The code in the previous section already shows this two-layer architecture.

See also

• RAG with reranker and hybrid retrieval — the retrieval that the semantic cache avoids running on hits; understanding how the vector search you are saving works
• PostgreSQL + Qdrant: document ingestion in microservices — the vector store that can double as a cache store, with the same Qdrant instance for corpus and cache
• KV cache in transformers — caching at the transformer’s attention level; different from the semantic cache at the RAG system’s query level, but complementary
• Continuous batching in LLM inference — the batching that processes cache misses; the semantic cache reduces the pressure of requests reaching the inference engine
• Tracing LLMs with OTel and GenAI — how to instrument cache hits vs misses with OpenTelemetry to measure the real saving in production

References

1. Bang, J. et al. (2024). MeanCache: User-Centric Semantic Cache for Large Language Model Based Web Applications. arXiv:2403.02694.
2. Zilliz. (2023). GPTCache: A Library for Creating Semantic Cache for LLM Queries. GitHub: zilliztech/GPTCache.
3. Qdrant Team. (2024). Qdrant Documentation: Filtering with payload. qdrant.tech/documentation.
4. Manning, C. D., Raghavan, P., Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press. Cap. 19: Web search (distribución Zipf).
5. Langfuse. (2024). Observability for LLM Applications. langfuse.com/docs.
6. Meta AI. (2024). Llama 3.1 Model Card. ai.meta.com.
7. Guo, Y. et al. (2023). Evaluating the Factual Consistency of Large Language Models Through Summarization. Referencia para BERTScore como métrica de evaluación de respuestas cacheadas.