KV cache: the working memory that holds up LLM inference
Contents
TL;DR
The KV cache is the working memory a language model keeps during a conversation. Without it, every new token would force a recomputation of the whole conversation from the start, at a cost that is quadratic in the length of the text. With it, the cost is linear, but in exchange the cache lives in VRAM and grows with every token. In practice, it is not the model that limits how much context you can serve: it is the KV cache. On an RTX 4090 running Llama 3 8B, the model fits in 16 GB and barely leaves room for ~64 K tokens of cache in total, adding up all the simultaneous sessions. Understanding that number is the difference between promising a client “128 K of context” and actually delivering it.
You are here: Deploy
This post opens the series on LLM inference fundamentals. Within the six-stage LLMOps pipeline that articulates the whole system, the KV cache lives in the Deploy stage: it is the piece that dictates how much traffic fits in your inference engine and, therefore, how much platform you can offer on top of it.
The analogy: the speaker with amnesia
Imagine you are at a two-hour technical conference. Every time the speaker is about to say a new sentence, he mentally rewinds the whole talk from the beginning, reassembles the thread, and only then carries on. His next sentence requires recalling the previous one; the one after that, the two before it; an hour in, every new word costs him an hour of recap. A conference like that would be materially impossible.
Now imagine the same speaker with a notebook in which he jots down, as he goes, the two or three key ideas of each sentence: subject, object, link to what came before. Before each new sentence he glances at the notebook and continues. His next word only costs a glance at the notebook, not a rewind of the entire talk.
That notebook, in a transformer, is called the KV cache. Without it, conversational language models would be unworkable. With it, they are commercial products. But the notebook weighs something, and understanding how much, where and why is what separates an inference infrastructure that works from one that falls over at the third concurrent client.
The mechanism itself (in plain terms)
A transformer generates text one token at a time. To decide the next token, the model applies a mechanism called attention over all previous tokens: it asks “which parts of the earlier context are relevant for predicting what comes next?”.
Internally, each input token is projected into three vectors:
- Q (Query): “what I am looking for”
- K (Key): “what this token offers”
- V (Value): “what information this token carries”
The attention of the current token against the context is computed by multiplying its Q against the K of all previous tokens, normalising with softmax, and weighting the corresponding V. The result is a contextualised representation of the current token.
Here is the key point: to predict token N, I only need the new Q (that of token N) and the K, V of all previous tokens. The K and V of tokens 1..N-1 have not changed since the previous iteration. Recomputing them would be throwing work away.
The KV cache is exactly that: the memory that holds the K and V of every token already processed, in every layer of the model, so they never have to be recomputed.
Why it exists: the quadratic cost without it
Generating a text of N tokens involves N steps. At step i, attention is computed over i previous tokens. Without a cache, at every step you recompute the K, V of the i-1 previous tokens plus those of the new one. The total count of attention computations grows as:
With a KV cache, you only process the new token at each step: cost linear in N.
The concrete numbers are devastating:
| Tokens generated | Without KV cache (operations) | With KV cache | Ratio |
|---|---|---|---|
| 128 | 8,256 | 128 | 64× |
| 1,024 | 524,800 | 1,024 | 512× |
| 4,096 | 8,390,656 | 4,096 | 2,048× |
| 32,768 | 536,887,296 | 32,768 | 16,384× |
At 32 K tokens, the cache saves you four orders of magnitude of compute. This is not an optimisation: it is what makes conversational inference possible.
The price: how heavy the backpack is
The KV cache is paid for in VRAM. The formula, per sequence, is:
KV_size = 2 · n_layers · n_kv_heads · head_dim · context_len · bytes_per_param
↑
K and V
Per token (dropping context_len), it is a constant of the model itself. Let us look at real numbers:
| Model | n_layers | n_kv_heads | head_dim | Bytes/token (BF16) | GB at 8 K ctx | GB at 32 K | GB at 128 K |
|---|---|---|---|---|---|---|---|
| Llama 3 8B (hypothetical MHA) | 32 | 32 | 128 | 524,288 | 4.00 | 16.00 | 64.00 |
| Llama 3 8B (real GQA) | 32 | 8 | 128 | 131,072 | 1.00 | 4.00 | 16.00 |
| Llama 3 70B (GQA) | 80 | 8 | 128 | 327,680 | 2.50 | 10.00 | 40.00 |
| Qwen3 8B (GQA) | 36 | 8 | 128 | 147,456 | 1.12 | 4.50 | 18.00 |
| Mistral 7B (GQA) | 32 | 8 | 128 | 131,072 | 1.00 | 4.00 | 16.00 |
Two immediate readings:
- Without GQA there is no 128 K worth talking about. A Llama 3 8B with classic multi-head attention would need 64 GB of KV cache alone for a single sequence with 128 K tokens. In other words, it does not fit on any consumer GPU. That is why Meta, Mistral and the rest adopted Grouped Query Attention.
- The KV cache can be larger than the model. Llama 3 8B in BF16 takes ~16 GB. With 128 K of context, its cache is another 16 GB. A single session ties with the model in VRAM.
The dashed red line marks the realistic VRAM available on an RTX 4090 after loading the model. Any model whose curve crosses that line will not be able to serve that context without additional strategies: cache quantisation, offload, partitioning.
Inference is memory-bound, not compute-bound
There is a common misconception: thinking that “fast GPU = fast inference”. In the regime where inference services with a KV cache actually operate, what gets measured is memory bandwidth. Every new token requires reading the K and V of all previous tokens from HBM. The compute is modest; the data movement is massive.
That is why an H100 SXM (3.35 TB/s of HBM3) can be 2-3× faster than an A100 (1.55-2 TB/s) without the clock frequency or the core count fully explaining the difference. Bandwidth explains it.
And that is also why offers of “cheap GPUs with plenty of VRAM but slow HBM” (some variants with GDDR6 or LPDDR5) disappoint in inference with long contexts: they have room to store the cache but take forever to re-read it.
Tricks to make the notebook thinner
Three techniques, in chronological order, have progressively flattened the size of the KV cache:
Multi-Head Attention (MHA). The original transformer design (Vaswani et al., 2017). Every attention head has its own K and V. Expensive in cache but theoretically maximal in expressiveness. It is what models had until around 2023.
Multi-Query Attention (MQA). A single K and V shared by all heads. It reduces the cache by a factor of n_heads. It works reasonably well but degrades generation quality on some benchmarks.
Grouped Query Attention (GQA). The middle ground that won. Heads are grouped: in Llama 3 8B, 32 query heads share K, V in groups of 4, giving 8 KV groups. It cuts the cache 4× relative to MHA with almost identical quality. It has been the de facto standard since 2024.
Multi-Head Latent Attention (MLA). DeepSeek-V2/V3’s innovation: instead of storing K, V per head, it compresses the state into a smaller latent vector and projects out to K, V on the fly. The cache can come down to 70 bytes/token, two orders of magnitude less than GQA. It is the main reason DeepSeek-V3 (671 B parameters, 37 B active) is servable on affordable infrastructure.
Note: the MLA bar is illustrative, using typical values published by DeepSeek; the exact implementation depends on the latent size. What matters is the order of magnitude.
On top of this comes a fourth, orthogonal technique: quantising the cache to FP8, INT8 or even INT4. vLLM and TensorRT-LLM already support it in production. Going from BF16 (2 bytes) to FP8 (1 byte) halves the cache at a small cost in quality. Going to INT4 divides it by four, at a somewhat higher cost.
The next dragon: fragmentation
So far we have talked about the cache as if it were a contiguous block. In practice, an inference server handles dozens of simultaneous sessions, each with its own cache growing at a different rate. Naive allocation, reserving the maximum possible per session, wastes between 60 % and 80 % of the VRAM according to the original PagedAttention paper.
PagedAttention, the idea from Kwon et al. (2023) that gave rise to vLLM, solves this by borrowing a technique from operating systems: divide the VRAM into small blocks (typically 16 tokens) and keep a page table mapping logical to physical blocks per session. A session no longer reserves one huge contiguous block: it grows one block at a time, and the blocks can be scattered across the VRAM. The result is an effective occupancy of 90 % instead of 30 %, and therefore 2-4× more aggregate throughput on the same hardware.
PagedAttention deserves an article of its own. I am noting it down for the next one.
Applied to generic on-premise hardware
Let us come down to concrete cases.
Case 1 — RTX 4090 (24 GB, Ada Lovelace)
A typical configuration with Qwen3-8B BF16:
Model in BF16: ~16 GB
Activations + overhead: ~2 GB
VRAM available for KV cache: ~6 GB (with margin)
At 144 KB/token (Qwen3-8B GQA), that comes to ~43 K total tokens of cache distributed across all simultaneous sessions. In practice:
| Concurrency | Maximum context per session |
|---|---|
| 1 | 32,768 |
| 4 | 8,192 |
| 8 | 4,096 |
| 16 | 2,048 |
If you need to advertise “we support 32 K of context” with concurrency of 4 or more, you have to quantise the cache (FP8 brings it down to 72 KB/token, doubling capacity) or move up the model range (a 4B with GQA and a quantised cache would have room to spare).
Case 2 — Cluster of 4×H100 (320 GB total, NVLink)
With tensor parallel = 4 and Llama 3 70B BF16:
Model in BF16: ~140 GB (35 GB/GPU)
vLLM overhead per GPU: ~2 GB
VRAM free for KV per GPU: ~43 GB → ~172 GB aggregate
At 320 KB/token (Llama 3 70B GQA), that comes to ~537 K total tokens of cache. Ample margin for long contexts with high concurrency:
| Concurrency | Maximum context per session |
|---|---|
| 4 | 134,000 |
| 16 | 33,500 |
| 64 | 8,375 |
For DeepSeek-V3 671 B with MLA the economics change radically, because the cache is ~100× thinner. What limits you is no longer the cache but the VRAM of the model itself (quantised to FP8 it is ~335 GB, so it fits on 4×H100 with room for the KV cache).
Operational implications
Three observations we repeat in every consulting engagement:
First, the maximum context a model advertises is not the one you can serve on your hardware. Llama 3 8B “supports” 128 K, but on a 4090 with 4 simultaneous sessions your effective context is ~8 K. It is trivial to check before promising it to a client.
Second, quantising the KV cache is one of the optimisations with the best cost/benefit ratio in the ENS context. It does not touch the weights, it does not affect audit reproducibility, and it doubles capacity. vLLM supports it via --kv-cache-dtype fp8.
Third, if the SLAs dictate long contexts with many concurrent users, GQA is necessary but not sufficient. In the medium term you need to look at models with MLA or attention variants with compression.
What we have not covered (upcoming articles)
- PagedAttention and its implementation in vLLM: blocks, page table, eviction.
- Prefix caching: when several requests share the system prompt, there is no need to recompute the K, V of the common part.
- Speculative decoding and its interaction with the cache.
- Cache offloading: moving cold blocks to RAM or NVMe, a key technique for contexts above 1 M.
See also
- The six-stage LLMOps pipeline — the master map of the system in production, in which the Deploy stage is one box out of six. This post goes into one of the critical decisions inside Deploy.
- Inside PagedAttention: blocks, page table, eviction and the state of the art of the KV cache in 2026 — a theoretical deep dive at block level plus a survey of derived optimisations (vAttention, EvicPress, RadixAttention, speculative decoding). It continues this post from the academic theory.
- Continuous fine-tuning in production: from real traffic to the deployed adapter — how the loop between inference and incremental training closes on the same stack (vLLM + Postgres), with VRAM budgets that explicitly include the KV cache during eval.
- Disaggregated serving: prefill and decode in specialised pods — the KV cache stops being a private GPU buffer and becomes the artefact transferred between pods. Here the cache size formula determines the economics of the transfer.
- The GPU cluster as a multi-tenant platform — how the cluster becomes a service with tenants, gateway, quotas and isolation. That is where the KV cache stops being only a performance resource and becomes a platform matter.
- vLLM on Kubernetes: the LLM inference piece that does scale — the engine that materialises everything discussed here, deployed on K8s with tensor parallel and autoscaling.
- Quantization for LLM inference: FP8, INT4 and GGUF — KV cache quantisation (
--kv-cache-dtype=fp8/int4), mentioned here as the fourth orthogonal technique, is taken apart there with the maths, the formats and the measurable loss. - Speculative decoding: the secretary who anticipates what the boss will say — the memory-bound decode regime caused by the KV cache is exactly what speculative decoding exploits: a forward pass with γ tokens costs almost the same as one with a single token.
- FlashAttention v1/v2/v3/v4: the librarian who never clears the desk — the kernel that sweeps the KV cache against Q on every iteration without materialising the N×N matrix. The compute layer underneath the cache.
- MoE inference: the call centre with 256 specialists — attention is still dense in every MoE of 2026 (Mixtral, DeepSeek, Qwen3, Llama 4, Kimi K2), so the KV cache keeps its shape; DeepSeek’s MLA is the orthogonal optimisation that compresses it ~10× to make long context viable on modest clusters.
- Continuous batching — the iterative scheduler manages the dynamic allocation of the KV cache across requests; without PagedAttention, continuous batching would fragment the HBM, and without continuous batching the KV cache would be underused because of padding.
- Capacity planning for on-premise LLM inference — the KV cache is the dominant component of the VRAM budget when sizing a cluster from an SLO; the spreadsheet is built there step by step.
- Optimising prefill in vLLM — the four concrete knobs (chunked prefill, prefix caching, FP8 KV, max-model-len) that turn KV cache theory into production parameters for the RTX 4090 and the L40.
- Optimising decode in vLLM — how
--gpu-memory-utilization, speculative decoding and an FP8 KV cache combine to squeeze small hardware during the generation phase.
References
- Vaswani et al., Attention Is All You Need (NeurIPS 2017) — foundational transformer paper.
- Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023).
- Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM paper.
- DeepSeek-AI, DeepSeek-V2 Technical Report (2024) — introduction of Multi-Head Latent Attention.
- Official vLLM documentation: https://docs.vllm.ai/.
- Llama 3 model card (Meta): GQA specs, n_layers, n_kv_heads.