KV cache: the working memory that holds up LLM inference

Mon, 18 May 2026 10:00:00 +0200

TL;DR

The KV cache is the working memory a language model keeps during a conversation. Without it, every new token would force a recomputation of the whole conversation from the start, at a cost that is quadratic in the length of the text. With it, the cost is linear, but in exchange the cache lives in VRAM and grows with every token. In practice, it is not the model that limits how much context you can serve: it is the KV cache. On an RTX 4090 running Llama 3 8B, the model fits in 16 GB and barely leaves room for ~64 K tokens of cache in total, adding up all the simultaneous sessions. Understanding that number is the difference between promising a client “128 K of context” and actually delivering it.

You are here: Deploy

This post opens the series on LLM inference fundamentals. Within the six-stage LLMOps pipeline that articulates the whole system, the KV cache lives in the Deploy stage: it is the piece that dictates how much traffic fits in your inference engine and, therefore, how much platform you can offer on top of it.

The analogy: the speaker with amnesia

Imagine you are at a two-hour technical conference. Every time the speaker is about to say a new sentence, he mentally rewinds the whole talk from the beginning, reassembles the thread, and only then carries on. His next sentence requires recalling the previous one; the one after that, the two before it; an hour in, every new word costs him an hour of recap. A conference like that would be materially impossible.

Now imagine the same speaker with a notebook in which he jots down, as he goes, the two or three key ideas of each sentence: subject, object, link to what came before. Before each new sentence he glances at the notebook and continues. His next word only costs a glance at the notebook, not a rewind of the entire talk.

That notebook, in a transformer, is called the KV cache. Without it, conversational language models would be unworkable. With it, they are commercial products. But the notebook weighs something, and understanding how much, where and why is what separates an inference infrastructure that works from one that falls over at the third concurrent client.

The mechanism itself (in plain terms)

A transformer generates text one token at a time. To decide the next token, the model applies a mechanism called attention over all previous tokens: it asks “which parts of the earlier context are relevant for predicting what comes next?”.

Internally, each input token is projected into three vectors:

Q (Query): “what I am looking for”
K (Key): “what this token offers”
V (Value): “what information this token carries”

The attention of the current token against the context is computed by multiplying its Q against the K of all previous tokens, normalising with softmax, and weighting the corresponding V. The result is a contextualised representation of the current token.

Q·Kᵀ → softmax × V

representation of token N

Here is the key point: to predict token N, I only need the new Q (that of token N) and the K, V of all previous tokens. The K and V of tokens 1..N-1 have not changed since the previous iteration. Recomputing them would be throwing work away.

The KV cache is exactly that: the memory that holds the K and V of every token already processed, in every layer of the model, so they never have to be recomputed.

Why it exists: the quadratic cost without it

Generating a text of N tokens involves N steps. At step i, attention is computed over i previous tokens. Without a cache, at every step you recompute the K, V of the i-1 previous tokens plus those of the new one. The total count of attention computations grows as:

$$\sum_{i=1}^{N} i = \frac{N(N+1)}{2} \approx \frac{N^2}{2}$$

With a KV cache, you only process the new token at each step: cost linear in N.

Cumulative compute to generate N tokens (schematic scale — exact data is in the table)

tokens generated (N) attention operations

0 1K 2K 3K 4K

with KV cache O(N) — linear

without KV cache O(N²) — quadratic

The concrete numbers are devastating:

Tokens generated	Without KV cache (operations)	With KV cache	Ratio
128	8,256	128	64×
1,024	524,800	1,024	512×
4,096	8,390,656	4,096	2,048×
32,768	536,887,296	32,768	16,384×

At 32 K tokens, the cache saves you four orders of magnitude of compute. This is not an optimisation: it is what makes conversational inference possible.

The price: how heavy the backpack is

The KV cache is paid for in VRAM. The formula, per sequence, is:

KV_size = 2 · n_layers · n_kv_heads · head_dim · context_len · bytes_per_param
↑
K and V

Per token (dropping context_len), it is a constant of the model itself. Let us look at real numbers:

Model	n_layers	n_kv_heads	head_dim	Bytes/token (BF16)	GB at 8 K ctx	GB at 32 K	GB at 128 K
Llama 3 8B (hypothetical MHA)	32	32	128	524,288	4.00	16.00	64.00
Llama 3 8B (real GQA)	32	8	128	131,072	1.00	4.00	16.00
Llama 3 70B (GQA)	80	8	128	327,680	2.50	10.00	40.00
Qwen3 8B (GQA)	36	8	128	147,456	1.12	4.50	18.00
Mistral 7B (GQA)	32	8	128	131,072	1.00	4.00	16.00

Two immediate readings:

Without GQA there is no 128 K worth talking about. A Llama 3 8B with classic multi-head attention would need 64 GB of KV cache alone for a single sequence with 128 K tokens. In other words, it does not fit on any consumer GPU. That is why Meta, Mistral and the rest adopted Grouped Query Attention.
The KV cache can be larger than the model. Llama 3 8B in BF16 takes ~16 GB. With 128 K of context, its cache is another 16 GB. A single session ties with the model in VRAM.

0 10 20 30 40 GB

0 8K 32K 64K 128K

≈ VRAM free after loading an 8B on a 4090

Llama 3 8B Qwen3 8B Llama 3 70B

The dashed red line marks the realistic VRAM available on an RTX 4090 after loading the model. Any model whose curve crosses that line will not be able to serve that context without additional strategies: cache quantisation, offload, partitioning.

Inference is memory-bound, not compute-bound

There is a common misconception: thinking that “fast GPU = fast inference”. In the regime where inference services with a KV cache actually operate, what gets measured is memory bandwidth. Every new token requires reading the K and V of all previous tokens from HBM. The compute is modest; the data movement is massive.

That is why an H100 SXM (3.35 TB/s of HBM3) can be 2-3× faster than an A100 (1.55-2 TB/s) without the clock frequency or the core count fully explaining the difference. Bandwidth explains it.

And that is also why offers of “cheap GPUs with plenty of VRAM but slow HBM” (some variants with GDDR6 or LPDDR5) disappoint in inference with long contexts: they have room to store the cache but take forever to re-read it.

Tricks to make the notebook thinner

Three techniques, in chronological order, have progressively flattened the size of the KV cache:

Multi-Head Attention (MHA). The original transformer design (Vaswani et al., 2017). Every attention head has its own K and V. Expensive in cache but theoretically maximal in expressiveness. It is what models had until around 2023.

Multi-Query Attention (MQA). A single K and V shared by all heads. It reduces the cache by a factor of n_heads. It works reasonably well but degrades generation quality on some benchmarks.

Grouped Query Attention (GQA). The middle ground that won. Heads are grouped: in Llama 3 8B, 32 query heads share K, V in groups of 4, giving 8 KV groups. It cuts the cache 4× relative to MHA with almost identical quality. It has been the de facto standard since 2024.

Multi-Head Latent Attention (MLA). DeepSeek-V2/V3’s innovation: instead of storing K, V per head, it compresses the state into a smaller latent vector and projects out to K, V on the fly. The cache can come down to 70 bytes/token, two orders of magnitude less than GQA. It is the main reason DeepSeek-V3 (671 B parameters, 37 B active) is servable on affordable infrastructure.

Note: the MLA bar is illustrative, using typical values published by DeepSeek; the exact implementation depends on the latent size. What matters is the order of magnitude.

On top of this comes a fourth, orthogonal technique: quantising the cache to FP8, INT8 or even INT4. vLLM and TensorRT-LLM already support it in production. Going from BF16 (2 bytes) to FP8 (1 byte) halves the cache at a small cost in quality. Going to INT4 divides it by four, at a somewhat higher cost.

The next dragon: fragmentation

So far we have talked about the cache as if it were a contiguous block. In practice, an inference server handles dozens of simultaneous sessions, each with its own cache growing at a different rate. Naive allocation, reserving the maximum possible per session, wastes between 60 % and 80 % of the VRAM according to the original PagedAttention paper.

session A

session B

session C

session D

→ ~70 % of VRAM reserved and empty

 <rect x="0" y="22" width="30" height="20" class="used blk"/>
<rect x="30" y="22" width="30" height="20" class="used blk"/>
<rect x="60" y="22" width="30" height="20" class="used blk"/>
<rect x="90" y="22" width="30" height="20" class="used blk"/>
<rect x="120" y="22" width="30" height="20" class="used blk"/>
<rect x="150" y="22" width="30" height="20" class="used blk"/>
<rect x="180" y="22" width="30" height="20" class="free blk"/>
<rect x="210" y="22" width="30" height="20" class="free blk"/>
<rect x="0" y="44" width="30" height="20" class="used blk"/>
<rect x="30" y="44" width="30" height="20" class="used blk"/>
<rect x="60" y="44" width="30" height="20" class="used blk"/>
<rect x="90" y="44" width="30" height="20" class="free blk"/>
<rect x="120" y="44" width="30" height="20" class="free blk"/>
<rect x="150" y="44" width="30" height="20" class="free blk"/>
<rect x="180" y="44" width="30" height="20" class="free blk"/>
<rect x="210" y="44" width="30" height="20" class="free blk"/>
</g>

→ < 4 % waste (vLLM paper)

PagedAttention, the idea from Kwon et al. (2023) that gave rise to vLLM, solves this by borrowing a technique from operating systems: divide the VRAM into small blocks (typically 16 tokens) and keep a page table mapping logical to physical blocks per session. A session no longer reserves one huge contiguous block: it grows one block at a time, and the blocks can be scattered across the VRAM. The result is an effective occupancy of 90 % instead of 30 %, and therefore 2-4× more aggregate throughput on the same hardware.

PagedAttention deserves an article of its own. I am noting it down for the next one.

Applied to generic on-premise hardware

Let us come down to concrete cases.

Case 1 — RTX 4090 (24 GB, Ada Lovelace)

A typical configuration with Qwen3-8B BF16:

Model in BF16: ~16 GB
Activations + overhead: ~2 GB
VRAM available for KV cache: ~6 GB (with margin)

At 144 KB/token (Qwen3-8B GQA), that comes to ~43 K total tokens of cache distributed across all simultaneous sessions. In practice:

Concurrency	Maximum context per session
1	32,768
4	8,192
8	4,096
16	2,048

If you need to advertise “we support 32 K of context” with concurrency of 4 or more, you have to quantise the cache (FP8 brings it down to 72 KB/token, doubling capacity) or move up the model range (a 4B with GQA and a quantised cache would have room to spare).

Case 2 — Cluster of 4×H100 (320 GB total, NVLink)

With tensor parallel = 4 and Llama 3 70B BF16:

Model in BF16: ~140 GB (35 GB/GPU)
vLLM overhead per GPU: ~2 GB
VRAM free for KV per GPU: ~43 GB → ~172 GB aggregate

At 320 KB/token (Llama 3 70B GQA), that comes to ~537 K total tokens of cache. Ample margin for long contexts with high concurrency:

Concurrency	Maximum context per session
4	134,000
16	33,500
64	8,375

For DeepSeek-V3 671 B with MLA the economics change radically, because the cache is ~100× thinner. What limits you is no longer the cache but the VRAM of the model itself (quantised to FP8 it is ~335 GB, so it fits on 4×H100 with room for the KV cache).

Operational implications

Three observations we repeat in every consulting engagement:

First, the maximum context a model advertises is not the one you can serve on your hardware. Llama 3 8B “supports” 128 K, but on a 4090 with 4 simultaneous sessions your effective context is ~8 K. It is trivial to check before promising it to a client.

Second, quantising the KV cache is one of the optimisations with the best cost/benefit ratio in the ENS context. It does not touch the weights, it does not affect audit reproducibility, and it doubles capacity. vLLM supports it via --kv-cache-dtype fp8.

Third, if the SLAs dictate long contexts with many concurrent users, GQA is necessary but not sufficient. In the medium term you need to look at models with MLA or attention variants with compression.

What we have not covered (upcoming articles)

PagedAttention and its implementation in vLLM: blocks, page table, eviction.
Prefix caching: when several requests share the system prompt, there is no need to recompute the K, V of the common part.
Speculative decoding and its interaction with the cache.
Cache offloading: moving cold blocks to RAM or NVMe, a key technique for contexts above 1 M.

References

Vaswani et al., Attention Is All You Need (NeurIPS 2017) — foundational transformer paper.
Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM paper.
DeepSeek-AI, DeepSeek-V2 Technical Report (2024) — introduction of Multi-Head Latent Attention.
Official vLLM documentation: https://docs.vllm.ai/.
Llama 3 model card (Meta): GQA specs, n_layers, n_kv_heads.

Transformers on lo0 — Tech Blog