Paged-Attention on lo0 — Tech Blog

vLLM on Kubernetes: the LLM inference piece that actually scales

Mon, 18 May 2026 13:00:00 +0200

TL;DR

vLLM is the inference engine that turns a general-purpose GPU into a productive LLM server. Its value is not in running a model, since any transformers.pipeline does that in three lines of Python, but in squeezing the GPU down to the last gigabyte and the last cycle: PagedAttention for the KV cache, continuous batching to mix requests, its own scheduler to share GPU time between sessions. Kubernetes is its natural habitat because vLLM behaves like a modern UNIX process, with a health endpoint, Prometheus metrics, orderly draining and declarable resources, and K8s already knows how to manage those. But there are traps: the standard HPA does not scale vLLM well, the model takes minutes to load, and naive rolling updates cut sessions halfway through decoding. This article takes the engine apart and then fits it, with real manifests, into a cluster that can actually serve it.

This article is the natural continuation of KV cache: the working memory that holds up LLM inference. There we explained why every token consumes VRAM. Here we look at what you do with that VRAM when you want to offer it as a service.

The analogy: a multiprocess kernel for your GPU

Imagine you have a single processor and need to serve a hundred concurrent processes without any of them blocking the rest. Nobody in their right mind would write a while-true loop dispatching processes one by one: they would install an operating system. The kernel takes care of scheduling, memory paging, isolation, priorities and cleanup on exit. The “process” becomes a convenient abstraction and the kernel does the dirty work.

vLLM is, for your GPU, what the kernel is for your CPU. As far as the GPU is concerned, a conversation with an LLM is a process that lives across many decoding steps, occupies a slice of VRAM (its KV cache) and demands compute time every time a token has to be generated. You have a hundred of those processes at once. You need to:

Share GPU time between them without pausing any of them entirely (it would be disastrous if one long conversation monopolised the GPU).
Manage memory with paging because, just as in RAM, contiguous allocation is inefficient.
Queue new requests when the GPU is saturated and serve them in a reasonable order.
Reclaim resources when a session ends.

PagedAttention is the virtual memory of the KV cache. Continuous batching is the time-slicing scheduler that shares the GPU token by token. The OpenAI-compatible server is the uniform syscall interface. Calling it a “kernel” for the GPU is marketing, but it is marketing that captures the idea well.

What vLLM does inside

Continuous batching: stop waiting for the slowest

The naive inference engine does static batching: it groups N requests, processes them until all of them are finished, returns and starts another round. The problem is obvious: if one request asks for 8 tokens and another asks for 800, the other seven wait for the slow one. GPU utilisation collapses.

Continuous batching (Yu et al., 2022, popularised by vLLM) changes the model. At every decode step, which produces one token for each active session, the engine composes the batch out of the active tokens of ALL sessions alive at that instant. When a session finishes its generation it releases its slot immediately and another request from the queue takes it. The batch never waits for the slowest session because nobody is blocked: everyone advances at one token per step.

The original paper measured 5–23× more throughput than the equivalent static batching. The exact number depends on how variable the response lengths are, but the order of magnitude holds up in practice.

The consequence for the operator is counterintuitive: a single vLLM replica performs like three naive replicas. There is no point adding pods without justifying it with real metrics.

PagedAttention: virtual memory for the KV cache

We already flagged it in the KV cache article: the naive engine reserves one contiguous block per session, sized for the worst case (max_context_len), and wastes 60–80% of the VRAM because real sessions come nowhere near their ceiling.

PagedAttention borrows the solution operating systems have been using for half a century: split the VRAM into small blocks (16 tokens in the default implementation) and keep a logical → physical page table per session. A session with 273 tokens of context occupies 18 blocks (not necessarily contiguous), and grows block by block as it generates. The paper measured <4% waste, an order of magnitude better than contiguous allocation, and that translates into 2–4× more aggregate throughput on the same hardware, because more sessions fit at once.

There is a cost: every attention operation has to be indirected through the page table. But vLLM’s CUDA kernels are written so that this indirection is cheap, and the net result is massively positive.

Prefill vs decode: two phases with opposite profiles

An LLM request has two phases with radically different GPU profiles:

Prefill: processes the entire prompt in one go. It is compute-bound: it uses the tensor cores intensively, the GPU sits above 90%, and it lasts between hundreds of ms and a few seconds depending on prompt size.
Decode: generates token by token. It is memory-bound: the compute is modest but the whole KV cache has to be read for every token, and it lasts from tens of ms per token up to minutes for long responses.

A naive server treats each request as a unit and serves the two phases in series. vLLM decouples them: it mixes requests in prefill with requests in decode in the same step (a technique called chunked prefill when it also splits long prefills into chunks). The result: the GPU is always busy doing something, the tensor cores with prefills, the HBM bandwidth with decodes, instead of oscillating between phases.

Operational implication: the “% GPU utilisation” metric from nvidia-smi is misleading. A GPU at 100% doing prefills may have its HBM bandwidth idle. A GPU at 40% doing decodes may have its HBM saturated. For LLM serving, the useful metric is effective HBM bandwidth, not the compute percentage.

Tensor parallel: when the model does not fit on one GPU

Llama 3 70B in BF16 is ~140 GB. There is no single GPU on the market that can hold it. The solution is tensor parallel: split each model layer by columns and run the partitions on N GPUs in parallel, synchronising with an all-reduce after each layer.

For N=5 GPUs and a 70B model, each GPU sees roughly 28 GB of weights. That sounds fine until you remember that the all-reduce on every layer means reading and writing large tensors between GPUs. If the GPUs share NVLink/NVSwitch (300–900 GB/s), the all-reduce is cheap. If they only share PCIe (~32 GB/s gen4 x16), the all-reduce eats half the time and throughput collapses.

The implication for K8s, which comes next: the scheduler has to guarantee that the N GPUs are physically close. That translates into NodeAffinity to the right product (NVIDIA-H100-80GB-HBM3), a single pod with nvidia.com/gpu: N (not N pods sharing) and, if multi-node is unavoidable, InfiniBand with NCCL as transport.

The OpenAI-compatible server

On top of all of the above, vLLM exposes an HTTP server with endpoints identical to OpenAI’s: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models. It supports Server-Sent Events streaming. It supports tool calling. It supports logprobs.

The value of this is enormous and underrated: any client using the OpenAI SDK works without changes. Your application points at https://vllm.your-cluster.local/v1 instead of https://api.openai.com/v1, and everything else, the LangChain, LlamaIndex, OpenAI Python and OpenAI JS SDKs, just works. It is the main reason vLLM has gained traction over technically comparable alternatives: it is the boring option that works.

Why Kubernetes is the natural habitat

vLLM is a well-behaved process: it starts, exposes metrics, serves a health endpoint, takes SIGTERM with dignity and declares the resources it needs. Kubernetes has spent ten years perfecting the management of processes like that. The only thing K8s took a while to absorb properly was the GPU, and that is now solved.

The GPU as a primitive resource

The plumbing goes like this:

The node has the NVIDIA driver installed (or the GPU Operator installs it).
A DaemonSet, nvidia-device-plugin, registers the physical GPUs as nvidia.com/gpu resources with kubelet.
The Kubernetes scheduler sees those resources the way it sees CPU and memory, puts them in its accounting and assigns them to Pods that ask for them.
The nvidia-container-toolkit makes sure containerd injects the right devices into the container at startup.

For the pod, asking for a GPU looks like this:

resources:
 requests:
 nvidia.com/gpu: 1
 limits:
 nvidia.com/gpu: 1

Without MIG, MPS or time-slicing configured, a GPU is not shared between pods: you ask for it whole or you do not ask for it. For vLLM, which wants the entire GPU to itself, that is exactly what you want.

The vLLM Pod lifecycle

Differences from a typical webapp Pod:

Long startup. Loading 16 GB of weights into VRAM over the network takes 30 seconds in the best case and 5 minutes in the worst. A readinessProbe with initialDelaySeconds: 30 and failureThreshold: 3 kills the pod before it starts. Solution: a startupProbe with a high threshold before the livenessProbe starts evaluating.
Warm-up matters. The first prefill compiles CUDA kernels specific to the input shape. The first 2–3 requests are noticeably slower. If latency matters from second one, it is worth firing a warm-up POST after ready.
Draining is not instantaneous. SIGTERM must not kill in-flight sessions. vLLM, configured with --disable-graceful-shutdown false (the default), finishes active requests before closing. That can take 30–180 seconds. terminationGracePeriodSeconds has to accommodate it.
Hostile rollouts. A naive rolling update (maxUnavailable: 1) can leave you with no replicas serving if the new one is slow to load. Set maxSurge: 1, maxUnavailable: 0 so the new pod is Ready before the old one is drained.

Anatomy of a serious deployment

First of all: GPU Operator

Without the GPU Operator (or an equivalent manual installation), a Pod with nvidia.com/gpu: 1 stays Pending forever. What the operator installs as DaemonSets on every GPU node:

nvidia-driver-daemonset — the kernel-mode driver (if you do not have it installed at host level).
nvidia-device-plugin-daemonset — registers the GPUs as a kubelet resource.
nvidia-container-toolkit-daemonset — the containerd integration.
nvidia-dcgm-exporter — Prometheus metrics for the GPU (utilisation, temperature, ECC errors, memory).
gpu-feature-discovery — node labels: nvidia.com/gpu.product, nvidia.com/gpu.memory, and so on, essential for NodeAffinity.

The recommended installation is the official Helm chart. The delicate part is aligning the driver with the host kernel version: if the nodes run kernel 6.x, the operator needs a compatible driver branch.

A complete, annotated vLLM Deployment

The following deploys Llama 3 8B with an FP8-quantised KV cache, up to 32K of context, on an RTX 4090. It is the reference manifest; the comments explain the non-obvious decisions.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 replicas: 1
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 0 # never run out of replicas during the rollout
 selector:
 matchLabels:
 app: vllm-llama3-8b
 template:
 metadata:
 labels:
 app: vllm-llama3-8b
 annotations:
 prometheus.io/scrape: "true"
 prometheus.io/port: "8000"
 prometheus.io/path: "/metrics"
 spec:
 # Only nodes with the GPU we expect
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-GeForce-RTX-4090
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 # Pre-download the weights if they are not on the shared PVC
 initContainers:
 - name: model-download
 image: ghcr.io/huggingface/huggingface-cli:latest
 command: ["sh", "-c"]
 args:
 - |
 if [ ! -f /models/llama-3-8b/config.json ]; then
 huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct \
 --local-dir /models/llama-3-8b --local-dir-use-symlinks False
 fi
 env:
 - name: HF_TOKEN
 valueFrom:
 secretKeyRef:
 name: huggingface
 key: token
 volumeMounts:
 - name: models
 mountPath: /models
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-8b
 - --served-model-name=llama-3-8b
 - --tensor-parallel-size=1
 - --max-model-len=32768
 - --kv-cache-dtype=fp8
 - --enable-chunked-prefill
 - --enable-prefix-caching
 - --gpu-memory-utilization=0.92
 - --port=8000
 ports:
 - name: http
 containerPort: 8000
 - name: metrics
 containerPort: 8000 # same port as http; /metrics
 resources:
 requests:
 cpu: "4"
 memory: 8Gi
 nvidia.com/gpu: 1
 limits:
 cpu: "8"
 memory: 16Gi
 nvidia.com/gpu: 1
 startupProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 10
 failureThreshold: 60 # 10 min of grace to load the model
 readinessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 5
 livenessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 20
 failureThreshold: 3
 volumeMounts:
 - name: models
 mountPath: /models
 readOnly: true # no process should write here at runtime
 - name: shm
 mountPath: /dev/shm  # vLLM uses shared memory for IPC between workers
 volumes:
 - name: models
 persistentVolumeClaim:
 claimName: model-cache
 - name: shm
 emptyDir:
 medium: Memory
 sizeLimit: 4Gi
 terminationGracePeriodSeconds: 120 # accommodates draining of active sessions
---
apiVersion: v1
kind: Service
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 selector:
 app: vllm-llama3-8b
 ports:
 - name: http
 port: 80
 targetPort: 8000

Five things that are not visible on a first read:

/dev/shm in memory, 4 GB. vLLM launches worker processes (one per GPU in tensor parallel, plus the driver) that communicate over shared memory. Docker’s default (64 MB) blows up as soon as the model is mid-sized. Without this, the pod starts but fails the moment it serves the first complex request.
--enable-prefix-caching. If the prompts in your workload share structure (a common system prompt, few-shot examples), vLLM reuses the KV cache of the shared part. A free 30–60% gain in TTFT.
--gpu-memory-utilization=0.92. vLLM reserves the given percentage of the VRAM for itself. The remaining 8% leaves room for activations, CUDA kernels and the overhead that does not get counted. Lowering it buys safety; raising it above 0.95 invites OOM.
A ReadOnlyMany PVC is ideal. The model does not change at runtime. Several pods can mount the same PVC without contention.
No livenessProbe shorter than the terminationGracePeriodSeconds. If a drain takes 90s and liveness kills at 60s, rollouts lose sessions.

Multi-pod tensor parallel: LeaderWorkerSet

When the model needs more GPUs than a single node has, the pattern is a group of coordinated pods, one per GPU, that behave as a single replica. For years this was modelled with a StatefulSet plus init scripts; since Kubernetes 1.32, the idiomatic primitive is LeaderWorkerSet (LWS):

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
 name: vllm-llama3-70b
 namespace: inference
spec:
 replicas: 1
 leaderWorkerTemplate:
 size: 5 # 1 leader + 4 workers = 5 pods, 5 GPUs
 restartPolicy: RecreateGroupOnPodRestart
 leaderTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-leader
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-70b
 - --tensor-parallel-size=5
 - --distributed-executor-backend=ray
 # ...
 workerTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-worker
 image: vllm/vllm-openai:v0.6.3
 # the workers join the leader's Ray cluster

LWS guarantees startup order (workers first, leader afterwards) and an atomic lifecycle (if a worker goes down, the whole group restarts, not a single pod). Without it, the coordination is manually fragile.

A simpler alternative, if all the tensor parallel GPUs fit on a single node (the case with HGX H100 boxes with 8 GPUs and an internal NVSwitch): one single Pod with nvidia.com/gpu: 5, --tensor-parallel-size=5, and vLLM handles everything internally. No Ray, no LWS, far simpler. It is the recommended path whenever it is available.

Autoscaling: the standard HPA is no use

An HPA on CPU% is useless for vLLM. The GPU does the work; the pod’s CPU sits at 5–10% even under maximum load. Nor is the GPU utilisation percentage from dcgm-exporter any use: a pod at 100% GPU% with gpu_cache_usage_perc=15% is serving one long session without being saturated, while a pod at 60% GPU% with gpu_cache_usage_perc=95% is on the edge of evicting sessions.

The right metrics are exported by vLLM itself at /metrics (Prometheus format):

Metric	What it says	When to scale
`vllm:num_requests_waiting`	Requests queued and not yet in the batch.	If it stays above 5–10.
`vllm:num_requests_running`	Active requests in the batch.	For capacity planning, not for scaling.
`vllm:gpu_cache_usage_perc`	% of the KV cache occupied.	If sustained above 80%, there is a risk of preemption.
`vllm:time_to_first_token_seconds`	Prefill latency (histogram).	If p95 exceeds your SLA.
`vllm:e2e_request_latency_seconds`	Total latency per request.	An output metric.

For the HPA to consume them there are two routes: Prometheus Adapter (exposes custom metrics to the K8s API) or KEDA (scales on Prometheus queries directly, far more convenient). With KEDA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama3-8b
 minReplicaCount: 1
 maxReplicaCount: 8
 pollingInterval: 10
 cooldownPeriod: 120 # 2 min before scale-down (long sessions)
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.monitoring:9090
 threshold: '5'
 query: |
 sum(vllm:num_requests_waiting{app="vllm-llama3-8b"})

The long cooldownPeriod matters: if you drop replicas while there are sessions decoding, you kill them. Better to leave 2 minutes of slack.

Observability: the four metrics that matter

Of everything /metrics exports, a minimal dashboard needs these four:

TTFT p50/p95 (time to first token) — what the user perceives when they hit send.
TPOT p50/p95 (time per output token) — the “speed” of the streaming.
Aggregate throughput (tokens generated per second across the cluster) — for capacity planning.
Queue depth (vllm:num_requests_waiting) — the leading indicator: if it grows, everything is about to degrade.

To that, add HBM utilisation and free memory per GPU (from dcgm-exporter) to spot bandwidth saturation and fragmentation problems. A decent Grafana dashboard with those 6 charts anticipates 90% of incidents.

Two concrete scenarios

We reuse the same hardware as the previous article for continuity. The same cache arithmetic, now with the engine on top.

Scenario A — 1×RTX 4090 (workstation or small K8s node)

Topology: 1 Pod, --tensor-parallel-size=1, 1 GPU, 1 node.
Model: up to 8B BF16 (Llama 3 8B, Qwen3 8B, Mistral 7B) or up to 14B in FP8/AWQ.
PVC: the node’s local SSD. The 4090 reads 1 TB/s from HBM; an NVMe SSD at 5 GB/s takes 5 seconds to feed 25 GB of weights into VRAM, negligible next to initialisation.
HPA: irrelevant inside the 4090 (always 1 vLLM replica per GPU), but useful across nodes: 3 replicas on 3 nodes with a 4090 each, and the K8s Service spreads round-robin.
Useful concurrency: 4–8 simultaneous sessions with 8K of context, 1–2 with 32K.
Natural use case: PoC, small teams, departmental environments, edge.

The manifest above is sized for this scenario. Changing only the model and the args, the same Deployment serves Qwen, Mistral or whatever is next.

Scenario B — 5×H100 SXM (cluster with NVLink/NVSwitch)

Topology: 1 Pod with nvidia.com/gpu: 5 on an HGX node, --tensor-parallel-size=5. If the platform does not allow grouping 5 GPUs into a single Pod, a LeaderWorkerSet with 5 pods coordinated by Ray.
Model: up to 70B BF16 (Llama 3 70B) or up to 200B+ in FP8 with cache quantisation.
PVC: NVMe attached directly to the node, or fast network storage (Ceph with a 25/100 GbE network, Lustre, GPFS). Loading 140 GB of weights over a slow network takes 5 minutes per start.
HPA: irrelevant inside the 5-GPU cluster (the 5 are an indivisible unit), but useful for adding whole HGX nodes when the load passes a given threshold. This combines with Cluster Autoscaler if the underlying infrastructure allows it.
Useful concurrency: 32–128 simultaneous sessions with mid-sized contexts, 4–16 with huge contexts.
Natural use case: internal corporate service, public exposure with an SLA, multi-tenant.

A and B, side by side

Aspect	A (1×4090)	B (5×H100 SXM)
Pod topology	1 pod, 1 GPU	1 pod with 5 GPUs (or an LWS of 5)
Maximum BF16 model	8 B	70 B
TTFT @ 8K context, idle	~250 ms	~80 ms
TPOT, idle	~30 ms/tok	~15 ms/tok
Throughput @ concurrency 16	~50 tok/s/session	~200 tok/s/session
Session drain	30–60 s	60–180 s
Useful autoscaling	Replicas on peer nodes	Whole nodes via Cluster Autoscaler
Reasonable multi-tenancy	Limited: 4–8 sessions	Comfortable: 32–128 sessions
Indicative cost (hardware)	~2 K €	~250 K € (≈ 125×)

The asymmetry is the same one as in the previous article: 125× more expensive, only ~4× more throughput per session and ~10× more concurrency. What the cluster buys is not proportional; it buys access to models an order of magnitude larger and latencies low enough for interactive use at scale. If your workload is batch or asynchronous agents where latency is not critical, several 4090s come surprisingly close.

vLLM against TensorRT-LLM and SGLang

Honestly, all three are good engines. The choice depends on practical criteria, not technical ones. A decision map, not a benchmark:

Criterion	vLLM	TensorRT-LLM	SGLang
Supported hardware	NVIDIA, AMD ROCm, Intel Gaudi	NVIDIA only	NVIDIA, AMD ROCm
Pure latency (TTFT)	Good	Best: kernels compiled for the exact hardware	Good
Aggregate throughput	Excellent	Excellent	Excellent (RadixAttention)
Deployment	Trivial: Docker image + args	Complex: build an engine per model and per GPU	Moderate
OpenAI-compatible API	Native, complete	Yes, through Triton Inference Server	Yes
Support for new models	Days after release	Weeks (recompile the engine)	Days
Quantization	AWQ, GPTQ, FP8 cache	INT4/INT8/FP8, very mature	AWQ, FP8
Multi-modal	Yes (Llava, Pixtral, Qwen-VL)	Yes	Excellent, a priority
Function calling / tool use	Good	Limited	First class
Community / release cadence	Very active, weekly	Active, NVIDIA-driven	Very active, academic
Licence	Apache 2.0	Apache 2.0	Apache 2.0

When to choose each one:

vLLM: the “boring choice” that works. The path of least friction to production. If your team has no dedicated inference serving specialist, this one. It supports varied hardware, up-to-date models, a stable API and a huge community.
TensorRT-LLM: when per-request latency is the only metric that matters and your model is stable (trained in-house, not swapped every fortnight). The price of the performance is that every model plus every GPU plus every TRT version requires an engine rebuild, and that blocks fast iteration.
SGLang: for agent-dominated workloads (heavy tool calling) or complex multi-modal. Its RadixAttention, structural caching of prompts with shared prefixes, shines in ReAct-style patterns where the same system prompt is repeated thousands of times.

For most teams starting out with on-prem LLM serving, vLLM is the right answer until you have production data pushing you elsewhere.

Common operational traps

A list of gotchas that come up again and again:

The model is downloaded on every rolling update

Symptom: every deploy takes 5+ minutes to become available. Cause: there is no shared PVC. Every new pod downloads the model from Hugging Face from scratch. Fix: a ReadOnlyMany PVC on fast storage, or a local mirror of the registry (a Pod with huggingface-cli serving a directory over HTTP). In CI/CD, hydrating the PVC before the rollout is 1 line of bash.

Short readiness timeouts that kill pods while they load

Symptom: new pods go into CrashLoopBackOff during the first model load. Cause: a readinessProbe with too low a timeout fires before vLLM finishes loading; the livenessProbe finishes the job. Fix: a startupProbe with failureThreshold: 60 or more (10 minutes of grace) before liveness starts evaluating.

An unquantised KV cache, then OOM

Symptom: the pod starts fine, serves for five minutes, then gets OOMKilled when session number five arrives with a long context. Cause: a BF16 KV cache (the default) consumes twice what FP8 does. Fix: --kv-cache-dtype=fp8. Negligible quality loss in the vast majority of cases, double the capacity.

Confusing replicas with concurrency

Symptom: the HPA scales to 8 replicas under light real load and the cloud bill goes up. Latency does not improve. Cause: someone configured targetAverageUtilization: 50% on CPU, thinking that is “load”. The reality: a single vLLM replica serves dozens of simultaneous sessions. Fix: an HPA on vllm:num_requests_waiting. If the queue is empty, one replica is enough even if the GPU is at 90%.

Tensor parallel on GPUs without NVLink

Symptom: throughput 3× worse than expected, GPUs at 30%, a lot of PCIe traffic. Cause: tensor_parallel=4 on 4 GPUs connected only by PCIe; the all-reduce saturates the bus on every layer. Fix: either the GPUs share NVLink/NVSwitch (SXM/HGX models), or you use pipeline parallel (worse latency but less all-reduce), or you reduce TP and accept that the whole model does not fit.

Sessions cut off during a rolling update

Symptom: users see truncated responses during the deploy. Cause: terminationGracePeriodSeconds: 30 (the default) is not enough to drain long generations. Fix: terminationGracePeriodSeconds: 120–180. Combined with maxUnavailable: 0, rollouts are invisible to active users.

What we have not covered (upcoming articles)

vLLM with hot LoRA adapters: serving a base model plus N tenant-specific adapters without reloading weights.
Disaggregated serving: separating prefill and decode into specialised pods, each optimised for its GPU profile.
Quantization deep-dive: AWQ vs GPTQ vs dynamic FP8 vs FP4, the real trade-offs, and when to use each.
Gateway API + AI Inference Extensions: the sigwg proposal to make LLMs first-class citizens in K8s (routing by model, sticky sessions per conversation, multi-tenant fairness).
Multi-modal serving: the same runtime, a different kind of request — images, audio, embeddings.

References

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — the original vLLM paper.
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — the paper that popularised continuous batching.
Official vLLM documentation — operational and well maintained.
NVIDIA GPU Operator — installation and troubleshooting of the GPU layer on Kubernetes.
LeaderWorkerSet — the primitive for coordinated workloads such as multi-pod tensor parallel.
KEDA — event-driven autoscaling, ideal for scaling on queue metrics.
TensorRT-LLM and SGLang — the two most serious comparables.
LMSYS Chatbot Arena — periodic benchmarks comparing the three engines.
Previous article on this blog: KV cache: the working memory that holds up LLM inference.

KV cache: the working memory that holds up LLM inference

Mon, 18 May 2026 10:00:00 +0200

TL;DR

The KV cache is the working memory a language model keeps during a conversation. Without it, every new token would force a recomputation of the whole conversation from the start, at a cost that is quadratic in the length of the text. With it, the cost is linear, but in exchange the cache lives in VRAM and grows with every token. In practice, it is not the model that limits how much context you can serve: it is the KV cache. On an RTX 4090 running Llama 3 8B, the model fits in 16 GB and barely leaves room for ~64 K tokens of cache in total, adding up all the simultaneous sessions. Understanding that number is the difference between promising a client “128 K of context” and actually delivering it.

You are here: Deploy

This post opens the series on LLM inference fundamentals. Within the six-stage LLMOps pipeline that articulates the whole system, the KV cache lives in the Deploy stage: it is the piece that dictates how much traffic fits in your inference engine and, therefore, how much platform you can offer on top of it.

The analogy: the speaker with amnesia

Imagine you are at a two-hour technical conference. Every time the speaker is about to say a new sentence, he mentally rewinds the whole talk from the beginning, reassembles the thread, and only then carries on. His next sentence requires recalling the previous one; the one after that, the two before it; an hour in, every new word costs him an hour of recap. A conference like that would be materially impossible.

Now imagine the same speaker with a notebook in which he jots down, as he goes, the two or three key ideas of each sentence: subject, object, link to what came before. Before each new sentence he glances at the notebook and continues. His next word only costs a glance at the notebook, not a rewind of the entire talk.

That notebook, in a transformer, is called the KV cache. Without it, conversational language models would be unworkable. With it, they are commercial products. But the notebook weighs something, and understanding how much, where and why is what separates an inference infrastructure that works from one that falls over at the third concurrent client.

The mechanism itself (in plain terms)

A transformer generates text one token at a time. To decide the next token, the model applies a mechanism called attention over all previous tokens: it asks “which parts of the earlier context are relevant for predicting what comes next?”.

Internally, each input token is projected into three vectors:

Q (Query): “what I am looking for”
K (Key): “what this token offers”
V (Value): “what information this token carries”

The attention of the current token against the context is computed by multiplying its Q against the K of all previous tokens, normalising with softmax, and weighting the corresponding V. The result is a contextualised representation of the current token.

Q·Kᵀ → softmax × V

representation of token N

Here is the key point: to predict token N, I only need the new Q (that of token N) and the K, V of all previous tokens. The K and V of tokens 1..N-1 have not changed since the previous iteration. Recomputing them would be throwing work away.

The KV cache is exactly that: the memory that holds the K and V of every token already processed, in every layer of the model, so they never have to be recomputed.

Why it exists: the quadratic cost without it

Generating a text of N tokens involves N steps. At step i, attention is computed over i previous tokens. Without a cache, at every step you recompute the K, V of the i-1 previous tokens plus those of the new one. The total count of attention computations grows as:

$$\sum_{i=1}^{N} i = \frac{N(N+1)}{2} \approx \frac{N^2}{2}$$

With a KV cache, you only process the new token at each step: cost linear in N.

Cumulative compute to generate N tokens (schematic scale — exact data is in the table)

tokens generated (N) attention operations

0 1K 2K 3K 4K

with KV cache O(N) — linear

without KV cache O(N²) — quadratic

The concrete numbers are devastating:

Tokens generated	Without KV cache (operations)	With KV cache	Ratio
128	8,256	128	64×
1,024	524,800	1,024	512×
4,096	8,390,656	4,096	2,048×
32,768	536,887,296	32,768	16,384×

At 32 K tokens, the cache saves you four orders of magnitude of compute. This is not an optimisation: it is what makes conversational inference possible.

The price: how heavy the backpack is

The KV cache is paid for in VRAM. The formula, per sequence, is:

KV_size = 2 · n_layers · n_kv_heads · head_dim · context_len · bytes_per_param
↑
K and V

Per token (dropping context_len), it is a constant of the model itself. Let us look at real numbers:

Model	n_layers	n_kv_heads	head_dim	Bytes/token (BF16)	GB at 8 K ctx	GB at 32 K	GB at 128 K
Llama 3 8B (hypothetical MHA)	32	32	128	524,288	4.00	16.00	64.00
Llama 3 8B (real GQA)	32	8	128	131,072	1.00	4.00	16.00
Llama 3 70B (GQA)	80	8	128	327,680	2.50	10.00	40.00
Qwen3 8B (GQA)	36	8	128	147,456	1.12	4.50	18.00
Mistral 7B (GQA)	32	8	128	131,072	1.00	4.00	16.00

Two immediate readings:

Without GQA there is no 128 K worth talking about. A Llama 3 8B with classic multi-head attention would need 64 GB of KV cache alone for a single sequence with 128 K tokens. In other words, it does not fit on any consumer GPU. That is why Meta, Mistral and the rest adopted Grouped Query Attention.
The KV cache can be larger than the model. Llama 3 8B in BF16 takes ~16 GB. With 128 K of context, its cache is another 16 GB. A single session ties with the model in VRAM.

0 10 20 30 40 GB

0 8K 32K 64K 128K

≈ VRAM free after loading an 8B on a 4090

Llama 3 8B Qwen3 8B Llama 3 70B

The dashed red line marks the realistic VRAM available on an RTX 4090 after loading the model. Any model whose curve crosses that line will not be able to serve that context without additional strategies: cache quantisation, offload, partitioning.

Inference is memory-bound, not compute-bound

There is a common misconception: thinking that “fast GPU = fast inference”. In the regime where inference services with a KV cache actually operate, what gets measured is memory bandwidth. Every new token requires reading the K and V of all previous tokens from HBM. The compute is modest; the data movement is massive.

That is why an H100 SXM (3.35 TB/s of HBM3) can be 2-3× faster than an A100 (1.55-2 TB/s) without the clock frequency or the core count fully explaining the difference. Bandwidth explains it.

And that is also why offers of “cheap GPUs with plenty of VRAM but slow HBM” (some variants with GDDR6 or LPDDR5) disappoint in inference with long contexts: they have room to store the cache but take forever to re-read it.

Tricks to make the notebook thinner

Three techniques, in chronological order, have progressively flattened the size of the KV cache:

Multi-Head Attention (MHA). The original transformer design (Vaswani et al., 2017). Every attention head has its own K and V. Expensive in cache but theoretically maximal in expressiveness. It is what models had until around 2023.

Multi-Query Attention (MQA). A single K and V shared by all heads. It reduces the cache by a factor of n_heads. It works reasonably well but degrades generation quality on some benchmarks.

Grouped Query Attention (GQA). The middle ground that won. Heads are grouped: in Llama 3 8B, 32 query heads share K, V in groups of 4, giving 8 KV groups. It cuts the cache 4× relative to MHA with almost identical quality. It has been the de facto standard since 2024.

Multi-Head Latent Attention (MLA). DeepSeek-V2/V3’s innovation: instead of storing K, V per head, it compresses the state into a smaller latent vector and projects out to K, V on the fly. The cache can come down to 70 bytes/token, two orders of magnitude less than GQA. It is the main reason DeepSeek-V3 (671 B parameters, 37 B active) is servable on affordable infrastructure.

Note: the MLA bar is illustrative, using typical values published by DeepSeek; the exact implementation depends on the latent size. What matters is the order of magnitude.

On top of this comes a fourth, orthogonal technique: quantising the cache to FP8, INT8 or even INT4. vLLM and TensorRT-LLM already support it in production. Going from BF16 (2 bytes) to FP8 (1 byte) halves the cache at a small cost in quality. Going to INT4 divides it by four, at a somewhat higher cost.

The next dragon: fragmentation

So far we have talked about the cache as if it were a contiguous block. In practice, an inference server handles dozens of simultaneous sessions, each with its own cache growing at a different rate. Naive allocation, reserving the maximum possible per session, wastes between 60 % and 80 % of the VRAM according to the original PagedAttention paper.

session A

session B

session C

session D

→ ~70 % of VRAM reserved and empty

 <rect x="0" y="22" width="30" height="20" class="used blk"/>
<rect x="30" y="22" width="30" height="20" class="used blk"/>
<rect x="60" y="22" width="30" height="20" class="used blk"/>
<rect x="90" y="22" width="30" height="20" class="used blk"/>
<rect x="120" y="22" width="30" height="20" class="used blk"/>
<rect x="150" y="22" width="30" height="20" class="used blk"/>
<rect x="180" y="22" width="30" height="20" class="free blk"/>
<rect x="210" y="22" width="30" height="20" class="free blk"/>
<rect x="0" y="44" width="30" height="20" class="used blk"/>
<rect x="30" y="44" width="30" height="20" class="used blk"/>
<rect x="60" y="44" width="30" height="20" class="used blk"/>
<rect x="90" y="44" width="30" height="20" class="free blk"/>
<rect x="120" y="44" width="30" height="20" class="free blk"/>
<rect x="150" y="44" width="30" height="20" class="free blk"/>
<rect x="180" y="44" width="30" height="20" class="free blk"/>
<rect x="210" y="44" width="30" height="20" class="free blk"/>
</g>

→ < 4 % waste (vLLM paper)

PagedAttention, the idea from Kwon et al. (2023) that gave rise to vLLM, solves this by borrowing a technique from operating systems: divide the VRAM into small blocks (typically 16 tokens) and keep a page table mapping logical to physical blocks per session. A session no longer reserves one huge contiguous block: it grows one block at a time, and the blocks can be scattered across the VRAM. The result is an effective occupancy of 90 % instead of 30 %, and therefore 2-4× more aggregate throughput on the same hardware.

PagedAttention deserves an article of its own. I am noting it down for the next one.

Applied to generic on-premise hardware

Let us come down to concrete cases.

Case 1 — RTX 4090 (24 GB, Ada Lovelace)

A typical configuration with Qwen3-8B BF16:

Model in BF16: ~16 GB
Activations + overhead: ~2 GB
VRAM available for KV cache: ~6 GB (with margin)

At 144 KB/token (Qwen3-8B GQA), that comes to ~43 K total tokens of cache distributed across all simultaneous sessions. In practice:

Concurrency	Maximum context per session
1	32,768
4	8,192
8	4,096
16	2,048

If you need to advertise “we support 32 K of context” with concurrency of 4 or more, you have to quantise the cache (FP8 brings it down to 72 KB/token, doubling capacity) or move up the model range (a 4B with GQA and a quantised cache would have room to spare).

Case 2 — Cluster of 4×H100 (320 GB total, NVLink)

With tensor parallel = 4 and Llama 3 70B BF16:

Model in BF16: ~140 GB (35 GB/GPU)
vLLM overhead per GPU: ~2 GB
VRAM free for KV per GPU: ~43 GB → ~172 GB aggregate

At 320 KB/token (Llama 3 70B GQA), that comes to ~537 K total tokens of cache. Ample margin for long contexts with high concurrency:

Concurrency	Maximum context per session
4	134,000
16	33,500
64	8,375

For DeepSeek-V3 671 B with MLA the economics change radically, because the cache is ~100× thinner. What limits you is no longer the cache but the VRAM of the model itself (quantised to FP8 it is ~335 GB, so it fits on 4×H100 with room for the KV cache).

Operational implications

Three observations we repeat in every consulting engagement:

First, the maximum context a model advertises is not the one you can serve on your hardware. Llama 3 8B “supports” 128 K, but on a 4090 with 4 simultaneous sessions your effective context is ~8 K. It is trivial to check before promising it to a client.

Second, quantising the KV cache is one of the optimisations with the best cost/benefit ratio in the ENS context. It does not touch the weights, it does not affect audit reproducibility, and it doubles capacity. vLLM supports it via --kv-cache-dtype fp8.

Third, if the SLAs dictate long contexts with many concurrent users, GQA is necessary but not sufficient. In the medium term you need to look at models with MLA or attention variants with compression.

What we have not covered (upcoming articles)

PagedAttention and its implementation in vLLM: blocks, page table, eviction.
Prefix caching: when several requests share the system prompt, there is no need to recompute the K, V of the common part.
Speculative decoding and its interaction with the cache.
Cache offloading: moving cold blocks to RAM or NVMe, a key technique for contexts above 1 M.

References

Vaswani et al., Attention Is All You Need (NeurIPS 2017) — foundational transformer paper.
Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (EMNLP 2023).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM paper.
DeepSeek-AI, DeepSeek-V2 Technical Report (2024) — introduction of Multi-Head Latent Attention.
Official vLLM documentation: https://docs.vllm.ai/.
Llama 3 model card (Meta): GQA specs, n_layers, n_kv_heads.

Paged-Attention on lo0 — Tech Blog

vLLM on Kubernetes: the LLM inference piece that actually scales

TL;DR

The analogy: a multiprocess kernel for your GPU

What vLLM does inside

Continuous batching: stop waiting for the slowest

PagedAttention: virtual memory for the KV cache

Prefill vs decode: two phases with opposite profiles

Tensor parallel: when the model does not fit on one GPU

The OpenAI-compatible server

Why Kubernetes is the natural habitat

The GPU as a primitive resource

The vLLM Pod lifecycle

Anatomy of a serious deployment

First of all: GPU Operator

A complete, annotated vLLM Deployment

Multi-pod tensor parallel: LeaderWorkerSet

Autoscaling: the standard HPA is no use

Observability: the four metrics that matter

Two concrete scenarios

Scenario A — 1×RTX 4090 (workstation or small K8s node)

Scenario B — 5×H100 SXM (cluster with NVLink/NVSwitch)

A and B, side by side

vLLM against TensorRT-LLM and SGLang

Common operational traps

The model is downloaded on every rolling update

Short readiness timeouts that kill pods while they load

An unquantised KV cache, then OOM

Confusing replicas with concurrency

Tensor parallel on GPUs without NVLink

Sessions cut off during a rolling update

What we have not covered (upcoming articles)

See also

References

KV cache: the working memory that holds up LLM inference

TL;DR

You are here: Deploy

The analogy: the speaker with amnesia

The mechanism itself (in plain terms)

Why it exists: the quadratic cost without it

The price: how heavy the backpack is

Inference is memory-bound, not compute-bound

Tricks to make the notebook thinner

The next dragon: fragmentation

Applied to generic on-premise hardware

Case 1 — RTX 4090 (24 GB, Ada Lovelace)

Case 2 — Cluster of 4×H100 (320 GB total, NVLink)

Operational implications

What we have not covered (upcoming articles)

See also

References