Rtx4090 on lo0 — Tech Blog

Optimising decode in vLLM: squeezing every token out of small hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Decode is the phase in which vLLM generates output tokens one at a time. It is memory-bound, not compute-bound: the GPU spends more time waiting for the weights to arrive from VRAM than doing arithmetic. On small hardware, an RTX 4090 (24 GB) or an L40 (48 GB), a badly configured decode wastes half the card’s capacity. Five vLLM parameters change the equation: gpu-memory-utilization, max-num-seqs, speculative decoding, FP8 KV cache and a swap-space correctly set to zero. Well calibrated, the difference is real: from 15 tokens/s to 35–50 tokens/s on the same hardware.

The analogy

An assembly-line worker putting cars together. Every car needs exactly the same process: walk to the store for the part, come back, bolt it on, repeat. The travel time to the store, the VRAM latency, is fixed and cannot be eliminated. But there are ways to make it less painful:

Have several cars in parallel on the line (more concurrency, the same travel time amortised).
Have an assistant prefabricate common parts (speculative decoding: the draft model proposes, the verifier confirms).
Keep only the most-used parts in the workshop (quantised KV cache: more contexts fit in the same space).

Those three strategies are exactly the three axes of decode optimisation in vLLM.

Why decode is memory-bound

During prefill, the GPU processes N tokens in parallel: the attention operation is a large matmul and the compute units are busy. During decode, it processes 1 token per step: the matmul becomes a vector-matrix product, an operation that underuses the tensor cores.

The typical compute utilisation ratio during decode on an RTX 4090:

$$\text{MFU}_{decode} \approx 5–15\% \quad \text{(vs 40–60\% in prefill)}$$

The bottleneck is not compute power, it is bandwidth. To generate each token, the model has to read its complete weights from VRAM:

$$\text{time per token} \approx \frac{\text{weight size in bytes}}{\text{VRAM bandwidth}}$$

For Qwen2.5-7B in BF16 (14 GB of weights) on an RTX 4090 (1,008 GB/s):

$$t \approx \frac{14 \times 10^9}{1.008 \times 10^{12}} \approx 13.9 \text{ ms/token} \approx 72 \text{ tokens/s theoretical maximum}$$

The real value is lower (~30–50 tok/s) because of scheduler overhead, attention over the growing KV cache and other latencies. But the theoretical limit sets the ceiling.

With Q4_K_M (weights ~4 GB):

$$t \approx \frac{4 \times 10^9}{1.008 \times 10^{12}} \approx 3.97 \text{ ms/token} \approx 252 \text{ tokens/s theoretical}$$

Quantising the model is the most direct way to improve decode throughput on memory-bound hardware. Everything else optimises on top of that ceiling.

The five levers

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

--gpu-memory-utilization (short form --gpu-mem-util) defines the fraction of available VRAM that vLLM may use for the KV cache, once the model weights are loaded. The rest is reserved for activations during the forward pass and for the CUDA context.

vllm serve my-model \
 --gpu-memory-utilization 0.92

The default value is 0.90. On bare metal where no other process uses the GPU, 0.92–0.95 is safe. Do not go above 0.95: vLLM needs headroom for activations during batch peaks, and running out of VRAM in the middle of an inference results in a process crash, not a clean error.

Why it matters: more KV cache available = more simultaneous requests in flight = better GPU utilisation during decode. PagedAttention allocates the KV cache in fixed-size blocks (16 tokens per block by default), and vLLM manages them like virtual memory pages. The more blocks available, the more requests it can serve without any of them waiting for space.

RTX 4090, Qwen2.5-7B-BF16 (14 GB weights):
Total VRAM: 24 GB
Weights: 14 GB
Available for KV cache: 10 GB
gpu-memory-utilization 0.90 → 0.90 × 10 GB = 9 GB for KV cache
gpu-memory-utilization 0.94 → 0.94 × 10 GB = 9.4 GB → ~4% more tokens in flight

The impact is modest with models that fit comfortably, but it is amplified with models that push VRAM to its limit.

2. Real concurrency — `--max-num-seqs`

--max-num-seqs is the maximum number of requests vLLM can have in process simultaneously (counting prefill and decode together). It is the parameter that controls the system’s effective concurrency.

vllm serve my-model \
 --max-num-seqs 128

The effect is direct: more requests in simultaneous decode = better amortisation of the fixed cost of reading the weights. When the decode batch grows from 1 to 8, the time to generate 8 tokens is almost the same as generating 1 (the weights are read once for all of them). Aggregate throughput scales almost linearly until the KV cache or the activation VRAM becomes the bottleneck.

$$\text{aggregate throughput}(B) \approx B \times \text{throughput}(1) \quad \text{for } B \ll B_{max}$$

Common mistake: raising --max-num-seqs without making sure there is enough KV cache in VRAM for all the requests. If vLLM cannot hold the KV caches of 128 simultaneous requests, it does preemption (pauses some request and frees its KV cache) at a latency cost. Monitor vllm:num_preemptions_total.

Interaction with --max-num-batched-tokens: the vLLM scheduler processes up to max-num-batched-tokens tokens per step. If you have 128 requests in decode generating 1 token each, that is 128 decode tokens. The decode budget consumes 128 tokens of the total budget; the rest goes to prefill in chunks. Tune both values together.

# For an RTX 4090 serving ~50 concurrent users with responses of up to 512 tokens
vllm serve my-model \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192
# 64 decode tokens per step + up to 8128 chunked prefill tokens

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

Speculative decoding is the highest-impact change for decode on small hardware. The idea is simple: a small draft model proposes several tokens at once, and the verifier model validates or rejects them in a single forward pass.

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1

Why it works: the 7B verifier has to read 14 GB of weights per step. With 5 proposed tokens, if the acceptance rate is 80%, an average of 4–5 tokens are generated per verifier step instead of 1. Effective throughput goes up without the GPU working any harder.

The acceptance rate (α) depends on how well the draft predicts the verifier’s distribution. Within the same domain, models from the same family usually have α > 0.75:

$$\text{speedup} \approx \frac{1 + \alpha \cdot k}{1 + \alpha \cdot k / \text{cost ratio}}$$

Where $k$ is the number of proposed tokens and cost_ratio is the draft/verifier cost ratio. For a 0.5B draft and a 7B verifier (ratio ~14×):

$$\text{speedup} \approx 1 + 0.8 \times 5 \approx 5 \text{ (theoretical maximum, not reachable)}$$

In practice, with α = 0.75 and k = 5 on hardware without NVLink: 1.8–2.5× more tokens/s compared with decode alone.

EAGLE-3 in 2026: the best drafters today are not small versions of the same model, but networks specialised in predicting the verifier’s distribution. EAGLE-3 reports a 3–6.5× speedup over vanilla decode in public benchmarks. In production with mixed batches the real speedup is more conservative (1.5–3×). vLLM supports EAGLE/EAGLE-2 via --speculative-model:

# With an EAGLE drafter (requires a drafter trained specifically for the base model)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --speculative-model yuhuili/EAGLE3-LLaMA3.1-Instruct-8B \
 --num-speculative-tokens 6

When speculative decoding does NOT help:

Very large batches (>32 requests): the acceptance rate varies between requests and the batch spends more time re-drafting than accepting.
High-entropy tasks (brainstorming, very creative code): the draft predicts worse, α falls below 0.5 and the draft overhead outweighs the gain.
If the draft model does not fit in the available VRAM alongside the verifier.

On an RTX 4090 with a 7B verifier and a 0.5B draft (BF16): 14 + 1 GB = 15 GB. That leaves 9 GB for the KV cache. It works.

4. Quantised KV cache — `--kv-cache-dtype fp8`

This was already covered in the prefill article for its effect on context capacity. From the decode point of view, the benefit is different: more tokens fit in the KV cache → more simultaneous requests without preemption → better aggregate throughput.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Precision warning in decode: the KV cache is read on every attention step of the decode. Quantisation introduces noise into the attention activations. For long texts (>4K tokens of context) it can accumulate. In quality benchmarks (MMLU, HellaSwag) the degradation with FP8 KV and --calculate-kv-scales is <0.5% on modern models. Without --calculate-kv-scales, the degradation can be larger because the scales are fixed statically.

Optimal combination for RTX 4090:

vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ \
 --quantization awq \  # weights in INT4: 4 GB model
 --kv-cache-dtype fp8 \  # KV cache at half the size
 --calculate-kv-scales \
 --gpu-memory-utilization 0.94

# Available VRAM: 24 - 4 = 20 GB for KV cache
# With FP8: ~40 KB/token (vs 80 KB BF16) → 20 GB / 40 KB = 500,000 tokens of total context
# With max-num-seqs 64 and a 4K ctx: 64 × 4096 × 40KB = 10 GB → fits with room to spare

5. Remove the swap — `--swap-space 0`

--swap-space defines how much system RAM (not VRAM) vLLM may use to preempt KV caches. When vLLM has more active requests than fit in VRAM, it can “pause” some by moving their KV cache to RAM and reactivating them later.

The problem: moving a 4K-token KV cache from VRAM to RAM and back has a latency of tens of milliseconds over PCIe. For a system where you want predictable latency, the swap introduces unacceptable jitter.

vllm serve my-model \
 --swap-space 0

With --swap-space 0, when vLLM cannot hold any more requests in VRAM, it simply queues them instead of doing preemption. The queue adds waiting latency, but it is predictable and it does not interrupt the requests already in flight.

When should you have swap? If your workload has short demand peaks and you can tolerate occasional jitter in exchange for not rejecting requests, a swap of 4–8 GB can be useful. In ENS deployments where latency is a contractual SLA, --swap-space 0 is the right option.

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --max-num-seqs 64 \
 --max-num-batched-tokens 8192 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-0.5B-Instruct \
 --num-speculative-tokens 5 \
 --speculative-draft-tensor-parallel-size 1 \
 --dtype bfloat16

Expected throughput: 35–55 tokens/s per user, up to 64 simultaneous, TTFT <500ms for prompts <1K tokens.

L40 (48 GB) — 14B model, multi-user

vllm serve Qwen/Qwen2.5-14B-Instruct \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --max-num-seqs 128 \
 --max-num-batched-tokens 16384 \
 --enable-chunked-prefill \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales \
 --swap-space 0 \
 --speculative-model Qwen/Qwen2.5-1.5B-Instruct \
 --num-speculative-tokens 5 \
 --dtype bfloat16

Expected throughput: 25–40 tokens/s per user, up to 128 simultaneous with speculative decoding active, TTFT <800ms for prompts <2K tokens.

How to measure that decode is optimised

# Key metrics at vllm:8000/metrics

vllm:generation_tokens_total # total tokens generated → trend
vllm:e2e_request_latency_seconds_* # end-to-end latency per percentile
vllm:time_per_output_token_seconds_* # ITL (inter-token latency)
vllm:num_preemptions_total # if it rises, the KV cache is filling up
vllm:spec_decode_draft_acceptance_rate # speculative decoding hit rate

If spec_decode_draft_acceptance_rate < 0.6, the drafter is not helping: turn speculative decoding off or find a drafter better trained for your model or domain.

If num_preemptions_total grows, you have too many simultaneous requests for the available KV cache. Options: lower max-num-seqs, enable FP8 KV cache, lower max-model-len, or quantise the model further.

Implications for on-premise inference

In a sovereign deployment with fixed hardware, you cannot buy more GPUs at will. Every well-calibrated tenth of gpu-memory-utilization, every point of speculative decoding acceptance rate and every MB of KV cache freed by FP8 is real capacity you do not have to provision with another node.

The combination of quantised weights (AWQ/GPTQ), FP8 KV cache and speculative decoding lets a 14B serve on one L40 what without optimisations would need two L40s in tensor parallel. That is the economic argument for investing time in these parameters.

Decode cannot be accelerated indefinitely on memory-bound hardware: the theoretical limit is set by VRAM bandwidth. But the difference between the minimum and the maximum achievable on that hardware can be 3–4× with the right levers.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees — Li et al., 2024
Fast Inference from Transformers via Speculative Decoding — Leviathan et al., 2022
vLLM Optimization and Tuning — documentación oficial
vLLM Speculative Decoding
FP8 KV Cache en vLLM

Optimising prefill in vLLM: the knobs your TTFT will not forgive

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

Prefill is the phase in which vLLM processes your input prompt and produces the initial KV cache. It is compute-bound (unlike decode, which is memory-bound), it takes longer the longer the prompt is, and it blocks the decode of every other queued request. There are four levers in vLLM that radically change its behaviour: chunked prefill, prefix caching, FP8 KV cache and the per-batch token budget. With modest hardware, a 24 GB RTX 4090 or a 48 GB L40, the difference between ignoring them and using them well can be a 3× lower TTFT and 40% more aggregate throughput.

The analogy

Picture a printing shop from the early twentieth century. Setting the lead type (preparing the forme) is slow and blocks the press. Printing the pages already set is fast, but it needs the forme ready before it can start.

Prefill is setting the type. Decode is printing. A press that can only do one thing at a time, either set or print, leaves the machinery idle half the time. The historical solution was to have one worker setting the next page while the previous one was already on the press. That is, exactly, chunked prefill.

What prefill really is

When a request arrives at vLLM, the engine has to process every token of the prompt at once before it can emit the first token of the response. During that processing it computes, for each prompt token, its attention Key and Value vectors. The result, the initial KV cache, is stored in VRAM and used throughout the subsequent decode.

Unlike decode, where the model processes one new token per step, in prefill it processes N tokens in one go. That makes it far more efficient in FLOPs/token (GPUs are good at large matmuls), but it carries a quadratic cost in attention:

$$\text{prefill attention FLOPs} \approx 4 \cdot N^2 \cdot d_{model}$$

With a prompt of 1,000 tokens and $d_{model} = 4096$ (Qwen2.5-7B): $4 \cdot 10^6 \cdot 4096 \approx 16 \times 10^9$ FLOPs in attention alone. With 4,000 tokens, 256× more, because of the quadratic nature.

Prefill (compute-bound):
prompt tokens → [attention O(N²)] → [FFN] → initial KV cache
Decode (memory-bound):
1 new token → [cross-attention over KV cache] → next token

Why prefill is a problem on small hardware

On an H100 with 3.35 TB/s of bandwidth, a long prefill amortises quickly. On an RTX 4090 (1.008 TB/s) or an L40 (864 GB/s), the bottleneck shows up sooner and has concrete consequences:

The head-of-line blocking problem. By default, vLLM processes a complete prefill before doing any decode. If you have 10 requests queued, 9 in decode and 1 with an 8,000-token prompt, those 9 requests stall while the GPU chews through the long prefill. Their users see the streaming freeze. This is called head-of-line blocking and it is the number one enemy of TTFT in production.

The four levers

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

Chunked prefill splits a long prefill into pieces (chunks) and interleaves them with decode steps in the same batch. In vLLM V1 (≥ 0.6) it is on by default.

vllm serve my-model \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096

--max-num-batched-tokens is the total token budget vLLM can process in a single engine step, counting prefill and decode together. It is the most important parameter for controlling the trade-off:

`max-num-batched-tokens`	Effect
Low (512–2048)	More decode steps per cycle → better ITL, worse TTFT
High (8192–32768)	Large prefill chunks → better TTFT and throughput, worse ITL

For an RTX 4090 serving 7B–13B models with mixed contexts (256–4096 tokens):

--max-num-batched-tokens 8192 # a reasonable balance point

For an L40 (48 GB) with larger models and longer prompts:

--max-num-batched-tokens 16384

How it works internally: with a budget of 4,096 tokens and a prefill of 10,000, vLLM splits it into 3 chunks (4,096 + 4,096 + 1,808). Between chunks, it processes the pending decode steps. The requests in decode keep advancing; the long prefill takes longer to finish, but it freezes nothing.

Without chunked prefill:
t=0 [prefill 10k tokens]─────────────────────────────┐
t=1 └─[decode r1,r2...r9]
With chunked prefill (budget 4096):
t=0 [prefill chunk 4096][decode r1..r9]
t=1 [prefill chunk 4096][decode r1..r9]
t=2 [prefill chunk 1808][decode r1..r9]
t=3 [decode all, including the new one]

The TTFT of the long request rises slightly (3 steps instead of 1), but the ITL of the other 9 requests is not interrupted.

2. Prefix caching — `--enable-prefix-caching`

If several requests share the same prefix, a system prompt, a few-shot block, a context document, vLLM can compute the KV cache of that prefix once and reuse it.

vllm serve my-model \
 --enable-prefix-caching

This is called Automatic Prefix Caching (APC). Internally, vLLM splits the KV cache into fixed-size blocks (16 tokens per block by default) and assigns them a content-based SHA hash. When a new request arrives, it checks whether any initial block is already in the cache. On a hit, that prefill is skipped.

The numerical impact: suppose a 512-token system prompt that appears in 80% of your requests, and a rate of 100 req/min:

Without APC: 80 req/min × 512 tokens × prefill_cost = 41,000 tokens/min of redundant prefill
With APC (80% hit rate): 20 req/min × 512 = 10,240 tokens/min of prefill → a 75% reduction

The TTFT of those 80 requests falls to whatever it costs to process only the new suffix.

Limitation with chunked prefill: when chunked prefill is active, only the first chunk of the prefill benefits from APC in the current vLLM implementation. For workloads where the cache hit rate is very high and the suffixes are short, consider lowering --max-num-batched-tokens so that the first chunk covers more of the shared prefix.

# Configuration optimised for a high prefix cache hit rate
vllm serve my-model \
 --enable-prefix-caching \
 --enable-chunked-prefill \
 --max-num-batched-tokens 4096 # smaller chunks = prefix fits in chunk 1

3. FP8 KV cache — `--kv-cache-dtype fp8`

The KV cache takes up VRAM. The more VRAM it consumes, the fewer concurrent requests you can keep in flight. On a 24 GB RTX 4090, the Qwen2.5-14B model in BF16 already takes ~28 GB, so it does not fit. In Q4 it takes ~9 GB, leaving ~14 GB for the KV cache.

How many context tokens fit in 14 GB of BF16 KV cache for a 14B with GQA?

$$\text{KV size per token} = 2 \cdot n_{kv\_heads} \cdot d_{head} \cdot n_{layers} \cdot 2 \text{ bytes}$$

For Qwen2.5-14B: $n_{kv\_heads}=8$, $d_{head}=128$, $n_{layers}=40$, BF16 → $2 \cdot 8 \cdot 128 \cdot 40 \cdot 2 = 163{,}840$ bytes ≈ 160 KB/token.

14 GB / 160 KB ≈ 87,500 tokens of total context. With 8 users in parallel and 4,096 tokens of context each: 32,768 tokens used out of 87,500. There is headroom, but it is finite.

Moving to FP8 (1 byte instead of 2):

$$\text{KV FP8} = 80 \text{ KB/token} \implies 14 \text{ GB} / 80 \text{ KB} = 175{,}000 \text{ tokens}$$

Twice the context capacity with the same VRAM. That buys either more concurrency or longer contexts.

vllm serve my-model \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales # calibrates the scales dynamically; without this there is degradation

Warning for RTX 4090 and L40: Ada Lovelace has FP8 instructions at the CUDA level but without Hopper’s (H100’s) dedicated scaling hardware. The memory reduction is real; the compute speedup is smaller than on an H100. Do not expect the same speedup as in a Hopper datacenter. On the L40S (the variant with optimised FP8 tensor cores) the benefit is larger than on an RTX 4090.

4. Context budget — `--max-model-len`

--max-model-len defines the maximum number of tokens vLLM can handle in a single request (prompt plus generation). It is the hard limit that determines how much VRAM is reserved for the KV cache in the worst case.

On small hardware, lowering it frees VRAM for more concurrency:

# 7B model on RTX 4090, typical context of 4K but the model supports 128K
vllm serve my-model \
 --max-model-len 8192 \  # instead of 131072
 --gpu-memory-utilization 0.92

With the context cut to 8,192 tokens, vLLM does not reserve KV cache for 131,072 potential tokens and can fit more simultaneous requests. The risk is obvious: requests above 8,192 tokens fail with an error. Tune it to the P99 of your real length distribution.

Interaction between parameters

The four parameters are not independent. A common mistake is enabling prefix caching without tuning the block size, or raising max-num-batched-tokens without checking that max-num-seqs allows it to be filled:

max-num-batched-tokens = 8192
max-num-seqs = 4
average prompt = 512 tokens → 4 × 512 = 2048 prefill tokens < 8192
The 8192 budget is never filled because max-num-seqs caps it first.
Fix: raise max-num-seqs or lower max-num-batched-tokens.

A balanced configuration for RTX 4090 + a 7B model:

vllm serve Qwen/Qwen2.5-7B-Instruct \
 --gpu-memory-utilization 0.92 \
 --max-model-len 8192 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 8192 \
 --max-num-seqs 64 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

Configuration for an L40 (48 GB) + a 14B model:

vllm serve Qwen/Qwen2.5-14B-Instruct-AWQ \
 --gpu-memory-utilization 0.90 \
 --max-model-len 16384 \
 --enable-chunked-prefill \
 --max-num-batched-tokens 16384 \
 --max-num-seqs 128 \
 --enable-prefix-caching \
 --kv-cache-dtype fp8 \
 --calculate-kv-scales

How to measure that it is working

The metrics that confirm prefill is optimised:

# In vLLM's Prometheus metrics (port 8000/metrics):
vllm:time_to_first_token_seconds_bucket → TTFT distribution
vllm:gpu_cache_usage_perc → KV cache utilisation
vllm:prefix_cache_hit_rate → APC hit rate (if enabled)
vllm:num_running_seqs → simultaneous in-flight requests

A prefix_cache_hit_rate below 30% on workloads with a fixed system prompt means something in the hash is not working (a system prompt that varies by timestamp, a date format in the prompt, and so on).

Implications for on-premise inference

Chunked prefill and prefix caching are zero cost: they are enabled with flags and require no extra hardware. FP8 KV cache requires the model to be compatible (almost every modern transformer is) and that you are on Ada Lovelace or newer.

For sovereign ENS deployments where the hardware is fixed and you cannot scale horizontally on demand, a well-configured prefill is the difference between needing 4 nodes and needing 2 for the same load.

The second article in this series covers the decode optimisations: speculative decoding, KV cache tuning to maximise concurrency and how to configure gpu-memory-utilization without vLLM running out of VRAM at midnight.

References

Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al., 2023
Sarathi-Serve: Chunked Prefill and Stall-Free Scheduling — Agrawal et al., 2024
vLLM Optimization and Tuning — documentación oficial
Quantized KV Cache en vLLM

Rtx4090 on lo0 — Tech Blog

Optimising decode in vLLM: squeezing every token out of small hardware

TL;DR

The analogy

Why decode is memory-bound

The five levers

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

2. Real concurrency — `--max-num-seqs`

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

4. Quantised KV cache — `--kv-cache-dtype fp8`

5. Remove the swap — `--swap-space 0`

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

L40 (48 GB) — 14B model, multi-user

How to measure that decode is optimised

Implications for on-premise inference

See also

References

Optimising prefill in vLLM: the knobs your TTFT will not forgive

TL;DR

The analogy

What prefill really is

Why prefill is a problem on small hardware

The four levers

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

2. Prefix caching — `--enable-prefix-caching`

3. FP8 KV cache — `--kv-cache-dtype fp8`

4. Context budget — `--max-model-len`

Interaction between parameters

How to measure that it is working

Implications for on-premise inference

See also

References

Rtx4090 on lo0 — Tech Blog

Optimising decode in vLLM: squeezing every token out of small hardware

TL;DR

The analogy

Why decode is memory-bound

The five levers

1. Give vLLM all the VRAM you can — --gpu-memory-utilization

2. Real concurrency — --max-num-seqs

3. Speculative decoding — --speculative-model + --num-speculative-tokens

4. Quantised KV cache — --kv-cache-dtype fp8

5. Remove the swap — --swap-space 0

The reference configuration per hardware

RTX 4090 (24 GB) — 7B model, internal use

L40 (48 GB) — 14B model, multi-user

How to measure that decode is optimised

Implications for on-premise inference

See also

References

Optimising prefill in vLLM: the knobs your TTFT will not forgive

TL;DR

The analogy

What prefill really is

Why prefill is a problem on small hardware

The four levers

1. Chunked prefill — --enable-chunked-prefill + --max-num-batched-tokens

2. Prefix caching — --enable-prefix-caching

3. FP8 KV cache — --kv-cache-dtype fp8

4. Context budget — --max-model-len

Interaction between parameters

How to measure that it is working

Implications for on-premise inference

See also

References

1. Give vLLM all the VRAM you can — `--gpu-memory-utilization`

2. Real concurrency — `--max-num-seqs`

3. Speculative decoding — `--speculative-model` + `--num-speculative-tokens`

4. Quantised KV cache — `--kv-cache-dtype fp8`

5. Remove the swap — `--swap-space 0`

1. Chunked prefill — `--enable-chunked-prefill` + `--max-num-batched-tokens`

2. Prefix caching — `--enable-prefix-caching`

3. FP8 KV cache — `--kv-cache-dtype fp8`

4. Context budget — `--max-model-len`