Batch sizing in vLLM: the two-hour grid search worth weeks of hardware

Fri, 05 Jun 2026 04:00:00 +0000

TL;DR

max-num-seqs and max-num-batched-tokens are the two dials that control how much work vLLM processes on each scheduler iteration. Their default values are calibrated to be safe on any hardware, not to maximise throughput on yours. A systematic grid search over 25 configurations, runnable in two hours, identifies the combination that, for your specific workload and hardware, can double throughput without changing a single line of the model or adding a GPU. The OTel metrics that confirm you found the optimum are vllm:num_waiting_seqs, vllm:num_preemptions_total and vllm:time_per_output_token_seconds.

The analogy

An industrial kitchen with one chef and ten stoves. If the maître d’ only sends one order at a time, the chef works at 10% of capacity. If he sends a hundred simultaneous orders but there are ingredients for only twenty, the chef spends half his time waiting for restocking. The optimum is the point where every stove is lit and the resupply never runs dry.

max-num-seqs is how many orders the chef can have in preparation at once. max-num-batched-tokens is how many ingredients he can process in a single movement of the wok. Getting either of them wrong leaves stoves empty.

The problem: the defaults are not for your hardware

In vLLM V1 (≥ 0.6), the defaults are:

max-num-seqs = 1024 (V1) / 256 (V0)
max-num-batched-tokens = 8192

These values guarantee that vLLM starts on any GPU without OOM. They do not guarantee optimal throughput. The reason: the optimal point depends on three variables vLLM does not know at start-up:

The length distribution of your real workload — a RAG system with 2K-token prompts needs a different budget from a chat with 50-token messages.
VRAM available for KV cache — determined by the model, the quantisation and --gpu-memory-utilization.
Real expected concurrency — how many simultaneous users arrive at the 95th percentile.

The interaction between these three factors makes it impossible for a universal default to be optimal in specific cases.

The arithmetic that matters

On each iteration, vLLM’s scheduler decides which tokens to process. The total budget available per step is max-num-batched-tokens. That budget is split between:

Decode tokens: 1 for each active request in the generation phase. With 64 requests in decode, 64 tokens of budget are consumed.
Prefill tokens (in chunks): the rest of the budget goes to processing new prompts.

$$\text{prefill tokens per step} = \text{max num batched tokens} - \text{num requests decode}$$

If max-num-batched-tokens = 8192 and you have 512 requests in decode, each step can only process 8192 - 512 = 7680 prefill tokens. With 2000-token prompts, that is ~3.8 new prompts per iteration.

The problem appears when max-num-seqs is very high relative to the available KV cache. Each active request in decode occupies KV cache blocks. If the blocks run out, vLLM performs preemption: it pauses a request, frees its KV cache and re-queues it. Each preemption costs additional latency for the paused request and complexity for the scheduler.

$$\text{KV budget} = \frac{\text{free VRAM} \times \text{gpu memory utilization}}{\text{bytes per token} \times \text{max model len}}$$

For a Qwen2.5-14B on an RTX 4090 with Q4_K_M (9 GB of model, 15 GB free):

$$\text{KV budget} = \frac{15 \times 0.92 \times 10^9}{40\,000} \approx 345\,000 \text{ tokens}$$

With max-model-len = 8192, the maximum number of simultaneous requests with a full context is:

$$\text{real max seqs} = \frac{345\,000}{8192} \approx 42 \text{ requests}$$

Setting max-num-seqs = 1024 with those numbers guarantees constant preemptions. The optimum is at 40-50.

The grid search: methodology

Step 1: measure the real workload

Before hunting for the optimum, you need to know your traffic’s percentiles. From Langfuse or the vLLM logs:

# Extract the length distribution from Langfuse
import langfuse
client = langfuse.Langfuse()

traces = client.fetch_traces(limit=1000).data
prompt_lens = [t.input_tokens for t in traces if t.input_tokens]
output_lens = [t.output_tokens for t in traces if t.output_tokens]

import numpy as np
print(f"Prompt p50={np.percentile(prompt_lens,50):.0f} p95={np.percentile(prompt_lens,95):.0f} p99={np.percentile(prompt_lens,99):.0f}")
print(f"Output p50={np.percentile(output_lens,50):.0f} p95={np.percentile(output_lens,95):.0f} p99={np.percentile(output_lens,99):.0f}")

Step 2: work out the KV budget

Run once with --dry-run, or read vLLM’s start-up log:

INFO: # GPU blocks: 4521, # CPU blocks: 512

Each block is 16 tokens. 4521 × 16 = 72,336 tokens of total KV budget.

Step 3: the grid

With the KV budget known and the p95 of prompt/output length:

# grid_search_batch.py
import subprocess, json, time

MODEL = "Qwen/Qwen2.5-14B-Instruct-AWQ"
PROMPT_LEN = 512 # p50 of your workload
OUTPUT_LEN = 256
CONCURRENCY = 32 # simultaneous users expected at peak

seqs_values = [32, 64, 128, 256, 512]
tokens_values = [4096, 8192, 16384, 32768, 65536]

results = []
for seqs in seqs_values:
 for tokens in tokens_values:
 cmd = [
 "python", "-m", "vllm.entrypoints.benchmark_throughput",
 "--model", MODEL,
 "--max-num-seqs", str(seqs),
 "--max-num-batched-tokens", str(tokens),
 "--num-prompts", "200",
 "--input-len", str(PROMPT_LEN),
 "--output-len", str(OUTPUT_LEN),
 ]
 out = subprocess.run(cmd, capture_output=True, text=True, timeout=300)
 # Parse throughput from the output
 for line in out.stdout.splitlines():
 if "Throughput" in line:
 tps = float(line.split(":")[1].strip().split()[0])
 results.append({"seqs": seqs, "tokens": tokens, "tps": tps})
 print(f"seqs={seqs} tokens={tokens} → {tps:.1f} tok/s")

# Save for analysis
with open("grid_results.json", "w") as f:
 json.dump(results, f, indent=2)

25 configurations × ~5 min = ~2 hours. Real execution time, not waiting time.

Step 4: read the surface

The result is a 5×5 throughput matrix. The typical shape:

max-num-batched-tokens → 4K 8K 16K 32K 64K
max-num-seqs ↓
32 180 310 380 390 385 ← max-num-seqs too low
64 185 350 480 510 508 ← optimal point for this workload
128 182 340 450 480 475
256 178 320 400 410 402 ← KV cache runs out, preemptions
512 170 290 360 370 368 ← high preemptions

The optimum in this example: max-num-seqs=64, max-num-batched-tokens=32768. Above that, preemptions cancel out the concurrency gain.

Confirmation with OTel

Once the optimal configuration is deployed, three Prometheus metrics confirm it is well calibrated:

# 1. Queued requests — should stay close to 0
# If it grows steadily: max-num-seqs too low or max-num-batched-tokens insufficient
vllm:num_waiting_seqs

# 2. Preemptions — should be 0 or very occasional (<1/min)
# If it grows: max-num-seqs too high for the available KV cache
rate(vllm:num_preemptions_total[5m]) * 60

# 3. ITL (inter-token latency) — should be stable, with no spikes
# Bimodality = badly calibrated batch size (some requests outside the CUDA graph bucket)
histogram_quantile(0.99, rate(vllm:time_per_output_token_seconds_bucket[5m]))

The optimal configuration produces:

num_waiting_seqs ≈ 0 under normal operation
num_preemptions_total stable (not growing)
time_per_output_token unimodal

If num_waiting_seqs is high with gpu_cache_usage_perc low: raise max-num-batched-tokens to process prefills faster. If num_preemptions_total grows: lower max-num-seqs or enable FP8 KV cache to free up blocks.

Reference configurations by profile

Based on the grid search for mid-range hardware (generic 4×H100, 14B-70B model):

Profile	Prompt p50	Output p50	max-num-seqs	max-num-batched-tokens
Conversational chat	150 tok	300 tok	256	16384
Enterprise RAG	1500 tok	200 tok	64	32768
Coding (completion)	800 tok	500 tok	128	32768
Summarisation	2500 tok	400 tok	32	65536
Batch processing	4000 tok	800 tok	16	65536

None of these is universal. They are starting points for the grid search on your real hardware and workload.

When not to touch the defaults

If your system sits below 50% KV cache utilisation (vllm:gpu_cache_usage_perc < 0.50) under real demand and with no num_waiting_seqs, the defaults are enough for your current load. The grid search contributes more when you are close to maximum capacity or when you want to extract the full performance of fixed hardware.

Batch-Sizing on lo0 — Tech Blog