--- title: "Batch sizing in vLLM: the two-hour grid search worth weeks of hardware" url: https://blog.lo0.es/en/posts/vllm-batch-sizing-two-hour-grid-search/ markdown: https://blog.lo0.es/en/posts/vllm-batch-sizing-two-hour-grid-search/index.md lang: en date: 2026-06-05 lastmod: 2026-09-13 author: "José Manuel Román" description: "Most deployments leave 40-60% of their throughput on the table by using vLLM's defaults. A complete methodology for finding the optimal configuration of max-num-seqs and max-num-batched-tokens in 2-8 GPU environments, with OTel metrics that confirm the result." tags: ["vllm","optimization","batch-sizing","throughput","opentelemetry","benchmarking"] translations: es: https://blog.lo0.es/posts/batch-sizing-vllm-grid-search/index.md --- # Batch sizing in vLLM: the two-hour grid search worth weeks of hardware ## TL;DR `max-num-seqs` and `max-num-batched-tokens` are the two dials that control how much work vLLM processes on each scheduler iteration. Their default values are calibrated to be safe on any hardware, not to maximise throughput on yours. A systematic grid search over 25 configurations, runnable in two hours, identifies the combination that, for your specific workload and hardware, can double throughput without changing a single line of the model or adding a GPU. The OTel metrics that confirm you found the optimum are `vllm:num_waiting_seqs`, `vllm:num_preemptions_total` and `vllm:time_per_output_token_seconds`. --- ## The analogy An industrial kitchen with one chef and ten stoves. If the maître d' only sends one order at a time, the chef works at 10% of capacity. If he sends a hundred simultaneous orders but there are ingredients for only twenty, the chef spends half his time waiting for restocking. The optimum is the point where every stove is lit and the resupply never runs dry. `max-num-seqs` is how many orders the chef can have in preparation at once. `max-num-batched-tokens` is how many ingredients he can process in a single movement of the wok. Getting either of them wrong leaves stoves empty. --- ## The problem: the defaults are not for your hardware In vLLM V1 (≥ 0.6), the defaults are: ``` max-num-seqs = 1024 (V1) / 256 (V0) max-num-batched-tokens = 8192 ``` These values guarantee that vLLM starts on any GPU without OOM. They do not guarantee optimal throughput. The reason: the optimal point depends on three variables vLLM does not know at start-up: 1. **The length distribution of your real workload** — a RAG system with 2K-token prompts needs a different budget from a chat with 50-token messages. 2. **VRAM available for KV cache** — determined by the model, the quantisation and `--gpu-memory-utilization`. 3. **Real expected concurrency** — how many simultaneous users arrive at the 95th percentile. The interaction between these three factors makes it impossible for a universal default to be optimal in specific cases. --- ## The arithmetic that matters On each iteration, vLLM's scheduler decides which tokens to process. The total budget available per step is `max-num-batched-tokens`. That budget is split between: - **Decode tokens**: 1 for each active request in the generation phase. With 64 requests in decode, 64 tokens of budget are consumed. - **Prefill tokens** (in chunks): the rest of the budget goes to processing new prompts. $$\text{prefill tokens per step} = \text{max num batched tokens} - \text{num requests decode}$$ If `max-num-batched-tokens = 8192` and you have 512 requests in decode, each step can only process `8192 - 512 = 7680` prefill tokens. With 2000-token prompts, that is ~3.8 new prompts per iteration. The problem appears when `max-num-seqs` is very high relative to the available KV cache. Each active request in decode occupies KV cache blocks. If the blocks run out, vLLM performs **preemption**: it pauses a request, frees its KV cache and re-queues it. Each preemption costs additional latency for the paused request and complexity for the scheduler. $$\text{KV budget} = \frac{\text{free VRAM} \times \text{gpu memory utilization}}{\text{bytes per token} \times \text{max model len}}$$ For a Qwen2.5-14B on an RTX 4090 with Q4_K_M (9 GB of model, 15 GB free): $$\text{KV budget} = \frac{15 \times 0.92 \times 10^9}{40\,000} \approx 345\,000 \text{ tokens}$$ With `max-model-len = 8192`, the maximum number of simultaneous requests with a full context is: $$\text{real max seqs} = \frac{345\,000}{8192} \approx 42 \text{ requests}$$ Setting `max-num-seqs = 1024` with those numbers guarantees constant preemptions. The optimum is at 40-50. --- ## The grid search: methodology ### Step 1: measure the real workload Before hunting for the optimum, you need to know your traffic's percentiles. From Langfuse or the vLLM logs: ```python # Extract the length distribution from Langfuse import langfuse client = langfuse.Langfuse() traces = client.fetch_traces(limit=1000).data prompt_lens = [t.input_tokens for t in traces if t.input_tokens] output_lens = [t.output_tokens for t in traces if t.output_tokens] import numpy as np print(f"Prompt p50={np.percentile(prompt_lens,50):.0f} p95={np.percentile(prompt_lens,95):.0f} p99={np.percentile(prompt_lens,99):.0f}") print(f"Output p50={np.percentile(output_lens,50):.0f} p95={np.percentile(output_lens,95):.0f} p99={np.percentile(output_lens,99):.0f}") ``` ### Step 2: work out the KV budget Run once with `--dry-run`, or read vLLM's start-up log: ``` INFO: # GPU blocks: 4521, # CPU blocks: 512 ``` Each block is 16 tokens. `4521 × 16 = 72,336 tokens` of total KV budget. ### Step 3: the grid With the KV budget known and the p95 of prompt/output length: ```python # grid_search_batch.py import subprocess, json, time MODEL = "Qwen/Qwen2.5-14B-Instruct-AWQ" PROMPT_LEN = 512 # p50 of your workload OUTPUT_LEN = 256 CONCURRENCY = 32 # simultaneous users expected at peak seqs_values = [32, 64, 128, 256, 512] tokens_values = [4096, 8192, 16384, 32768, 65536] results = [] for seqs in seqs_values: for tokens in tokens_values: cmd = [ "python", "-m", "vllm.entrypoints.benchmark_throughput", "--model", MODEL, "--max-num-seqs", str(seqs), "--max-num-batched-tokens", str(tokens), "--num-prompts", "200", "--input-len", str(PROMPT_LEN), "--output-len", str(OUTPUT_LEN), ] out = subprocess.run(cmd, capture_output=True, text=True, timeout=300) # Parse throughput from the output for line in out.stdout.splitlines(): if "Throughput" in line: tps = float(line.split(":")[1].strip().split()[0]) results.append({"seqs": seqs, "tokens": tokens, "tps": tps}) print(f"seqs={seqs} tokens={tokens} → {tps:.1f} tok/s") # Save for analysis with open("grid_results.json", "w") as f: json.dump(results, f, indent=2) ``` 25 configurations × ~5 min = **~2 hours**. Real execution time, not waiting time. ### Step 4: read the surface The result is a 5×5 throughput matrix. The typical shape: ``` max-num-batched-tokens → 4K 8K 16K 32K 64K max-num-seqs ↓ 32 180 310 380 390 385 ← max-num-seqs too low 64 185 350 480 510 508 ← optimal point for this workload 128 182 340 450 480 475 256 178 320 400 410 402 ← KV cache runs out, preemptions 512 170 290 360 370 368 ← high preemptions ``` The optimum in this example: `max-num-seqs=64, max-num-batched-tokens=32768`. Above that, preemptions cancel out the concurrency gain. --- ## Confirmation with OTel Once the optimal configuration is deployed, three Prometheus metrics confirm it is well calibrated: ```promql # 1. Queued requests — should stay close to 0 # If it grows steadily: max-num-seqs too low or max-num-batched-tokens insufficient vllm:num_waiting_seqs # 2. Preemptions — should be 0 or very occasional (<1/min) # If it grows: max-num-seqs too high for the available KV cache rate(vllm:num_preemptions_total[5m]) * 60 # 3. ITL (inter-token latency) — should be stable, with no spikes # Bimodality = badly calibrated batch size (some requests outside the CUDA graph bucket) histogram_quantile(0.99, rate(vllm:time_per_output_token_seconds_bucket[5m])) ``` The optimal configuration produces: - `num_waiting_seqs` ≈ 0 under normal operation - `num_preemptions_total` stable (not growing) - `time_per_output_token` unimodal If `num_waiting_seqs` is high with `gpu_cache_usage_perc` low: raise `max-num-batched-tokens` to process prefills faster. If `num_preemptions_total` grows: lower `max-num-seqs` or enable FP8 KV cache to free up blocks. --- ## Reference configurations by profile Based on the grid search for mid-range hardware (generic 4×H100, 14B-70B model): | Profile | Prompt p50 | Output p50 | max-num-seqs | max-num-batched-tokens | |:---|---:|---:|---:|---:| | Conversational chat | 150 tok | 300 tok | 256 | 16384 | | Enterprise RAG | 1500 tok | 200 tok | 64 | 32768 | | Coding (completion) | 800 tok | 500 tok | 128 | 32768 | | Summarisation | 2500 tok | 400 tok | 32 | 65536 | | Batch processing | 4000 tok | 800 tok | 16 | 65536 | None of these is universal. They are starting points for the grid search on your real hardware and workload. --- ## When not to touch the defaults If your system sits below 50% KV cache utilisation (`vllm:gpu_cache_usage_perc < 0.50`) under real demand and with no `num_waiting_seqs`, the defaults are enough for your current load. The grid search contributes more when you are close to maximum capacity or when you want to extract the full performance of fixed hardware. --- ## See also - [`max-num-batched-tokens` is the budget chunked prefill uses to interleave decode; this article covers the tuning of that parameter](https://blog.lo0.es/en/posts/optimising-prefill-vllm-knobs-ttft-will-not-forgive/index.md) - [`max-num-seqs` interacts directly with `gpu-memory-utilization` and the KV cache capacity for decode](https://blog.lo0.es/en/posts/optimising-decode-vllm-squeezing-every-token-small-hardware/index.md) - [the `num_waiting_seqs`, `num_preemptions_total` and `time_per_output_token` metrics configured in the full OTel pipeline](https://blog.lo0.es/en/posts/instrumenting-vllm-otel-measuring-optimisations/index.md) - [the KV budget formula that determines the real maximum of `max-num-seqs` for your hardware](https://blog.lo0.es/en/posts/kv-cache-working-memory-llm-inference/index.md) - [hardware sizing starts from the optimal throughput this grid search determines](https://blog.lo0.es/en/posts/capacity-planning-on-premise-llm-inference/index.md) ### In this same series - [the second free optimisation: taking the prefix cache hit rate from 15% to 75%](https://blog.lo0.es/en/posts/prefix-cache-hit-rate-engineering/index.md) - [FP8 in weights and KV cache: +40-60% throughput measured before and after with an eval suite](https://blog.lo0.es/en/posts/fp8-end-to-end-enable-measure-quality-decide/index.md) - [TP=4×1 vs TP=2×2: when the crossover point changes the platform decision](https://blog.lo0.es/en/posts/one-big-replica-or-many-small-ones-platform-decision/index.md) --- ## References - [vLLM Optimization and Tuning — documentación oficial](https://docs.vllm.ai/en/stable/configuration/optimization/) - [Tuning vLLM for Maximum Throughput (2026)](https://medium.com/@mahernaija/tuning-vllm-for-maximum-throughput-a-research-engineers-field-guide-21f341d71248) - [vLLM V1 performance optimization — ROCm](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/vllm-optimization.html) - [Anyscale: Tune parameters for LLMs](https://docs.anyscale.com/llm/serving/parameter-tuning)