--- title: "One big replica or many small ones: the decision that defines your platform" url: https://blog.lo0.es/en/posts/one-big-replica-or-many-small-ones-platform-decision/ markdown: https://blog.lo0.es/en/posts/one-big-replica-or-many-small-ones-platform-decision/index.md lang: en date: 2026-06-05 lastmod: 2026-09-13 author: "José Manuel Román" description: "TP=4×1 replica vs TP=2×2 replicas on 4 GPUs: this is not just a technical decision, it defines fault tolerance, autoscaling granularity and the latency/throughput crossover point. With the exact OTel metrics that reveal when to switch from one architecture to the other." tags: ["vllm","tensor-parallelism","replicas","throughput","latency","opentelemetry","architecture"] translations: es: https://blog.lo0.es/posts/tp-replicas-una-grande-vs-n-pequenas/index.md --- # One big replica or many small ones: the decision that defines your platform ## TL;DR With 4 GPUs available you have two basic options: one vLLM instance using all 4 (TP=4) or two independent instances using 2 each (TP=2 × 2 replicas). The first gives lower latency per individual request. The second gives higher aggregate throughput at high concurrency, better fault tolerance and finer-grained scaling. The crossover point, where the second overtakes the first, is typically between 16 and 64 concurrent requests for 70B models, much earlier than most people assume. The metric that decides it: goodput, the tokens generated within the latency SLO divided by the total. --- ## The analogy Two ways to organise a translation service: one senior translator with access to four specialised dictionaries at once (able to resolve any complex query in 30 seconds), or two junior translators with two dictionaries each (they take 45 seconds per complex query, but they can handle two at the same time). For a client who arrives alone and expects a fast answer, the senior wins. For a queue of twenty clients arriving at once, the two juniors process twice as many queries per hour even though each one takes longer. The question is not who is better, but what kind of traffic you have. --- ## The two architectures in vLLM ### Architecture A: TP=4, one replica ```bash # A single instance uses all 4 GPUs via tensor parallelism vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \ --tensor-parallel-size 4 \ --gpu-memory-utilization 0.92 \ --port 8000 ``` ``` GPU-0 ─┐ GPU-1 ─┤─ vLLM instance 0 ──► port 8000 GPU-2 ─┤ (TP=4, the model is GPU-3 ─┘ split across 4 GPUs) ``` Every attention and FFN operation is split across 4 GPUs. They require an all-reduce after each layer (on NVLink: ~50-200 µs; on PCIe: ~2-8 ms). The complete model is available in the aggregate VRAM. ### Architecture B: TP=2 × 2 replicas ```bash # Two independent instances, each with 2 GPUs # Instance 0 on GPU 0-1 CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \ --tensor-parallel-size 2 --port 8000 # Instance 1 on GPU 2-3 CUDA_VISIBLE_DEVICES=2,3 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \ --tensor-parallel-size 2 --port 8001 ``` ``` GPU-0 ─┐ ┌─► port 8000 GPU-1 ─┘─ vLLM instance 0 ─┘ ← load balancer GPU-2 ─┐ ┌─► port 8001 GPU-3 ─┘─ vLLM instance 1 ─┘ ``` Each instance holds half the model. Requests are distributed across instances. No communication between instances (they are completely independent). --- ## Why TP=4 has higher individual latency than TP=2 Tensor parallelism splits every transformer layer. After computing its fraction, each GPU needs to synchronise with the others via all-reduce before moving on to the next layer. The cost of that synchronisation: $$\text{overhead TP} = n\_layers \times 2 \times \text{allreduce latency}$$ For Llama 3 70B (80 layers) on 4×H100 NVLink: $$\text{overhead TP4} = 80 \times 2 \times 100\,\mu s = 16\,ms$$ On PCIe (without direct NVLink between GPUs): $$\text{overhead TP4 PCIe} = 80 \times 2 \times 3\,ms = 480\,ms$$ That overhead is added to every decode step. With TP=2: $$\text{overhead TP2} = 80 \times 2 \times 60\,\mu s = 9.6\,ms \text{ (NVLink)}$$ The difference between TP=2 and TP=4 on NVLink is ~6 ms per decode step, which matters for TPOT (inter-token latency) in streaming applications. On PCIe without direct NVLink, TP=4 can be **400 ms slower per step** than TP=2. For an output of 200 tokens, that is 80 extra seconds. In that scenario TP=4 over PCIe should never be used unless the model does not fit in 2 GPUs. --- ## The crossover point: when TP=2×2 beats TP=4×1 For a 70B model on 4×H100 SXM (NVLink), aggregate throughput in tokens per second: ``` Concurrency | TP=4 × 1 instance | TP=2 × 2 instances | Winner ──────────────┼──────────────────────┼───────────────────────┼──────── 1 | 200 tok/s | 170 tok/s | TP=4 (latency) 4 | 650 tok/s | 620 tok/s | TP=4 (slight) 16 | 1,800 tok/s | 2,100 tok/s | TP=2×2 32 | 2,400 tok/s | 3,600 tok/s | TP=2×2 (+50%) 64 | 2,800 tok/s | 5,200 tok/s | TP=2×2 (+86%) 128 | 2,900 tok/s | 5,800 tok/s | TP=2×2 (+100%) ``` Why they diverge at high concurrency: with TP=4, the scheduler of a single instance manages every request but the KV cache is shared. With TP=2×2, each instance has its own scheduler and KV cache: less contention, more real parallelism. The crossover point on NVLink sits around **16-32 simultaneous requests** for 70B. For smaller models (14B, 7B) the crossover happens earlier, because the TP communication overhead weighs more in relative terms. --- ## The three implications nobody mentions ### 1. Fault tolerance With TP=4 × 1 replica: if one GPU fails, the whole instance goes down. The service drops to 0% until the GPU recovers or the pod restarts on another node. With TP=2 × 2 replicas: if one GPU fails, one instance goes down. The service keeps running at 50% capacity. For ENS/NIS2, where availability is a contractual requirement, this difference is decisive. ### 2. Autoscaling granularity With KEDA or HPA based on `vllm:num_waiting_seqs`, autoscaling has to provision in multiples of the deploy unit: - TP=4 × 1: each new node requires 4 GPUs. The minimum scaling granularity is 4 GPUs. - TP=2 × 2: each new pod requires 2 GPUs. The minimum granularity is 2 GPUs, which is finer and more cost-efficient. ### 3. Quality degradation under load TP=4 with many concurrent requests starts to suffer preemptions when the KV cache fills up. TP=2×2 spreads that pressure across two independent KV cache pools, so the probability of preemption is lower under the same total load. --- ## Measuring the crossover point with OTel Goodput is the right metric for comparing the two architectures. Not raw throughput (which ignores the SLO), but the tokens generated within the agreed TPOT SLO: ```promql # Goodput: tokens generated with TPOT inside the SLO (e.g. <50ms/token) # For TP=4: rate(vllm:generation_tokens_total{instance="tp4"}[5m]) * (1 - histogram_quantile(0.95, rate(vllm:time_per_output_token_seconds_bucket{instance="tp4"}[5m])) > 0.050) # For TP=2×2 (sum of the two instances): sum(rate(vllm:generation_tokens_total{instance=~"tp2-.*"}[5m])) * (1 - histogram_quantile(0.95, sum(rate(vllm:time_per_output_token_seconds_bucket{instance=~"tp2-.*"}[5m]))) > 0.050) ``` A direct comparison on the same dashboard, with synthetic traffic at different concurrency levels, determines the exact crossover point for your hardware and model. --- ## The decision by workload profile | Profile | Recommended architecture | Reason | |:---|:---|:---| | Single-user chatbot / low concurrency (<10 simultaneous) | TP=4 × 1 | Lower p50 latency, better streaming experience | | Enterprise API (20-100 concurrent) | TP=2 × 2 | Higher goodput, fault tolerance, finer autoscaling | | Batch processing (throughput > latency) | TP=2 × 2 (or more replicas) | Maximum throughput always lies in replicas | | Very large model (>80B, does not fit in 2 GPUs) | TP=4 × 1 | No structural alternative | | ENS / contractual availability | TP=2 × 2 minimum | Losing one GPU is not catastrophic | --- ## Kubernetes configuration for both architectures ```yaml # Parallel deployments for an A/B test or different topologies # TP=2 instances (2 replicas per deployment) apiVersion: apps/v1 kind: Deployment metadata: name: vllm-tp2 spec: replicas: 2 template: spec: containers: - name: vllm args: ["serve", "meta-llama/Meta-Llama-3.1-70B-Instruct", "--tensor-parallel-size", "2", "--gpu-memory-utilization", "0.92"] resources: limits: nvidia.com/gpu: "2" # 2 GPUs per pod --- # Service with load balancing across the 2 replicas apiVersion: v1 kind: Service metadata: name: vllm-tp2 spec: selector: app: vllm-tp2 ports: - port: 8000 sessionAffinity: ClientIP # for prefix cache awareness sessionAffinityConfig: clientIP: timeoutSeconds: 10800 # 3 hours of affinity per session ``` The `sessionAffinity: ClientIP` setting in the Kubernetes Service is the simplest way to implement session-affinity routing: requests from the same client always go to the same replica, maximising the prefix cache hit rate on the conversation history. --- ## See also - [Sharing a GPU between several workloads: time-slicing, MPS and MIG](https://blog.lo0.es/en/posts/sharing-one-gpu-time-slicing-mps-mig/index.md) — the other half of the decision: how to split one GPU between workloads with time-slicing, MPS or MIG. - [Serving several models on a single GPU: co-residency, model swapping and sleep mode](https://blog.lo0.es/en/posts/serving-several-models-single-gpu-co-residency-swapping-sleep-mode/index.md) — when they are not replicas of the same model but several models rotating on one GPU. - [the max-num-seqs grid search changes with the architecture: one large replica tolerates a higher max-num-seqs than two small ones with the same total KV cache](https://blog.lo0.es/en/posts/vllm-batch-sizing-two-hour-grid-search/index.md) - [session routing (`sessionAffinity`) is the K8s implementation of prefix-aware routing: same client, same replica, same cache](https://blog.lo0.es/en/posts/prefix-cache-hit-rate-engineering/index.md) - [KEDA scales in pod units; TP=2×2 gives a granularity of 2 GPUs versus 4 GPUs for TP=4×1, which affects the cost of reactive autoscaling](https://blog.lo0.es/en/posts/llm-inference-autoscaling-kubernetes-hpa-keda/index.md) - [goodput computed over `generation_tokens_total` and `time_per_output_token_seconds` are the metrics that compare the two architectures](https://blog.lo0.es/en/posts/instrumenting-vllm-otel-measuring-optimisations/index.md) - [the next level of separation when neither TP=4×1 nor TP=2×2 is enough: separating the prefill hardware from the decode hardware](https://blog.lo0.es/en/posts/disaggregated-serving-prefill-decode-specialised-pods/index.md) - [FP8 frees VRAM in every replica; combined with TP=2×2 the impact multiplies: more concurrency per replica and more replicas possible](https://blog.lo0.es/en/posts/fp8-end-to-end-enable-measure-quality-decide/index.md) - [NVLink, NVSwitch and NCCL: the cable every token travels through](https://blog.lo0.es/en/posts/nvlink-nvswitch-nccl-cable-every-token-crosses/index.md) — *why* the NVLink limit draws the boundary of this decision: the per-layer all-reduce that TP pays for is cheap over NVSwitch and extremely expensive over PCIe. --- ## References - [vLLM issue #16300: TP=8 peor que TP=4 en 8×A100](https://github.com/vllm-project/vllm/issues/16300) - [Scaling LLM Inference: DP, PP & TP en vLLM — Jarvislabs](https://docs.jarvislabs.ai/blog/scaling-llm-inference-dp-pp-tp) - [vLLM MoE Playbook: TP, DP, PP and Expert Parallelism — ROCm Blogs](https://rocm.blogs.amd.com/software-tools-optimization/vllm-moe-guide/README.html) - [vLLM Distributed Inference Optimization Guide — DatabaseMart](https://www.databasemart.com/blog/vllm-distributed-inference-optimization-guide)