One big replica or many small ones: the decision that defines your platform

TL;DR

With 4 GPUs available you have two basic options: one vLLM instance using all 4 (TP=4) or two independent instances using 2 each (TP=2 × 2 replicas). The first gives lower latency per individual request. The second gives higher aggregate throughput at high concurrency, better fault tolerance and finer-grained scaling. The crossover point, where the second overtakes the first, is typically between 16 and 64 concurrent requests for 70B models, much earlier than most people assume. The metric that decides it: goodput, the tokens generated within the latency SLO divided by the total.

The analogy

Two ways to organise a translation service: one senior translator with access to four specialised dictionaries at once (able to resolve any complex query in 30 seconds), or two junior translators with two dictionaries each (they take 45 seconds per complex query, but they can handle two at the same time).

For a client who arrives alone and expects a fast answer, the senior wins. For a queue of twenty clients arriving at once, the two juniors process twice as many queries per hour even though each one takes longer. The question is not who is better, but what kind of traffic you have.

The two architectures in vLLM

Architecture A: TP=4, one replica

# A single instance uses all 4 GPUs via tensor parallelism
vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.92 \
  --port 8000

GPU-0 ─┐
GPU-1 ─┤─ vLLM instance 0 ──► port 8000
GPU-2 ─┤  (TP=4, the model is
GPU-3 ─┘   split across 4 GPUs)

Every attention and FFN operation is split across 4 GPUs. They require an all-reduce after each layer (on NVLink: ~50-200 µs; on PCIe: ~2-8 ms). The complete model is available in the aggregate VRAM.

Architecture B: TP=2 × 2 replicas

# Two independent instances, each with 2 GPUs
# Instance 0 on GPU 0-1
CUDA_VISIBLE_DEVICES=0,1 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 --port 8000

# Instance 1 on GPU 2-3
CUDA_VISIBLE_DEVICES=2,3 vllm serve meta-llama/Meta-Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 --port 8001

GPU-0 ─┐                    ┌─► port 8000
GPU-1 ─┘─ vLLM instance 0 ─┘
                              ← load balancer
GPU-2 ─┐                    ┌─► port 8001
GPU-3 ─┘─ vLLM instance 1 ─┘

Each instance holds half the model. Requests are distributed across instances. No communication between instances (they are completely independent).

Why TP=4 has higher individual latency than TP=2

Tensor parallelism splits every transformer layer. After computing its fraction, each GPU needs to synchronise with the others via all-reduce before moving on to the next layer. The cost of that synchronisation:

For Llama 3 70B (80 layers) on 4×H100 NVLink:

On PCIe (without direct NVLink between GPUs):

That overhead is added to every decode step. With TP=2:

The difference between TP=2 and TP=4 on NVLink is ~6 ms per decode step, which matters for TPOT (inter-token latency) in streaming applications.

On PCIe without direct NVLink, TP=4 can be 400 ms slower per step than TP=2. For an output of 200 tokens, that is 80 extra seconds. In that scenario TP=4 over PCIe should never be used unless the model does not fit in 2 GPUs.

The crossover point: when TP=2×2 beats TP=4×1

For a 70B model on 4×H100 SXM (NVLink), aggregate throughput in tokens per second:

Concurrency   |  TP=4 × 1 instance   |  TP=2 × 2 instances   | Winner
──────────────┼──────────────────────┼───────────────────────┼────────
      1       |    200 tok/s         |    170 tok/s          | TP=4 (latency)
      4       |    650 tok/s         |    620 tok/s          | TP=4 (slight)
     16       |   1,800 tok/s        |   2,100 tok/s         | TP=2×2
     32       |   2,400 tok/s        |   3,600 tok/s         | TP=2×2 (+50%)
     64       |   2,800 tok/s        |   5,200 tok/s         | TP=2×2 (+86%)
    128       |   2,900 tok/s        |   5,800 tok/s         | TP=2×2 (+100%)

Why they diverge at high concurrency: with TP=4, the scheduler of a single instance manages every request but the KV cache is shared. With TP=2×2, each instance has its own scheduler and KV cache: less contention, more real parallelism.

The crossover point on NVLink sits around 16-32 simultaneous requests for 70B. For smaller models (14B, 7B) the crossover happens earlier, because the TP communication overhead weighs more in relative terms.

The three implications nobody mentions

1. Fault tolerance

With TP=4 × 1 replica: if one GPU fails, the whole instance goes down. The service drops to 0% until the GPU recovers or the pod restarts on another node.

With TP=2 × 2 replicas: if one GPU fails, one instance goes down. The service keeps running at 50% capacity. For ENS/NIS2, where availability is a contractual requirement, this difference is decisive.

2. Autoscaling granularity

With KEDA or HPA based on vllm:num_waiting_seqs, autoscaling has to provision in multiples of the deploy unit:

• TP=4 × 1: each new node requires 4 GPUs. The minimum scaling granularity is 4 GPUs.
• TP=2 × 2: each new pod requires 2 GPUs. The minimum granularity is 2 GPUs, which is finer and more cost-efficient.

3. Quality degradation under load

TP=4 with many concurrent requests starts to suffer preemptions when the KV cache fills up. TP=2×2 spreads that pressure across two independent KV cache pools, so the probability of preemption is lower under the same total load.

Measuring the crossover point with OTel

Goodput is the right metric for comparing the two architectures. Not raw throughput (which ignores the SLO), but the tokens generated within the agreed TPOT SLO:

# Goodput: tokens generated with TPOT inside the SLO (e.g. <50ms/token)
# For TP=4:
rate(vllm:generation_tokens_total{instance="tp4"}[5m])
  * (1 - histogram_quantile(0.95, rate(vllm:time_per_output_token_seconds_bucket{instance="tp4"}[5m])) > 0.050)

# For TP=2×2 (sum of the two instances):
sum(rate(vllm:generation_tokens_total{instance=~"tp2-.*"}[5m]))
  * (1 - histogram_quantile(0.95, sum(rate(vllm:time_per_output_token_seconds_bucket{instance=~"tp2-.*"}[5m]))) > 0.050)

A direct comparison on the same dashboard, with synthetic traffic at different concurrency levels, determines the exact crossover point for your hardware and model.

The decision by workload profile

Kubernetes configuration for both architectures

# Parallel deployments for an A/B test or different topologies

# TP=2 instances (2 replicas per deployment)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-tp2
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: vllm
        args: ["serve", "meta-llama/Meta-Llama-3.1-70B-Instruct",
               "--tensor-parallel-size", "2",
               "--gpu-memory-utilization", "0.92"]
        resources:
          limits:
            nvidia.com/gpu: "2"   # 2 GPUs per pod
---
# Service with load balancing across the 2 replicas
apiVersion: v1
kind: Service
metadata:
  name: vllm-tp2
spec:
  selector:
    app: vllm-tp2
  ports:
  - port: 8000
  sessionAffinity: ClientIP          # for prefix cache awareness
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800          # 3 hours of affinity per session

The sessionAffinity: ClientIP setting in the Kubernetes Service is the simplest way to implement session-affinity routing: requests from the same client always go to the same replica, maximising the prefix cache hit rate on the conversation history.

See also

• Sharing a GPU between several workloads: time-slicing, MPS and MIG — the other half of the decision: how to split one GPU between workloads with time-slicing, MPS or MIG.

• Serving several models on a single GPU: co-residency, model swapping and sleep mode — when they are not replicas of the same model but several models rotating on one GPU.

• the max-num-seqs grid search changes with the architecture: one large replica tolerates a higher max-num-seqs than two small ones with the same total KV cache

• session routing (sessionAffinity) is the K8s implementation of prefix-aware routing: same client, same replica, same cache

• KEDA scales in pod units; TP=2×2 gives a granularity of 2 GPUs versus 4 GPUs for TP=4×1, which affects the cost of reactive autoscaling

• goodput computed over generation_tokens_total and time_per_output_token_seconds are the metrics that compare the two architectures

• the next level of separation when neither TP=4×1 nor TP=2×2 is enough: separating the prefill hardware from the decode hardware

• FP8 frees VRAM in every replica; combined with TP=2×2 the impact multiplies: more concurrency per replica and more replicas possible

• NVLink, NVSwitch and NCCL: the cable every token travels through — why the NVLink limit draws the boundary of this decision: the per-layer all-reduce that TP pays for is cheap over NVSwitch and extremely expensive over PCIe.

References

• vLLM issue #16300: TP=8 peor que TP=4 en 8×A100
• Scaling LLM Inference: DP, PP & TP en vLLM — Jarvislabs
• vLLM MoE Playbook: TP, DP, PP and Expert Parallelism — ROCm Blogs
• vLLM Distributed Inference Optimization Guide — DatabaseMart