Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

Contents

This post complements those on LLM autoscaling on Kubernetes (the autoscaler coexists with the rollout and must respect it), GPU observability for LLM inference (the metrics that act as gates come from there), Evals for LLMs (the eval that decides whether the new model is ready), LLM-as-judge (the technique that puts the “quality” into the canary gate) and Retrain: closing the loop (the previous step that the new model comes out of).

TL;DR

Promoting a new version of an LLM model to the production cluster without cutting traffic or breaking the SLO demands progressive deployment. The three canonical strategies, blue-green, canary and shadow, answer different questions and have different costs. Blue-green: a complete new pool brought up in parallel, atomic load balancer switch. Instant rollback (point back at the old pool); it demands twice the GPUs during the window. Canary: traffic is shared progressively between the old version and the new one (1 % → 5 % → 25 % → 100 %), measuring regression gates at each step; it consumes incrementally less hardware but exposes real users to the new model from the first percentage point. Shadow / mirror: the old model serves 100 % of the real traffic to the client and, in parallel, a copy of every request goes to the new model without returning its response to the user; it isolates you from quality risk but spends the new model’s GPU on responses nobody consumes, and it does not work well with long SSE streaming. The choice depends on three factors: available GPU budget, service criticality and the availability of a fast automatic eval. The five regression metrics any LLM canary should gate on are: TTFT P95, error rate (HTTP 5xx + premature finish_reason="length"), quality score with LLM-as-judge over a golden set, statistical drift of output embeddings (Wasserstein or KL against the baseline distribution) and cost per request (tokens/s and kW/request). On Kubernetes, Argo Rollouts manages traffic and the AnalysisTemplates as automatic gates; Flagger is the more opinionated alternative. vLLM v1 does not support hot model swap robustly as of May 2026, so the rollout unit is the whole replica (deployment v2 alongside deployment v1). The three specific pitfalls: LB sticky sessions break the statistical comparability of the canary (a client A always lands on the new one, B on the old one, so the populations are not equivalent); semantic eval with LLM-as-judge takes 2–8 seconds per sample and is no use as a real-time gate (it is used in post-analysis or offline before promotion); SSE streaming complicates shadowing because the new model’s response has to be discarded without affecting the old one’s. This post includes a minimal Argo Rollouts manifest applicable to a generic cluster with the NVIDIA GPU Operator.

You are here: DEPLOY (and the transition to RETRAIN)

A new model does not appear in the cluster by magic: it comes from the retrain loop or from a weights provider update. The step between “I have an artifact that passed offline eval” and “it is serving 100 % of the traffic” is exactly this post.

The analogy: opening night of a play

A theatre company is about to premiere a new version of a play that has been running successfully for a year. The company knows several hard facts: the current audience pays for a consistent experience; a bad opening damages the business for months; but premiering nothing leaves the company obsolete against the competition.

The three opening routes the management can choose are the same three of the LLM rollout.

Dress rehearsal behind closed doors (shadow / mirror). The new actors perform the whole play to an empty theatre. There is no audience; nobody buys a ticket. Three full run-throughs are enough to check continuity, timing and the chemistry of the cast. It is expensive because there are salaries and theatre rent, but it does not expose the audience to risk. Useful when the new cast is untested and the director wants to see how it holds up over a complete performance before selling it. In LLM terms: the new model processes every real request in parallel with the old one but its responses are discarded; you spend the new model’s GPU on responses nobody sees.

Cast by performance, alternating (canary). Instead of changing the whole cast at once, Thursday performances belong to the new cast, Friday ones to the old one, Saturday ones half and half. The management reads the visitors’ book comments and the audience numbers performance by performance, deciding after two weeks whether to promote the new cast to permanent or withdraw it. Cheaper than the dress rehearsal because the performances still sell tickets, but it exposes a real audience to risk from the first Thursday. In LLM terms: traffic is shared progressively between the old version and the new one, measuring gates at each step.

Double company with an atomic switch (blue-green). The company hires the new cast, prepares it for a month behind closed doors, and one Saturday announces: “from the next opening night every performance is with the new cast”. If the first performance goes badly, they go back to the old cast in 24 hours, but during that month of preparation double salaries are paid to both companies. In LLM terms: two complete pools of the same size, instant LB switch from one to the other, rollback in seconds if the metrics break.

The analogy also supports the decision: the choice depends on how critical the play is for the business (criticality of the LLM service), how much budget there is to sustain two casts at once (GPU budget), and how much confidence there is in the new cast from the chamber rehearsals (offline eval prior to the canary).

The three strategies in detail

Blue-green

The operator keeps two replica pools identical in size: the blue one (production version v1) and the green one (candidate version v2). When v2 is validated offline (eval passed, smoke tests), the LoadBalancer switch redirects 100 % of the traffic from blue to green in a single step. If the SLO metrics break, the switch goes back in seconds.

Cost: 2× GPUs for the whole window (preparing v2 + the post-switch observation window). For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), preparing the blue-green needs 16 extra GPUs for 1–3 days.

Risk: the switch is atomic. If v2 has a problem that did not show up in offline eval but does show up at scale (for example, edge cases that only appear at 200 RPS), 100 % of users notice it at once. Rollback is instant, but the requests of the first minute after the switch were already affected. So blue-green is preferable when you have high confidence in v2 (a minor change: same architecture, same format, just a new version of the weights) and you prioritise immediate rollback over gradual exposure.

Canary

The operator deploys v2 with a small number of replicas (typically 1) next to the v1 pool. The LoadBalancer shares the traffic out progressively on a schedule: 1 % for 30 minutes → 5 % for 1 hour → 25 % for 2 hours → 50 % for 4 hours → 100 %. Between each step, an analysis gate evaluates regression metrics over the traffic that is already landing on v2. If the gate fails, the rollback withdraws traffic from v2 automatically and leaves v1 serving everything.

Cost: incremental. At the start (1 % of traffic) one v2 replica is enough; at 50 % you need half as many v2 replicas as v1 has in total. Peak extra GPU during the canary: about 30–50 % above baseline.

Risk: real users are seeing v2 from the first 1 %. If v2 produces responses with degraded quality but normal TTFT and error rate, the affected users perceive the degradation without the gate detecting it (unless the gate includes quality drift, which is slow). So canary is preferable when you have medium confidence in v2 (a significant change: different architecture or training) and you accept that a low percentage of users will be guinea pigs.

Shadow / mirror

The LoadBalancer sends 100 % of the real traffic to v1 (which answers the client) and duplicates every request towards v2 (whose response is discarded or stored for offline analysis). The client never sees v2; it is never exposed to the risk.

Cost: 100 % extra compute for v2 with no direct user value for the whole shadow window. For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), a shadow of the same size consumes 16 extra GPUs full time.

Risk: shadow is the safest for the user. But it has two serious limitations: (a) if v2 has a bottleneck that makes the copied request to the shadow take a long time, the shadowing proxy can consume LB connections; it must be out-of-band (asynchronous); (b) long SSE streaming complicates mirroring because you have to keep two parallel streams and discard one while the other flows to the client. Common pattern: shadow only non-streaming requests (short completions, classification), manual offline eval of the streaming requests.

The five regression metrics that act as a gate

Without automatic gates, the “canary” is just a nice name for “manual rollout with a variable percentage”. The gates are the piece that turns the canary into a defensible operation.

Metric 1 — TTFT P95. Comparison of the new model’s P95 against the baseline (v1) P95 in 5-minute windows. Gate: ttft_p95(v2) / ttft_p95(v1) < 1.10. It detects prefill latency regressions (a slower new model) or engine problems (suboptimal config). Source: vllm:time_to_first_token_seconds_bucket (see GPU observability for LLM inference).

Metric 2 — Error rate. Sum of HTTP 5xx + unexpected 4xx + rate of premature finish_reason="length" (responses cut short because the new model does not generate EOS). Gate: error_rate(v2) - error_rate(v1) < 0.01 (1 percentage point). It detects engine crashes, a broken tokenizer, generation problems. Source: vllm:request_success_total{status=...}.

Metric 3 — Quality score (LLM-as-judge). Over a golden set of 200–1,000 representative prompts, v1 and v2 are run offline and a judge model (typically larger: GPT-4 class, Claude, local Llama 405B) scores each pair. Typical gate: mean_score(v2) >= mean_score(v1) - 0.05. This metric is not measured in real time during the canary: judge inference takes 2–8 seconds per sample and does not scale as an inline gate. It is used as an offline pre-promotion gate (before starting the canary) and as a post-mortem over a sample of real traffic captured during the canary. See LLM-as-judge for the mechanics.

Metric 4 — Statistical output drift. For every request that lands on v2 during the canary, embed the response with a lightweight embedding model (e5, BGE) and compare the distribution of v2 embeddings against the distribution of the v1 baseline over the same window. Usable metrics: Wasserstein distance, KL divergence, or more simply, comparing means and variances per dimension. Gate: normalised distance < a calibrated threshold (typically Wasserstein < 0.15). It detects subtle changes in style, length and vocabulary that LLM-as-judge does not capture without going through it too. It is fast: the lightweight embedding takes around 50 ms per response.

Metric 5 — Cost per request. Output tokens per request and kW per request. Gate: cost_per_request(v2) / cost_per_request(v1) < 1.20. It detects new models that generate significantly longer responses or that consume more energy for the same load (quantisation degradation, optimisations failing). Without this gate, an “update” can silently double the bill.

Metric	Type	Measurement latency	Typical gate	Detection
TTFT P95	Quantitative	5 min	`< 110% baseline`	Latency regression
Error rate	Quantitative	1 min	`< 1pp over baseline`	Crashes, generation broken
Quality (LLM-judge)	Offline semantic	hours, over golden set	`> baseline − 0.05`	Functional quality
Statistical drift	Statistical	~5 min	Wasserstein < 0.15	Style, length, vocabulary
Cost per request	Quantitative	5 min	`< 120% baseline`	Economic/energy efficiency

The mechanics on Kubernetes: Argo Rollouts

Argo Rollouts extends the standard Kubernetes Deployment with a new Rollout resource that orchestrates traffic progression and automatic analyses. It integrates with any service mesh (Istio, Linkerd) or ingress controller that supports traffic splitting (NGINX, Traefik, Gateway API).

A minimal example of a 1 → 5 → 25 → 100 % canary with TTFT and error rate gates:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: vllm-llama70b }
spec:
  replicas: 4
  strategy:
    canary:
      canaryService: vllm-llama70b-canary
      stableService: vllm-llama70b-stable
      trafficRouting:
        nginx:
          stableIngress: vllm-llama70b-ingress
      steps:
        - setWeight: 1
        - pause: { duration: 30m }
        - analysis: { templates: [{ templateName: ttft-error-gate }] }
        - setWeight: 5
        - pause: { duration: 1h }
        - analysis: { templates: [{ templateName: ttft-error-gate }] }
        - setWeight: 25
        - pause: { duration: 2h }
        - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
        - setWeight: 50
        - pause: { duration: 4h }
        - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
        - setWeight: 100
  selector: { matchLabels: { app: vllm-llama70b } }
  template:
    metadata: { labels: { app: vllm-llama70b } }
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:v0.10.0
          args: [ --model=/models/llama-70b-fp8-v2 ]   # new version
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: ttft-error-gate }
spec:
  metrics:
    - name: ttft-p95-ratio
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result < 1.10
      provider:
        prometheus:
          address: http://prometheus.observability.svc:9090
          query: |
            histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
            /
            histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))            
    - name: error-rate-diff
      interval: 1m
      count: 5
      failureLimit: 1
      successCondition: result < 0.01
      provider:
        prometheus:
          address: http://prometheus.observability.svc:9090
          query: |
            sum(rate(vllm:request_total{version="v2",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v2"}[5m]))
            -
            sum(rate(vllm:request_total{version="v1",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v1"}[5m]))

If any of the AnalysisTemplates fails, Argo Rollouts backs out automatically: it sets weight=0 on the canary, alerts the operator and keeps v1 serving 100 %. Human operation is reduced to investigating the failure and deciding whether to relaunch or abort.

Flagger offers a more opinionated alternative: weight progression is automatic as a function of metric success instead of a fixed pause; the operator defines a target (maxWeight: 100, stepWeight: 10, metrics: [...]) and Flagger raises or lowers it according to behaviour. Both are mature as of May 2026; the choice is usually dictated by which service mesh is already in the cluster.

The vLLM detail: why there is no “hot swap” of the model

As of May 2026, vLLM v1 does not support hot swapping the model inside the same replica without restarting the engine. The --model argument is evaluated at start-up; changing it requires re-instantiating the LLMEngine, which restarts connections and discards the KV cache. So the rollout unit is the whole replica: you do not do “v1 loads the new model on one of its GPUs” but “a v2 replica is brought up next to a v1 replica and traffic is shared out via the LB”.

TensorRT-LLM with Triton has a similar mechanism: changing the model requires a reload of the Triton backend. SGLang does not support robust hot swap either. The operational consequence: LLM rollout is always going to cost extra GPUs during the window, and the choice between blue-green, canary and shadow is exactly the question of how many extra and for how long.

The three pitfalls specific to LLM rollout

Pitfall 1 — sticky sessions break canary comparability. If the LoadBalancer does session affinity by client IP (common in NGINX and Traefik with loadbalancer.kubernetes.io/session-affinity: ClientIP), a user A always lands on v2 while B always lands on v1. Load distributions, prompt profiles and client behaviour are not random between the two pools, which statistically invalidates any gate comparison. Solution: for a canary, disable session affinity (sessionAffinity: None) or use affinity by random request-id. If the client app requires stickiness for functional reasons (conversational memory persisted in cache), canary is not the right strategy; use blue-green or shadow.

Pitfall 2 — LLM-as-judge is not an inline real-time gate. The temptation to use quality score as a live gate is strong, but the judge’s latency (2–8 s per sample) makes it unfeasible to evaluate more than a 1–2 % sampling of the traffic, and the results arrive minutes late. Operational solutions: (a) offline pre-canary eval over a golden set as a prerequisite for starting (if it fails, the canary does not even begin); (b) during the canary, capture requests + responses from v2 in real time and run the judge asynchronously in a batch job that finishes before the next step; (c) use statistical embedding drift as a fast proxy for inline quality, and reserve the judge for intermediate gates between steps.

Pitfall 3 — SSE streaming complicates shadowing. Classic traffic mirroring (NGINX mirror, Istio MirrorPolicy) is designed for request/response HTTP: it copies the request, lets the primary server answer the client, and duplicates the request to the secondary, discarding the response. With SSE, the secondary’s response is a continuous stream of several seconds, and keeping two streams in parallel doubles the load on the proxy. Solutions: (a) shadow only non-streaming requests (chat without streaming, embeddings, classification, batch eval), (b) shadow the streaming traffic but with a short timeout on the secondary (discard the shadow if it takes longer than 30 s), (c) replace the shadow with a small-weight canary (1 %), which does handle streaming well.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB = 16 GPUs, serving Llama 70B FP8 with TP=4 (4 possible replicas, one per node):

Blue-green: impossible to keep two complete pools of 4 replicas without extra GPUs. Practical solution: blue-green with reduced pools (2 v1 replicas + 2 v2 replicas) during the window, with accepted capacity degradation (half the sustained RPS SLO), or having a parallel cluster (another node) reserved for rollouts.
Canary: feasible. Start with 3 v1 replicas + 1 v2 replica (25 % nominal weight, though the traffic weight also varies). Move to 2 v1 + 2 v2 at 50 %, then 1 v1 + 3 v2, finally 0 v1 + 4 v2.
Shadow: awkward because of the GPU cost. Reserve it for pre-canary validation of major changes, during a short window (4–8 hours) with shadowed traffic limited to a sample (10–20 % of requests, not 100 %).

For clusters of 8 GPU nodes, the three patterns are all sustainable. The operational rule: the rollout budget is typically 25–30 % of the cluster’s sustained capacity, and buying for the peak plus that head-room makes the capacity planning numbers add up.

What we have not covered (upcoming articles)

Multi-region rollouts: how to coordinate a canary when the cluster is geographically distributed.
A/B testing of prompts (not of models): the same model with two different system prompts, measuring conversion.
Embedding rollback: changing the embedding model of a RAG system means re-embedding the whole corpus, so the canary mechanics are different. See RAG corpus curation.
Feature flags for LLM: granularity per tenant or per feature within the same model.
End-to-end continuous deployment: integration with the retrain pipeline so a new adapter is promoted automatically after passing evals.

References

Argo Rollouts project — argoproj.io/argo-rollouts (Rollout and AnalysisTemplate CRDs).
Flagger project — fluxcd.io/flagger (alternative with automatic progression).
Istio — Traffic Mirroring (mirror configurable at VirtualService level).
NGINX Ingress — Canary annotations (nginx.ingress.kubernetes.io/canary-*).
vLLM project — issue tracker on hot model swap (status as of May 2026: in design, not production-ready).
Hou et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024) — reference on goodput metrics applicable to canary gates.
Bürkner et al. — Statistical methods for detecting model drift in production (various articles on Wasserstein and KL in ML monitoring).