Canary on lo0 — Tech Blog

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

Sun, 26 Jul 2026 20:00:00 +0200

This blog has devoted a good part of its 145 articles to what happens inside the inference process and another good part to what happens around it. What is missing is a thread that cuts across all of that and answers a question that takes thirty seconds to ask in an audit and does not get answered in three weeks in production: when the endpoint returns a token, where did that model come from, who signed it, and who proves that the binary running on the GPU is the one that was approved?

This series walks that chain backwards from the only visible point, the endpoint: the control plane and the API contract (this article), the registry and the distribution of the bytes (2/4), the signing and provenance that let you trust them (3/4), and the identity and isolation of whoever runs them (4/4). All of it with projects under the governance of CNCF, LF AI & Data or the Linux Foundation: a chain of trust that depends on a proprietary product is no better than a support contract.

TL;DR

A vLLM Deployment serves tokens perfectly well. What it does not give you is a contract: no model versioning, no rollout by percentage, no scale-to-zero, and no common surface for the thirty non-LLM artifacts of an inference factory.
KServe provides the vocabulary: InferenceService, ServingRuntime/ClusterServingRuntime, storage initializer and storageUri. It has been CNCF incubating since 29 September 2025, after seven years as KFServing and a spell in LF AI & Data. The cadence has accelerated: v0.18.0 on 29 April 2026 and v0.19.0 on 14 June 2026.
The LLM branch is recent and different: the LLMInferenceService CRD (serving.kserve.io/v1alpha1) arrived in v0.16 in November 2025, on top of llm-d, with spec.prefill, spec.worker, spec.parallelism and spec.router.
The Open Inference Protocol (V2) is the contract: /v2/health/*, /v2/models/<m>, /v2/models/<m>/infer, in REST and gRPC, implemented by Triton, OpenVINO Model Server, Seldon MLServer and AMD Inference Server. It is tensor-centric, and the LLM world standardised de facto on the OpenAI API: KServe serves both, with the OpenAI ones under the openai/ prefix.
The honest position: the portability of an LLM today comes from the OpenAI API, not from V2. Accepting the dual track is correct; pretending there is a single standard is not.
KServe is overkill if you serve one model on one node. It starts to pay off from the fifth heterogeneous artifact onwards, and it competes on a different axis from the LLM operators: contract and lifecycle versus engine and topology.

The analogy: the station master and the track gauge

An inference factory is like a railway station with several lines.

The station master does not drive any train. His job is the service timetable: which formation leaves from which platform, how many units are coupled each hour, when an engine is pulled out for maintenance and, when new rolling stock arrives, how to divert ten per cent of the passengers to the test train before committing the whole service. That is the control plane. KServe does not generate a single token, but it decides what runs, with what configuration, with how many replicas and with what traffic split. The trains are the engines (vLLM, SGLang, TensorRT-LLM, Triton, TEI), and a competent station master does not marry a manufacturer.

And then there is the track gauge: as long as everybody builds to the same one, any locomotive rolls on any line and changing supplier is a commercial decision, not a civil engineering project. That gauge is the Open Inference Protocol, the contract that makes a client written against Triton work against OpenVINO Model Server without touching a line.

The analogy has a twist that needs no explanation in Spain: the agreed gauge is not the gauge the new rolling stock runs on. The conventional network was built to Iberian gauge and the high-speed one to standard gauge, and the practical solution was not to convert the network but to install gauge changers at the boundary points. The same thing has happened in inference: V2 is the agreed gauge and the OpenAI API is the gauge everything generative runs on. What sustains interoperability is the changer, not a single standard. We will come back to it.

A `Deployment` serves tokens, but offers no contract

In vLLM on Kubernetes we built a service with Deployment, Service, HPA and an initContainer that pulls the weights down. It works, and for one model on one node it is the right answer. What it does not give you, and cannot give you without reinventing it every time:

A stable API contract. The endpoint is “whatever the image you put there exposes”: moving from vLLM to TensorRT-LLM changes routes, error format and metric names, and the clients find out in production.

Versioning with semantics. The model is a path inside a volume; knowing which version a pod is serving requires kubectl exec. There is no declarative field that says “this service serves artifact X” and can be audited from outside.

Progressive model rollout. A RollingUpdate replaces pods; it does not split traffic between two versions by percentage. For that you need an L7 gateway in front, like the one in canary, blue-green and shadow, or a control plane that emits it.

Scale-to-zero. The HPA does not go below one replica: for a model queried twelve times a day, that is an 80 GB GPU tied up twenty-four hours.

And, above all, a common surface for what is not an LLM. A real factory does not serve three LLMs: it serves two or three LLMs, four embedding models, a couple of rerankers, ticket classifiers, a PII detector, an OCR, industrial vision and three scikit-learn artifacts that nobody remembers training. Fifteen or thirty artifacts, each with its own runtime and its own way of saying “I am ready”. The cost is not in serving the first one: it is in making sure the thirtieth does not cost thirty times what the first one did.

KServe: from KFServing to a CNCF incubating project

KServe was born in 2019 as KFServing, inside Kubeflow, in a collaboration between Google, IBM, Bloomberg, NVIDIA and Seldon. In February 2022 it was donated to the LF AI & Data Foundation; in September 2022 it split from Kubeflow and was renamed. The relevant jump: on 29 September 2025 the CNCF accepted it directly at incubating level, without going through sandbox, with a public announcement on 11 November. The figures it declared then, more than 300 contributors, 19 maintainers, more than 30 adopting organisations (Bloomberg, Red Hat, Cloudera, Nutanix, SAP, NVIDIA), come from an interested party, but the list is verifiable and the 625 contributing organisations on the CNCF dashboard support the thesis of governance not captured by a vendor.

On cadence you have to be precise, because the cliché of “KServe is slow” no longer describes reality:

Version	Date	What it brought
v0.15.0	31 March 2025	`LocalModelCache`, multi-node inference, OpenAI-compatible embeddings API, Gateway API in raw mode, KEDA integration
v0.16.0	November 2025	`LLMInferenceService`; rename of `RawDeployment` to `Standard` and of `Serverless` to `KNative`; progressive rollout in raw mode
v0.18.0	29 April 2026	WVA + KEDA/HPA autoscaling for LLMISvc, `/v1/responses` route (OpenAI Responses API), `restricted` Pod Security Standards by default on LLMISvc, namespace-scoped `ModelCache`
v0.19.0	14 June 2026	Routing by model name, static LoRA adapters, balancing across heterogeneous GPUs, dual REST/gRPC routing in Standard mode, automatic migration to `InferencePool` v1

Two major releases in under two months, with the bulk of the changelog in the LLM branch. The project is not slow: it has pivoted from classic predictive serving to generative serving. The consequences run in both directions.

The negative one is called ModelMesh. It was the high-density mode (hundreds of small models sharing processes, with dynamic load and unload) and it was, on paper, the natural answer to the problem of thirty heterogeneous artifacts. Its last release was v0.12.0 on 9 July 2023, and the kserve/modelmesh-serving repository was archived read-only on 14 April 2026. Red Hat published specific guidance for migrating to Standard mode. If somebody proposes ModelMesh in a design today, it is a sign that the documentation they read is three years old.

Anatomy of the control plane

`InferenceService`: predictor, transformer, explainer

The core CRD is InferenceService (serving.kserve.io/v1beta1), and its contribution, the one that became the common vocabulary of the field, is to break an inference service down into three components each with its own contract:

predictor: mandatory. It declares modelFormat, optionally runtime, the storageUri the weights come from and the protocolVersion it exposes.
transformer: optional. Pre- and post-processing. That it is a separate component, rather than code inside the server, is what allows it to be scaled separately (it is usually CPU-bound while the predictor is GPU-bound) and versioned separately.
explainer: optional, and the one that has aged worst: the V2 protocol does not support the explain endpoint, which only exists in V1. Anyone who needs explainability with a standard contract has a real gap here.

All three share ComponentExtensionSpec: minReplicas, maxReplicas, scaleTarget, scaleMetric, containerConcurrency, timeout and canaryTrafficPercent.

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

A ServingRuntime is a pod template that knows how to serve one or several formats. It declares supportedModelFormats (with name, version, autoSelect and priority), protocolVersions (v1, v2) and containers, with args parameterised by templates in the {{.Name}} style that are substituted with metadata from the InferenceService.

Selection is what makes this scale. If the InferenceService names a runtime, the controller looks for it first in the namespace and then at cluster level; if it does not name one, it picks among those with autoSelect: true for that modelFormat, breaking ties by priority. That is what lets a platform publish a curated catalogue (“here, huggingface is served with this vLLM and these flags”) without the product teams having to know what --gpu-memory-utilization is. In multi-tenant setups, the ClusterServingRuntimes belong to the platform and the InferenceServices to the teams: the cleanest separation-of-duties line KServe offers.

The storage initializer and `storageUri`

The storageUri field triggers the storage initializer: an init container that downloads the artifact to /mnt/models before the server starts. Schemes supported as of July 2026: s3://, gs://, https://, pvc://, hf://, oci:// and git repositories; credentials are attached via a ServiceAccount with annotated secrets. That oci:// connects with the second article of the series: the weights can live in the same registry as the images, with the same access control and the same signing capability.

The deployment modes

This is where nearly everything operational is decided. It is selected with the serving.kserve.io/deploymentMode annotation.

	KNative (formerly Serverless)	Standard (formerly RawDeployment)	ModelMesh
Resources emitted	Knative `Service` + `Revision`, `VirtualService` with Istio	`Deployment`, `Service`, `HPA`, Gateway API `HTTPRoute`	Its own high-density runtime
Dependencies	Knative Serving + Istio (or an alternative gateway)	Kubernetes 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1 and a controller	Its own components
Autoscaling	KPA (concurrency / RPS)	HPA or KEDA	—
Scale-to-zero	Yes, with an activator that holds the request	No for HTTP requests	—
`canaryTrafficPercent`	Yes, by revision weights in Istio	Silently ignored (issue #5335, open since 2 April 2026)	—
Sidecars in the pod	`queue-proxy` (+ `istio-proxy`)	None	—
Status as of July 2026	Current	Current and the default mode of the LLM branch	Archived on 14 April 2026

That table has an uncomfortable reading: the two most quoted KServe capabilities, scale-to-zero and canary by percentage, live in the mode that drags Knative and Istio along, while the mode the project has pivoted towards is the other one.

The LLM branch: `LLMInferenceService`

Since v0.16 KServe has a separate CRD for LLMs: LLMInferenceService, in serving.kserve.io/v1alpha1. It is not an extension of the classic InferenceService and does not share its API version, and confusing the two is the first mistake made by anyone arriving from old documentation.

It is built on llm-d, the project that Red Hat, Google, IBM, CoreWeave and NVIDIA donated to the CNCF. The division of roles the project itself declares is clean: KServe is the control plane (lifecycle, scaling and operational governance) and llm-d contributes distributed scheduling, with GPU utilisation, queue depth, cache residency and SLA. In the analogy: llm-d decides which line each train takes; KServe decides which trains exist and when they are withdrawn.

The fields that matter: spec.model (URI, invocation name, scheduling criticality and LoRA adapters, with static adapter reconciliation from v0.19, which connects with multi-LoRA serving); spec.template (single node, or the decode pool); spec.worker (multi-node, which triggers LeaderWorkerSet); spec.prefill (the disaggregation we analysed in disaggregated serving turned into a declarative field); spec.parallelism (tensor, data and expert parallelism); and spec.router (gateway, route, scheduler), where KServe leans on the Gateway API Inference Extension, v1.5.0 of 19 April 2026, already GA, with automatic migration to InferencePool v1 from v0.19.

The endpoint it exposes is OpenAI-compatible: /v1/chat/completions with streaming, and since v0.18 also /v1/responses. That detail underpins the next section: the LLM branch of KServe does not speak V2, it speaks OpenAI.

For multi-node there is the kserve-huggingfaceserver-multinode runtime, based on vLLM over Ray, with workerSpec.tensorParallelSize and workerSpec.pipelineParallelSize. Its restrictions are hard: it requires a ReadWriteMany PVC, it only works in Standard mode, it does not support autoscaling and it requires exactly one head pod. A reference 4×H100 SXM node with NVLink needs none of this, because a 70B fits with tensor parallel 4 on a single node; you need it from there upwards.

The Open Inference Protocol: the contract itself

The routes and the payload

The Open Inference Protocol (historically the “V2 Inference Protocol”) lives in kserve/open-inference-protocol, with REST and gRPC specifications and versioning under SemVer 2.0. Its HTTP surface is deliberately small:

Resource	Verb	Route
Server metadata	GET	`/v2`
Liveness	GET	`/v2/health/live`
Server readiness	GET	`/v2/health/ready`
Model metadata	GET	`/v2/models/<model>[/versions/<v>]`
Model readiness	GET	`/v2/models/<model>[/versions/<v>]/ready`
Inference	POST	`/v2/models/<model>[/versions/<v>]/infer`

The payload is tensorial: id, a list of inputs (each with name, shape, datatype and data) and optionally the desired outputs. The types are BOOL, signed and unsigned integers from 8 to 64 bits, 16-, 32- and 64-bit floating point, and BYTES. In gRPC the same contract is GRPCInferenceService, with six RPCs: ServerLive, ServerReady, ModelReady, ServerMetadata, ModelMetadata and ModelInfer. For high-volume embeddings, the serialisation difference is not cosmetic.

Who actually implements it

The specification repository declares KServe, NVIDIA Triton, Seldon MLServer and Core v2, OpenVINO Model Server, AMD Inference Server and TorchServe as adopters. Not all entries are worth the same as of July 2026:

Triton implements it in a documented way (“Triton exposes HTTP/REST and gRPC endpoints based on the standard inference protocols proposed by the KServe project”), plus its own extensions. It is the de facto reference implementation.
OpenVINO Model Server exposes the KServe API over gRPC and REST, the TensorFlow Serving one and OpenAI-compatible endpoints: the clearest example of a server that speaks both track gauges.
TorchServe forces you to be honest. Its official documentation carries the warning: “This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches.” An adopter on zero maintenance is not an adopter: it is a row nobody has updated.

On the health of the standard, the most eloquent data point is not in any press release: the specification repository does not have a single published release, has accumulated something like thirty commits and seventy-five stars, and its governance consists of monthly meetings and a Slack channel. There is also a “vendor neutral” fork at open-inference/open-inference-protocol with more commits than the original and no traction whatsoever. V2 is a stable contract, widely implemented and barely evolving. For a serialisation format that can be a virtue; for a standard aspiring to cover the generative world, it is a death sentence.

The `generate` extension and why it did not win

The standard did try to cover generation: the specification contains a generate_rest.yaml with two routes, /v2/models/<m>/versions/<v>/generate and .../generate_stream, and a minimalist schema: a request with text_input and parameters (temperature, top_p, max_tokens defaulting to 20, stop, details); a response with text_output, model_name, model_version and, if requested, finish_reason and logprobs; streaming over server-sent events. It is clean, it is enough for a simple completion and nobody uses it: the KServe generative inference documentation, as of July 2026, lists four endpoints, and none of them is generate:

Task	Endpoint in KServe
Text generation	`openai/v1/completions`
Chat	`openai/v1/chat/completions`
Embeddings	`openai/v1/embeddings`
Reranking	`openai/v1/rerank`

The openai/ prefix exists, according to the documentation itself, “to avoid confusion with the V1 API protocol”, and it is changed with KSERVE_OPENAI_ROUTE_PREFIX. It is plumbing that says a lot: the in-house standard was relegated to a defensive prefix while the outside one takes all the traffic.

The tension, unvarnished

V2 solved the problem of 2021, and it solved it well. A tensorial, typed contract with metadata and health checks, in REST and gRPC, is what you need to serve a classifier, a vision model, a ranker or an embedding, and it is still the right answer for that half of the catalogue: swapping an embedding server from MLServer to OpenVINO Model Server without touching clients is the track gauge doing its job.

The LLM world standardised somewhere else, not by committee, but because the entire client ecosystem (SDKs, agent frameworks, gateways, evaluation) was written against the OpenAI API, and any engine that wanted to be used had to speak it. What that means for an architecture:

There is not one contract: there are two, and both are legitimate. Designing as if there were one does violence to half the catalogue.
LLM portability is superficial. It covers chat and completion well; as soon as you use detailed logprobs, guided decoding, priority headers, prefix caching parameters or tool calling with specific schemas, you leave the common subset and get tied back to the engine. It is the warning from structured output: what is portable is the envelope, not always the contents.
The gauge changer is the gateway. What makes the set switchable is the translation and routing layer of choosing an OSS LLM inference gateway and the router as an L7 gateway, not a single standard.
Crossing over loses the versioning. V2 has /versions/<v> in the route; the OpenAI API only has the model field. The version has to be encoded in the name (asistente-70b-2026-07) or managed in the control plane. That is the difference between being able to audit which version answered and not being able to.

Four manifests you can actually apply

Platform runtime for the blog’s reference node, 4×H100 SXM 80 GB with NVLink, with tensor parallel 4:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
 name: vllm-h100-tp4
spec:
 annotations:
 prometheus.kserve.io/port: "8080"
 prometheus.kserve.io/path: "/metrics"
 supportedModelFormats:
 - name: huggingface
 version: "1"
 autoSelect: true
 priority: 10
 protocolVersions:
 - v2
 containers:
 - name: kserve-container
 # ALWAYS pin by digest in production; the tag is only for readability
 image: vllm/vllm-openai:v0.11.0
 command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
 args:
 - --port=8080
 - --model=/mnt/models
 - --served-model-name={{.Name}}
 - --tensor-parallel-size=4
 - --max-model-len=32768
 - --gpu-memory-utilization=0.92
 - --enable-prefix-caching
 - --kv-cache-dtype=fp8
 resources:
 requests: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 limits: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 # the probe that stops the pod dying while the model loads
 startupProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 15
 failureThreshold: 40 # up to 10 minutes of loading
 readinessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 10
 livenessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 30

The InferenceService that consumes it, with KEDA on vLLM metrics:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 serving.kserve.io/deploymentMode: "Standard"
 serving.kserve.io/autoscalerClass: "keda"
spec:
 predictor:
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 # the OCI registry from article 2/4 of this series
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-07"
 autoScaling:
 metrics:
 - type: External
 external:
 metric:
 backend: "prometheus"
 serverAddress: "http://prometheus.monitoring.svc.cluster.local:9090"
 query: 'sum(vllm:num_requests_waiting{model_name="asistente-70b"})'
 target:
 type: Value
 value: "8"

The canary, with its small print:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 # WATCH OUT: canaryTrafficPercent only takes effect in KNative mode.
 # In Standard mode it is silently ignored (issue kserve/kserve#5335, open).
 serving.kserve.io/deploymentMode: "KNative"
spec:
 predictor:
 canaryTrafficPercent: 10
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-08"

KServe keeps three references in the status (LatestReadyRevision, LatestRolledoutRevision and PreviousRolledoutRevision): promoting means setting the percentage to 100, reverting means pinning traffic on the previous one, and a revision that does not become ready automatically receives zero traffic.

And the two calls. Against the V2 contract, for a classifier served with MLServer or Triton:

# metadata: which model it is, what inputs it expects, what types
curl -s http://clasificador.inferencia.example/v2/models/clasificador-tickets | jq .

# readiness of the specific model, not of the server
curl -s -o /dev/null -w '%{http_code}\n' \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/ready

curl -s -X POST \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/infer \
 -H 'Content-Type: application/json' \
 -d '{ "id": "req-4711",
 "inputs": [ { "name": "input-0", "shape": [1, 4], "datatype": "FP32",
 "data": [5.1, 3.5, 1.4, 0.2] } ],
 "outputs": [ { "name": "output-0" } ] }'

Against the same control plane, but on the OpenAI-compatible surface of the LLM:

curl -s -N -X POST \
 http://asistente-70b.inferencia.example/openai/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{ "model": "asistente-70b",
 "messages": [ {"role": "user",
 "content": "Summarise last night'\''s incident report."} ],
 "max_tokens": 256,
 "stream": true }'

Two contracts and two payload shapes, with the same InferenceService as the unit of governance. That is the value proposition of a control plane: the contract is chosen by the model and the management is single.

Autoscaling and the problem of zero

Three autoscalers coexist, and choosing the wrong one is the most common cause of disappointment. KPA, in KNative mode, scales by concurrency or RPS and is the only one that goes down to zero in a usable way, because Knative provides an activator that holds the request while the pod starts; its metric does not distinguish a 30-token request from an 8,000-token one. HPA, in Standard, scales by CPU or memory: for LLMs it is useless, because the CPU of a saturated vLLM pod and that of an idle one look too much alike. KEDA, in Standard, is the right option, the one we develop in autoscaling with KEDA: it is enabled with serving.kserve.io/autoscalerClass: "keda" and autoScaling.metrics accepts external metrics against Prometheus (vllm:num_requests_running or vllm:num_requests_waiting) or, with the OpenTelemetry add-on, pushed metrics. Since v0.18 there is also Workload Variant Autoscaling, which uses HPA or KEDA as a backend.

That leaves the serious problem, which no autoscaler solves: zero. A 70B model in fp8 is on the order of 70 GB to move into HBM, and that path, from disk to HBM, is measured in minutes: the first request after idling takes that long, or it fails.

Worse: the installation documentation is explicit as of July 2026: “Scale from Zero is currently not supported in Standard mode for HTTP requests”. The mode the project has pivoted towards, and the only one that supports KEDA, does not know how to come up from zero on an HTTP request: KEDA can go down to zero (default cooldown of 300 seconds), but the way back requires an activator that Standard does not have. Anyone who wants real zero with HTTP activation still needs Knative.

The levers, in order of effectiveness: do not scale the hot model to zero (minReplicas: 1 for anything with an SLA; zero is for the long tail); attack the load time, the only solution that is not a patch, with the techniques from speeding up cold start; use LocalModelCache, available since v0.15 and extended to LLMInferenceService in v0.19, to preload artifacts into a node PVC; and set correct probes, which is what comes next.

Operational traps

The probe that kills the pod while the model loads. The most frequent and the most expensive to diagnose, because the symptom is a CrashLoopBackOff with no error in the server logs. Until early 2026 the KServe vLLM templates used initialDelaySeconds of 120 to 300 seconds on liveness, with the result described in issue #5062 (12 February 2026, now resolved): the fast models waited too long and the large ones died before finishing loading. The fix applied: a startupProbe of up to ten minutes and probes with no initial delay. A legacy runtime has to be reviewed by hand. In KNative mode there is a worse variant (issue #3795, closed as not planned): what dies is not the model container but the queue-proxy, which does not respond OK within the deadline and takes the whole pod down with it.

canaryTrafficPercent silently ignored. It deserves its own trap because there is no error and no condition in the status to give it away: in Standard mode the field does nothing. The team believes it is serving 10 % with the new version and it is serving 100 %. While issue #5335 remains open, in Standard the canary is done with HTTPRoute weights by hand, or in the gateway.

Canary and scarce GPUs do not get along. A 10 % canary of a 70B with tensor parallel 4 costs a whole 4-GPU node, not 10 % of the resources, because the minimum granularity is a complete replica. During the window there are two versions occupying twice the hardware, or capacity has to be taken from the stable one. That is the difference between a microservice canary and a model canary, and the reason a shadow over duplicated traffic often pays off better.

storageUri and credentials. Storage initializer failures are always the same ones: a ServiceAccount without the annotated secret, a KMS-encrypted bucket with a role that has no decryption permission, an S3-compatible but not exactly S3 endpoint, or intermittent 403s, with a specific fix in v0.18. It is diagnosed in the init container logs, not in the server’s. And one standing recommendation: no downloading over https:// from the internet in production; that is precisely the problem of article 2/4.

The model size against the emptyDir. By default the artifact lands in an ephemeral volume on the node’s disk. Seventy gigabytes per replica fill a system disk with surprising ease, and then the kubelet evicts pods for disk pressure, including ones that have nothing to do with it. You have to set sizeLimit or use a dedicated pvc://. For multi-node the restriction is harder, with a mandatory ReadWriteMany PVC, which rules out a good part of local block storage and pushes you towards shared file systems, with the consequences covered in storage for AI.

The dependency on Knative and Istio. The real cost of KNative mode: two more control planes to version and correlate with Kubernetes, and two sidecars, queue-proxy and istio-proxy, in every pod, with their CPU and memory consumption and their network hop per request. On a platform that already has a mesh it is a marginal cost; on a cluster dedicated to inference it has to be justified by the capabilities you are actually going to use.

Observability: the name of the metric. The ServingRuntime declares prometheus.kserve.io/port and /path annotations, but the metrics that come out are the engine’s, not KServe’s: vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:gpu_cache_usage_perc. The control plane does not normalise names across runtimes, so a dashboard written against vLLM is no use for Triton, and correlating with DCGM is still your own work. v0.19 raises the HPA/KEDA scaling status into the service conditions and emits events on readiness transitions: exactly what was needed to alert on the control plane and not only on the engine.

Decision map: when KServe, and when it is overkill

The useful comparison is not “KServe versus OME, vLLM Production Stack, Dynamo or llm-d”. Those four are covered in LLM inference operators and compete on a different axis:

Axis	What gets decided	Who competes
Contract and control plane	Which CRD describes a service, which API clients see, how the model is versioned, how the rollout is done, how LLM and non-LLM coexist	KServe, Seldon Core v2
Engine and topology	Which engine runs, how prefill/decode is disaggregated, how routing is done with cache awareness, how multi-node tensor parallel is done	vLLM Production Stack, OME, NVIDIA Dynamo, llm-d

The axes are orthogonal, and LLMInferenceService proves it: KServe did not reimplement distributed scheduling, it leaned on llm-d. Choosing KServe is not choosing instead of an LLM operator: it is choosing the contract layer that operator lives on.

Choose KServe when the catalogue is heterogeneous, LLMs plus embeddings, rerankers, classifiers and vision, which is its decisive advantage and the one none of the four operators has: a service like the one in TEI in production fits into the same vocabulary as the LLM. Also when you need a contract decoupled from the engine because you have clients you do not control; when you need declarative, auditable governance (which artifact, from which origin and with which runtime, in an object versionable in git and subject to admission policies), the piece that makes the rest of this series possible; and when you want a catalogue of platform runtimes separated from the teams’ services.

KServe is overkill when you serve one model, on one node, with up to three replicas: the Deployment from vLLM on Kubernetes is the right answer. When your workload is LLM only and the priority is performance per GPU, because there the value is in the engine. When you are going to use neither scale-to-zero nor canary by revision, because what remains is a Deployment with more YAML. And when you already have an LLM operator and only serve LLMs: putting KServe on top for completeness is adding CRDs without adding decisions.

For an inference factory

First: separate the runtime catalogue from the service catalogue, and do it on day one. The ClusterServingRuntimes belong to the platform; the InferenceServices, to the teams. That boundary turns “serving the thirtieth model” into a five-line change instead of a negotiation about vLLM flags, and it fits with GitOps with Flux: the runtime is reviewed as platform code, the service as product configuration.

Second: accept the two contracts and put the gauge changer where it belongs. V2 for the tensorial world, the OpenAI API for the generative one, and the translation in the gateway, not in every client. And since the OpenAI API does not version in the route, encode the version in the model name and make that name the same identifier that appears in the registry, in the signature and in the traces. That is what makes a request traceable back to the artifact.

Third: measure start-up before promising elasticity. The time from the pod being scheduled to the readiness probe going green, with the real model and from the real origin, decides whether scale-to-zero is a FinOps lever or an incident machine. That number, and not the price of the GPU-hour, is the one to take into the cost conversation.

And here this piece falls short on purpose. KServe gives you the contract, the lifecycle and the rollout, but its whole trust model hangs off a text string: storageUri. The storage initializer downloads whatever is at that destination and puts it in front of the GPU without asking anything else; nothing in the InferenceService guarantees that the artifact is the one that was approved, that it has not changed since the last audit, or that the origin is legitimate. The next article goes there: where the model bytes come from, and why an OCI registry with ORAS, Harbor, MLflow or the Kubeflow Model Registry is the difference between distributing models and scattering them.

Sources

CNCF, KServe becomes a CNCF incubating project (11 de noviembre de 2025) — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
CNCF, KServe project page (incubating, aceptado el 29 de septiembre de 2025) — https://www.cncf.io/projects/kserve/
CNCF TOC, Project Moving Levels Checklist: KServe joining CNCF at Incubation level — https://github.com/cncf/toc/issues/1905
KServe, Releases (v0.15.0, v0.16.0, v0.18.0, v0.19.0) — https://github.com/kserve/kserve/releases
KServe, Inference Protocol V2 (Open Inference Protocol) — https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol
KServe, open-inference-protocol (especificación, adoptantes, gobernanza) — https://github.com/kserve/open-inference-protocol
KServe, open-inference-protocol — generate_rest.yaml — https://github.com/kserve/open-inference-protocol/blob/main/specification/protocol/generate_rest.yaml
open-inference, Vendor neutral fork of kserve/open-inference-protocol — https://github.com/open-inference/open-inference-protocol
KServe, Generative Inference — Runtime Overview (endpoints con prefijo openai/) — https://kserve.github.io/website/docs/model-serving/generative-inference/overview
KServe, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
KServe, Cloud-Native AI Inference at Scale using KServe and llm-d (5 de marzo de 2026) — https://kserve.github.io/website/blog/cloud-native-ai-inference-kserve-llm-d
KServe, Serving Runtime (ServingRuntime y ClusterServingRuntime) — https://kserve.github.io/website/docs/concepts/resources/servingruntime
KServe, Kubernetes Deployment Installation Guide (K8s 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1, sin scale-from-zero en Standard) — https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment
KServe, Canary Rollout Strategy — https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary
KServe, Autoscaling with KEDA — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/keda-autoscaler
KServe, Multi-node/Multi-GPU Inference (PVC RWX, sin autoescalado) — https://kserve.github.io/website/docs/model-serving/generative-inference/multi-node
KServe, Storage Options for Model Artifacts — https://kserve.github.io/website/docs/model-serving/storage/overview
KServe, issue #5335 — Support canaryTrafficPercent for RawDeployment mode via Gateway API HTTPRoute weights (abierta, 2 de abril de 2026) — https://github.com/kserve/kserve/issues/5335
KServe, issue #5062 — Add startupProbe to vLLM main containers (12 de febrero de 2026) — https://github.com/kserve/kserve/issues/5062
KServe, modelmesh-serving (última release v0.12.0 de julio de 2023; repositorio archivado el 14 de abril de 2026) — https://github.com/kserve/modelmesh-serving
Red Hat, Converting ModelMesh and Serverless InferenceServices to RawDeployment (Standard) Mode — https://access.redhat.com/articles/7134025
NVIDIA, Triton Inference Server — Inference Protocols and APIs — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inference_protocols.html
OpenVINO, What is OpenVINO Model Server (API KServe, TFS y endpoints compatibles con OpenAI) — https://docs.openvino.ai/2026/model-server/ovms_what_is_openvino_model_server.html
PyTorch, TorchServe — Notice: Limited Maintenance — https://docs.pytorch.org/serve/
Kubernetes SIG Network, Gateway API Inference Extension (v1.5.0, 19 de abril de 2026) — https://github.com/kubernetes-sigs/gateway-api-inference-extension

Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

Mon, 01 Jun 2026 16:30:00 +0200

This post complements those on LLM autoscaling on Kubernetes (the autoscaler coexists with the rollout and must respect it), GPU observability for LLM inference (the metrics that act as gates come from there), Evals for LLMs (the eval that decides whether the new model is ready), LLM-as-judge (the technique that puts the “quality” into the canary gate) and Retrain: closing the loop (the previous step that the new model comes out of).

TL;DR

Promoting a new version of an LLM model to the production cluster without cutting traffic or breaking the SLO demands progressive deployment. The three canonical strategies, blue-green, canary and shadow, answer different questions and have different costs. Blue-green: a complete new pool brought up in parallel, atomic load balancer switch. Instant rollback (point back at the old pool); it demands twice the GPUs during the window. Canary: traffic is shared progressively between the old version and the new one (1 % → 5 % → 25 % → 100 %), measuring regression gates at each step; it consumes incrementally less hardware but exposes real users to the new model from the first percentage point. Shadow / mirror: the old model serves 100 % of the real traffic to the client and, in parallel, a copy of every request goes to the new model without returning its response to the user; it isolates you from quality risk but spends the new model’s GPU on responses nobody consumes, and it does not work well with long SSE streaming. The choice depends on three factors: available GPU budget, service criticality and the availability of a fast automatic eval. The five regression metrics any LLM canary should gate on are: TTFT P95, error rate (HTTP 5xx + premature finish_reason="length"), quality score with LLM-as-judge over a golden set, statistical drift of output embeddings (Wasserstein or KL against the baseline distribution) and cost per request (tokens/s and kW/request). On Kubernetes, Argo Rollouts manages traffic and the AnalysisTemplates as automatic gates; Flagger is the more opinionated alternative. vLLM v1 does not support hot model swap robustly as of May 2026, so the rollout unit is the whole replica (deployment v2 alongside deployment v1). The three specific pitfalls: LB sticky sessions break the statistical comparability of the canary (a client A always lands on the new one, B on the old one, so the populations are not equivalent); semantic eval with LLM-as-judge takes 2–8 seconds per sample and is no use as a real-time gate (it is used in post-analysis or offline before promotion); SSE streaming complicates shadowing because the new model’s response has to be discarded without affecting the old one’s. This post includes a minimal Argo Rollouts manifest applicable to a generic cluster with the NVIDIA GPU Operator.

You are here: DEPLOY (and the transition to RETRAIN)

A new model does not appear in the cluster by magic: it comes from the retrain loop or from a weights provider update. The step between “I have an artifact that passed offline eval” and “it is serving 100 % of the traffic” is exactly this post.

The analogy: opening night of a play

A theatre company is about to premiere a new version of a play that has been running successfully for a year. The company knows several hard facts: the current audience pays for a consistent experience; a bad opening damages the business for months; but premiering nothing leaves the company obsolete against the competition.

The three opening routes the management can choose are the same three of the LLM rollout.

Dress rehearsal behind closed doors (shadow / mirror). The new actors perform the whole play to an empty theatre. There is no audience; nobody buys a ticket. Three full run-throughs are enough to check continuity, timing and the chemistry of the cast. It is expensive because there are salaries and theatre rent, but it does not expose the audience to risk. Useful when the new cast is untested and the director wants to see how it holds up over a complete performance before selling it. In LLM terms: the new model processes every real request in parallel with the old one but its responses are discarded; you spend the new model’s GPU on responses nobody sees.

Cast by performance, alternating (canary). Instead of changing the whole cast at once, Thursday performances belong to the new cast, Friday ones to the old one, Saturday ones half and half. The management reads the visitors’ book comments and the audience numbers performance by performance, deciding after two weeks whether to promote the new cast to permanent or withdraw it. Cheaper than the dress rehearsal because the performances still sell tickets, but it exposes a real audience to risk from the first Thursday. In LLM terms: traffic is shared progressively between the old version and the new one, measuring gates at each step.

Double company with an atomic switch (blue-green). The company hires the new cast, prepares it for a month behind closed doors, and one Saturday announces: “from the next opening night every performance is with the new cast”. If the first performance goes badly, they go back to the old cast in 24 hours, but during that month of preparation double salaries are paid to both companies. In LLM terms: two complete pools of the same size, instant LB switch from one to the other, rollback in seconds if the metrics break.

The analogy also supports the decision: the choice depends on how critical the play is for the business (criticality of the LLM service), how much budget there is to sustain two casts at once (GPU budget), and how much confidence there is in the new cast from the chamber rehearsals (offline eval prior to the canary).

The three strategies in detail

Blue-green

The operator keeps two replica pools identical in size: the blue one (production version v1) and the green one (candidate version v2). When v2 is validated offline (eval passed, smoke tests), the LoadBalancer switch redirects 100 % of the traffic from blue to green in a single step. If the SLO metrics break, the switch goes back in seconds.

Cost: 2× GPUs for the whole window (preparing v2 + the post-switch observation window). For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), preparing the blue-green needs 16 extra GPUs for 1–3 days.

Risk: the switch is atomic. If v2 has a problem that did not show up in offline eval but does show up at scale (for example, edge cases that only appear at 200 RPS), 100 % of users notice it at once. Rollback is instant, but the requests of the first minute after the switch were already affected. So blue-green is preferable when you have high confidence in v2 (a minor change: same architecture, same format, just a new version of the weights) and you prioritise immediate rollback over gradual exposure.

Canary

The operator deploys v2 with a small number of replicas (typically 1) next to the v1 pool. The LoadBalancer shares the traffic out progressively on a schedule: 1 % for 30 minutes → 5 % for 1 hour → 25 % for 2 hours → 50 % for 4 hours → 100 %. Between each step, an analysis gate evaluates regression metrics over the traffic that is already landing on v2. If the gate fails, the rollback withdraws traffic from v2 automatically and leaves v1 serving everything.

Cost: incremental. At the start (1 % of traffic) one v2 replica is enough; at 50 % you need half as many v2 replicas as v1 has in total. Peak extra GPU during the canary: about 30–50 % above baseline.

Risk: real users are seeing v2 from the first 1 %. If v2 produces responses with degraded quality but normal TTFT and error rate, the affected users perceive the degradation without the gate detecting it (unless the gate includes quality drift, which is slow). So canary is preferable when you have medium confidence in v2 (a significant change: different architecture or training) and you accept that a low percentage of users will be guinea pigs.

Shadow / mirror

The LoadBalancer sends 100 % of the real traffic to v1 (which answers the client) and duplicates every request towards v2 (whose response is discarded or stored for offline analysis). The client never sees v2; it is never exposed to the risk.

Cost: 100 % extra compute for v2 with no direct user value for the whole shadow window. For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), a shadow of the same size consumes 16 extra GPUs full time.

Risk: shadow is the safest for the user. But it has two serious limitations: (a) if v2 has a bottleneck that makes the copied request to the shadow take a long time, the shadowing proxy can consume LB connections; it must be out-of-band (asynchronous); (b) long SSE streaming complicates mirroring because you have to keep two parallel streams and discard one while the other flows to the client. Common pattern: shadow only non-streaming requests (short completions, classification), manual offline eval of the streaming requests.

The five regression metrics that act as a gate

Without automatic gates, the “canary” is just a nice name for “manual rollout with a variable percentage”. The gates are the piece that turns the canary into a defensible operation.

Metric 1 — TTFT P95. Comparison of the new model’s P95 against the baseline (v1) P95 in 5-minute windows. Gate: ttft_p95(v2) / ttft_p95(v1) < 1.10. It detects prefill latency regressions (a slower new model) or engine problems (suboptimal config). Source: vllm:time_to_first_token_seconds_bucket (see GPU observability for LLM inference).

Metric 2 — Error rate. Sum of HTTP 5xx + unexpected 4xx + rate of premature finish_reason="length" (responses cut short because the new model does not generate EOS). Gate: error_rate(v2) - error_rate(v1) < 0.01 (1 percentage point). It detects engine crashes, a broken tokenizer, generation problems. Source: vllm:request_success_total{status=...}.

Metric 3 — Quality score (LLM-as-judge). Over a golden set of 200–1,000 representative prompts, v1 and v2 are run offline and a judge model (typically larger: GPT-4 class, Claude, local Llama 405B) scores each pair. Typical gate: mean_score(v2) >= mean_score(v1) - 0.05. This metric is not measured in real time during the canary: judge inference takes 2–8 seconds per sample and does not scale as an inline gate. It is used as an offline pre-promotion gate (before starting the canary) and as a post-mortem over a sample of real traffic captured during the canary. See LLM-as-judge for the mechanics.

Metric 4 — Statistical output drift. For every request that lands on v2 during the canary, embed the response with a lightweight embedding model (e5, BGE) and compare the distribution of v2 embeddings against the distribution of the v1 baseline over the same window. Usable metrics: Wasserstein distance, KL divergence, or more simply, comparing means and variances per dimension. Gate: normalised distance < a calibrated threshold (typically Wasserstein < 0.15). It detects subtle changes in style, length and vocabulary that LLM-as-judge does not capture without going through it too. It is fast: the lightweight embedding takes around 50 ms per response.

Metric 5 — Cost per request. Output tokens per request and kW per request. Gate: cost_per_request(v2) / cost_per_request(v1) < 1.20. It detects new models that generate significantly longer responses or that consume more energy for the same load (quantisation degradation, optimisations failing). Without this gate, an “update” can silently double the bill.

Metric	Type	Measurement latency	Typical gate	Detection
TTFT P95	Quantitative	5 min	`< 110% baseline`	Latency regression
Error rate	Quantitative	1 min	`< 1pp over baseline`	Crashes, generation broken
Quality (LLM-judge)	Offline semantic	hours, over golden set	`> baseline − 0.05`	Functional quality
Statistical drift	Statistical	~5 min	Wasserstein < 0.15	Style, length, vocabulary
Cost per request	Quantitative	5 min	`< 120% baseline`	Economic/energy efficiency

The mechanics on Kubernetes: Argo Rollouts

Argo Rollouts extends the standard Kubernetes Deployment with a new Rollout resource that orchestrates traffic progression and automatic analyses. It integrates with any service mesh (Istio, Linkerd) or ingress controller that supports traffic splitting (NGINX, Traefik, Gateway API).

A minimal example of a 1 → 5 → 25 → 100 % canary with TTFT and error rate gates:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: vllm-llama70b }
spec:
 replicas: 4
 strategy:
 canary:
 canaryService: vllm-llama70b-canary
 stableService: vllm-llama70b-stable
 trafficRouting:
 nginx:
 stableIngress: vllm-llama70b-ingress
 steps:
 - setWeight: 1
 - pause: { duration: 30m }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 5
 - pause: { duration: 1h }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 25
 - pause: { duration: 2h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 50
 - pause: { duration: 4h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 100
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata: { labels: { app: vllm-llama70b } }
 spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args: [ --model=/models/llama-70b-fp8-v2 ]  # new version
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: ttft-error-gate }
spec:
 metrics:
 - name: ttft-p95-ratio
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 1.10
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
 /
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))
 - name: error-rate-diff
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 0.01
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 sum(rate(vllm:request_total{version="v2",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v2"}[5m]))
 -
 sum(rate(vllm:request_total{version="v1",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v1"}[5m]))

If any of the AnalysisTemplates fails, Argo Rollouts backs out automatically: it sets weight=0 on the canary, alerts the operator and keeps v1 serving 100 %. Human operation is reduced to investigating the failure and deciding whether to relaunch or abort.

Flagger offers a more opinionated alternative: weight progression is automatic as a function of metric success instead of a fixed pause; the operator defines a target (maxWeight: 100, stepWeight: 10, metrics: [...]) and Flagger raises or lowers it according to behaviour. Both are mature as of May 2026; the choice is usually dictated by which service mesh is already in the cluster.

The vLLM detail: why there is no “hot swap” of the model

As of May 2026, vLLM v1 does not support hot swapping the model inside the same replica without restarting the engine. The --model argument is evaluated at start-up; changing it requires re-instantiating the LLMEngine, which restarts connections and discards the KV cache. So the rollout unit is the whole replica: you do not do “v1 loads the new model on one of its GPUs” but “a v2 replica is brought up next to a v1 replica and traffic is shared out via the LB”.

TensorRT-LLM with Triton has a similar mechanism: changing the model requires a reload of the Triton backend. SGLang does not support robust hot swap either. The operational consequence: LLM rollout is always going to cost extra GPUs during the window, and the choice between blue-green, canary and shadow is exactly the question of how many extra and for how long.

The three pitfalls specific to LLM rollout

Pitfall 1 — sticky sessions break canary comparability. If the LoadBalancer does session affinity by client IP (common in NGINX and Traefik with loadbalancer.kubernetes.io/session-affinity: ClientIP), a user A always lands on v2 while B always lands on v1. Load distributions, prompt profiles and client behaviour are not random between the two pools, which statistically invalidates any gate comparison. Solution: for a canary, disable session affinity (sessionAffinity: None) or use affinity by random request-id. If the client app requires stickiness for functional reasons (conversational memory persisted in cache), canary is not the right strategy; use blue-green or shadow.

Pitfall 2 — LLM-as-judge is not an inline real-time gate. The temptation to use quality score as a live gate is strong, but the judge’s latency (2–8 s per sample) makes it unfeasible to evaluate more than a 1–2 % sampling of the traffic, and the results arrive minutes late. Operational solutions: (a) offline pre-canary eval over a golden set as a prerequisite for starting (if it fails, the canary does not even begin); (b) during the canary, capture requests + responses from v2 in real time and run the judge asynchronously in a batch job that finishes before the next step; (c) use statistical embedding drift as a fast proxy for inline quality, and reserve the judge for intermediate gates between steps.

Pitfall 3 — SSE streaming complicates shadowing. Classic traffic mirroring (NGINX mirror, Istio MirrorPolicy) is designed for request/response HTTP: it copies the request, lets the primary server answer the client, and duplicates the request to the secondary, discarding the response. With SSE, the secondary’s response is a continuous stream of several seconds, and keeping two streams in parallel doubles the load on the proxy. Solutions: (a) shadow only non-streaming requests (chat without streaming, embeddings, classification, batch eval), (b) shadow the streaming traffic but with a short timeout on the secondary (discard the shadow if it takes longer than 30 s), (c) replace the shadow with a small-weight canary (1 %), which does handle streaming well.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB = 16 GPUs, serving Llama 70B FP8 with TP=4 (4 possible replicas, one per node):

Blue-green: impossible to keep two complete pools of 4 replicas without extra GPUs. Practical solution: blue-green with reduced pools (2 v1 replicas + 2 v2 replicas) during the window, with accepted capacity degradation (half the sustained RPS SLO), or having a parallel cluster (another node) reserved for rollouts.
Canary: feasible. Start with 3 v1 replicas + 1 v2 replica (25 % nominal weight, though the traffic weight also varies). Move to 2 v1 + 2 v2 at 50 %, then 1 v1 + 3 v2, finally 0 v1 + 4 v2.
Shadow: awkward because of the GPU cost. Reserve it for pre-canary validation of major changes, during a short window (4–8 hours) with shadowed traffic limited to a sample (10–20 % of requests, not 100 %).

For clusters of 8 GPU nodes, the three patterns are all sustainable. The operational rule: the rollout budget is typically 25–30 % of the cluster’s sustained capacity, and buying for the peak plus that head-room makes the capacity planning numbers add up.

What we have not covered (upcoming articles)

Multi-region rollouts: how to coordinate a canary when the cluster is geographically distributed.
A/B testing of prompts (not of models): the same model with two different system prompts, measuring conversion.
Embedding rollback: changing the embedding model of a RAG system means re-embedding the whole corpus, so the canary mechanics are different. See RAG corpus curation.
Feature flags for LLM: granularity per tenant or per feature within the same model.
End-to-end continuous deployment: integration with the retrain pipeline so a new adapter is promoted automatically after passing evals.

References

Argo Rollouts project — argoproj.io/argo-rollouts (Rollout and AnalysisTemplate CRDs).
Flagger project — fluxcd.io/flagger (alternative with automatic progression).
Istio — Traffic Mirroring (mirror configurable at VirtualService level).
NGINX Ingress — Canary annotations (nginx.ingress.kubernetes.io/canary-*).
vLLM project — issue tracker on hot model swap (status as of May 2026: in design, not production-ready).
Hou et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024) — reference on goodput metrics applicable to canary gates.
Bürkner et al. — Statistical methods for detecting model drift in production (various articles on Wasserstein and KL in ML monitoring).

Canary on lo0 — Tech Blog

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

TL;DR

The analogy: the station master and the track gauge

A `Deployment` serves tokens, but offers no contract

KServe: from KFServing to a CNCF incubating project

Anatomy of the control plane

`InferenceService`: predictor, transformer, explainer

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

The storage initializer and `storageUri`

The deployment modes

The LLM branch: `LLMInferenceService`

The Open Inference Protocol: the contract itself

The routes and the payload

Who actually implements it

The `generate` extension and why it did not win

The tension, unvarnished

Four manifests you can actually apply

Autoscaling and the problem of zero

Operational traps

Decision map: when KServe, and when it is overkill

For an inference factory

See also

Sources

Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

TL;DR

You are here: DEPLOY (and the transition to RETRAIN)

The analogy: opening night of a play

The three strategies in detail

Blue-green

Canary

Shadow / mirror

The five regression metrics that act as a gate

The mechanics on Kubernetes: Argo Rollouts

The vLLM detail: why there is no “hot swap” of the model

The three pitfalls specific to LLM rollout

Applied to typical on-premise hardware

What we have not covered (upcoming articles)

See also

References

Canary on lo0 — Tech Blog

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

TL;DR

The analogy: the station master and the track gauge

A Deployment serves tokens, but offers no contract

KServe: from KFServing to a CNCF incubating project

Anatomy of the control plane

InferenceService: predictor, transformer, explainer

ServingRuntime and ClusterServingRuntime: the catalogue of trains

The storage initializer and storageUri

The deployment modes

The LLM branch: LLMInferenceService

The Open Inference Protocol: the contract itself

The routes and the payload

Who actually implements it

The generate extension and why it did not win

The tension, unvarnished

Four manifests you can actually apply

Autoscaling and the problem of zero

Operational traps

Decision map: when KServe, and when it is overkill

For an inference factory

See also

Sources

Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

TL;DR

You are here: DEPLOY (and the transition to RETRAIN)

The analogy: opening night of a play

The three strategies in detail

Blue-green

Canary

Shadow / mirror

The five regression metrics that act as a gate

The mechanics on Kubernetes: Argo Rollouts

The vLLM detail: why there is no “hot swap” of the model

The three pitfalls specific to LLM rollout

Applied to typical on-premise hardware

What we have not covered (upcoming articles)

See also

References

A `Deployment` serves tokens, but offers no contract

`InferenceService`: predictor, transformer, explainer

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

The storage initializer and `storageUri`

The LLM branch: `LLMInferenceService`

The `generate` extension and why it did not win