Kserve on lo0 — Tech Blog

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

Mon, 31 Aug 2026 10:00:00 +0200

Closing the platform and self-service run, after Backstage and Kubeflow. The question here is purely economic with a technical trap inside: switching off the GPU when nobody is using it looks like free money, until the first user of the day waits two minutes.

TL;DR

A GPU costs the same serving as it does idle. An H100 on dedicated cloud runs at 2 to 4 euros an hour whether it is used or not, so switching the service off when there is no traffic looks like money given away. Knative is the piece that does it: its autoscaler scales down to zero replicas and brings them back when a request arrives, measuring concurrency instead of CPU, which is what the Kubernetes autoscaler cannot do.

The trap is in the cold start. Knative brings up a pod in seconds, but that is only the beginning: an LLM has to load its weights from disk into HBM and compile its graphs, and there you go from tens of seconds to minutes depending on model size and storage. One reference study measures vLLM startup at around 20 seconds even for a 3 billion parameter model, and weight loading grows linearly with size. The first user of every traffic trough pays that whole wait.

Hence the rule. Switching off the GPU pays off with sporadic traffic, many lightly used models or development environments, and it is a trap with a strict latency SLO, sustained traffic or huge models whose startup breaks any interactive budget. Between all and nothing there are three middle grounds that almost always win: keep one replica warm, share the GPU with model-swap, or speed up weight loading so that startup fits inside the autoscaler’s cycle.

The analogy: the restaurant that switches off the kitchen

A restaurant with the kitchen running at four in the afternoon, without a single customer, burns gas for nothing. The temptation is to switch it off and light it again when somebody walks in. The problem is the time it takes to heat up. If the kitchen is an induction hob, it heats in seconds and switching it off between services is obvious. If it is a wood-fired oven that takes two hours to come up to temperature, switching it off means the first customer of the night eats cold or leaves.

Serving a small model is an induction hob. Serving a 70 billion parameter model is the wood-fired oven: the weights that have to go up into HBM are the wood you have to burn until you reach temperature, and wishing does not speed it up. The decision to switch off the kitchen, that is, to scale to zero, does not depend on whether you want to save gas, but on how long your kitchen takes to heat and on how long your first customer is willing to wait. This whole article is that calculation.

How Knative scales

Knative graduated in the CNCF in October 2025 and has three components; the one that concerns us is Serving, the serverless application component with autoscaling. Its object model has four pieces: the high-level Service, the Configuration that describes the desired state, the immutable Revision which is what actually scales, and the Route that splits traffic between revisions.

What makes scaling to zero possible are two data plane components. The queue-proxy is a sidecar in every pod that measures concurrency and applies the simultaneous request limit. The activator is the key piece: when the service is at zero replicas, requests are routed to it, and it parks them in a queue, tells the autoscaler that capacity is needed, and forwards the parked request once a pod is ready. It is the waiter who takes your order and tells you the kitchen is warming up, instead of shutting the door on you.

KPA versus HPA

Knative’s own autoscaler, the KPA, measures concurrency and requests per second, not CPU or memory. That is the difference that matters: the Kubernetes autoscaler, the HPA, scales on CPU and does not know how to go down to zero, its operational minimum is one. To switch off completely you need the KPA.

The KPA works with two windows. The stable one, 60 seconds by default, averages normal traffic. The panic one, much shorter, reacts to sharp spikes: when demand exceeds twice the capacity of the current replicas, it enters panic mode and scales in one go. The default concurrency target per replica is 100, but for LLM inference the useful value is usually 1 to 4, because each request occupies the GPU heavily; the official KServe example uses a target of 1.

Two timers govern the shutdown. The scale-to-zero-grace-period, 30 seconds by default, is how long the system waits to have the restart machinery ready before withdrawing the last replica. The scale-to-zero-pod-retention-period, zero by default, is the minimum time the last pod survives after the decision to switch off. Raising this second value is one of the levers for not switching off as soon as there is a micro-trough in traffic.

Why an LLM cold start is not starting a pod

Here is the core of the article, and there is a nuance that most discussions skip. Knative bringing up a pod in seconds does not mean the model is ready in seconds. Starting an inference service has several phases, and the two costs that dominate are not the ones you would expect.

A recent study measuring vLLM cold start breaks it down step by step. For a 3 billion parameter model, total startup is around 20 seconds, and the counterintuitive finding is that most of it is CPU-bound, not GPU-bound: process initialisation, library imports, engine construction. Inside that there are two costs that grow, and they are the ones the acceleration techniques attack.

The first is weight loading, which scales linearly with model size. In that same study it goes from half a second for a small model to almost five for a 16 billion one. Projected onto real sizes, the on-disk weight rules: an 8 billion model takes around 16 GB in FP16 and 8 in FP8; a 70 billion one, 140 and 70; a 405 billion one, 810 and 405. Loading 140 GB from disk into HBM does not happen in a blink, and if storage is slow or sits on the network, the minute arrives on its own.

The second is graph compilation. Graph transformation and capture with torch.compile costs 11 to 21 seconds if there is no compilation cache, and drops to 3 to 6 with one. It is half a battle that has nothing to do with the weights, and it is won by keeping the compilation cache between startups.

The operational conclusion is that the enemy is not only HBM: it is weight loading plus compilation. For small models the weights are only a fraction of startup, and switching off the GPU is almost painless. For large models the weights dominate, and the cold start becomes the problem. Loading a 70 billion model with the naive loader of the usual library can run to several minutes, and that is exactly the time your first user waits.

The techniques that shorten the wait

The good news is that the cold start can be attacked on both fronts. On the weights side, streaming loaders such as NVIDIA’s transfer the weights to the GPU concurrently instead of sequentially: the published data takes an 8 billion model from around 48 seconds with the standard loader to around 14 with streaming from SSD, and to under 8 with fast storage. The most graphic case is a 122 billion model, 233 GB on disk, which goes from three and a half minutes to around 37 seconds. That number has a direct consequence: 37 seconds fit inside an autoscaler polling cycle of 30 to 60 seconds; three and a half minutes do not. It is the difference between scaling to zero being viable with large models or not being viable at all.

On the compilation side, the technique is cheaper still: keep the torch.compile cache between startups, and with it the 11 to 21 seconds become 3 to 6. Tools such as Tensorizer and OCI images with the weights packaged inside attack the same problem from another angle, as we saw in the article on speeding up cold start.

Knative and KServe: the two modes

KServe leans on Knative for its serverless mode, and this is what connects everything above to a real platform. KServe has two deployment modes.

The Serverless mode uses Knative Serving and the KPA, supports scaling to zero and measures concurrency and requests per second. It is enabled by setting minReplicas: 0 in the InferenceService. It is the natural path for serving many intermittent models.

The Standard or raw mode uses the Kubernetes HPA, does not scale to zero, and measures CPU and memory. You pick it when you need something Knative restricts, such as mounting several volumes, and you do not need to switch off completely.

There is a third path of particular interest to anyone who already has KEDA deployed: in Standard mode, KServe recommends KEDA for generative inference, and KEDA can scale to zero on Prometheus metrics such as queue length or tokens per second. It is a route to scaling to zero on your own metrics, outside Knative, and it fits better with autoscaling guided by the model’s real saturation signal rather than by generic concurrency.

For anyone serving many lightweight predictive models there is also ModelMesh, the high-density pattern that packs many models into the same pods and rotates them in memory, without leaning on Knative. It is the equivalent of model-swap: instead of switching off and on per model, it shares one GPU between many. It is aimed at classic short-response inference, not at generative LLMs.

The economics

The decision comes down to comparing what you save by switching off with what you risk at startup. The saving is easy to estimate. If a GPU costs $ C $ euros per hour and your service is idle $ H $ hours a day, switching it off saves on the order of

$$\text{daily saving} \approx C \cdot H$$

With an H100 at 3 euros an hour and 16 idle hours a day, that is 48 euros a day per GPU, on the order of 1,400 a month. Across a fleet of several lightly used models the figure multiplies and stops being negligible.

The cost is on the other side, and it is not measured in euros: it is measured in SLO. Scaling to zero only pays off if the cold start fits inside what your worst-case latency tolerates. With a small model and fast storage, tens of seconds at most, and for an internal assistant used in bursts that may be acceptable. With a 70 billion model and the naive loader, minutes, and no interactive SLO survives that.

Between switching off completely and never switching off there are three middle grounds, and almost always one of them is the right answer:

Warm replica, with minReplicas: 1. You remove the first user’s cold start in exchange for paying a continuous GPU-hour. The break-even point is direct: keeping it warm pays off when the cost of that GPU running hot is lower than the reputational or SLO cost of the first user waiting. For a customer-facing service, almost always.

Model-swap or sleep on a shared GPU, as we saw in serving several models on one GPU. The process stays alive and the model is unloaded and reloaded in seconds instead of starting from scratch. It is the middle ground for several models that do not justify a GPU each but do not tolerate a full startup either.

Speed up loading so that the cold start fits inside the autoscaler’s cycle, with weight streaming and a compilation cache. It is what makes real scaling to zero viable with large models, and what turns “three and a half minutes” into “37 seconds”.

Operational traps and honest scepticism

The pod starts in seconds, the model does not. Do not confuse Knative’s startup time with the time until the model answers. The difference is the whole article.

The default concurrency target is 100, and for an LLM that is a disaster. Each request occupies the GPU; a target of 1 to 4 is the reasonable one. Leaving the default 100 means accepting a hundred simultaneous requests on a replica that handles four.

The cold start is not just weight loading. Graph compilation costs what it costs, and it is attacked separately, with a cache. Optimising only storage leaves half the problem untouched.

Cilium and Gateway API are not a supported path yet. Knative supports Gateway API in beta and tests with Istio, Contour and Envoy Gateway; integration with Cilium does not appear as tested and has documented friction. If your ingress is Cilium, verify it before counting on it.

KEDA may be a better signal than the KPA. Scaling on generic concurrency is worse than scaling on the model’s real saturation metric. If you already have KEDA, Standard mode with scaling to zero on your own metrics deserves a test against the default Serverless.

For an inference factory

The decision to switch off the GPU is not ideological, it is arithmetic, and it depends on two numbers: how long your model takes to start and how long your worst user waits. Three rules close the matter.

First, switch off what is intermittent and cheap to start: development environments, small models used sporadically, the long tail of models almost nobody touches. There, scaling to zero is direct money with no appreciable SLO cost.

Second, do not switch off what is critical and expensive to start: the customer-facing model with a strict SLO, and the huge models whose cold start is measured in minutes. There a warm replica costs less than a user waiting.

Third, before deciding, measure your real startup and fix it. Many services are ruled out of scaling to zero over a three-minute startup that, with weight streaming and a compilation cache, drops to thirty seconds and changes the answer. The most profitable lever is not choosing well between switching off or not: it is making switching off cheap.

With this we close the platform and self-service run. The portal shows what is there, the ML toolbox gives what is missing without duplicating what you already have, and scaling to zero decides how much of all that is switched on when nobody is looking. All three answer the same question from different angles: how to put an expensive platform at the disposal of its users without going broke keeping it switched on.

Sources

CNCF, Cloud Native Computing Foundation announces Knative’s graduation — https://www.cncf.io/announcements/2025/10/08/cloud-native-computing-foundation-announces-knatives-graduation/
Knative Docs, Serving architecture — https://knative.dev/docs/serving/architecture/
Knative Docs, Request flow — https://knative.dev/docs/serving/request-flow/
Knative Docs, Configuring KPA-specific autoscaling — https://knative.dev/docs/serving/autoscaling/kpa-specific/
Knative Docs, Configuring concurrency — https://knative.dev/docs/serving/autoscaling/concurrency/
Knative Docs, Configuring scale-to-zero — https://knative.dev/docs/serving/autoscaling/scale-to-zero/
Knative Blog, Managing Gateway API ingress with the Knative Operator — https://knative.dev/blog/articles/gateway-api-ingress-with-knative-operator/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Autoscaling with Kubernetes HPA (Serverless vs Standard) — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/hpa-autoscaler
KServe Docs, Autoscaler for generative inference (KEDA) — https://kserve.github.io/website/docs/model-serving/generative-inference/autoscaling
KServe GitHub, docs/samples/autoscaling (InferenceService minReplicas 0) — https://github.com/kserve/kserve/blob/master/docs/samples/autoscaling/README.md
Wang et al. (MLSys 2026), Breaking the ice: analyzing cold start latency in vLLM — https://arxiv.org/abs/2606.07362
NVIDIA Developer Blog, Reducing cold start latency for LLM inference with NVIDIA Run:ai Model Streamer — https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer
Microsoft Azure SDK Blog, Eliminate LLM cold starts: load models up to 6x faster — https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/
Hugging Face Blog, Llama 3.1 (VRAM and precision tables) — https://huggingface.co/blog/llama31
CloudZero, H100 GPU cost 2026 — https://www.cloudzero.com/blog/h100-gpu-cost/

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

Mon, 31 Aug 2026 09:30:00 +0200

Second post in the platform and self-service run, after Backstage and before Knative. The question here is not what Kubeflow is, but which parts of Kubeflow still make sense once you have built half a platform on your own.

TL;DR

Kubeflow graduated in the CNCF on 17 August 2026, and that headline hides the important one: it is no longer a monolithic platform, it is an umbrella of independent subprojects, each with its own repository, its own version and its own life. That changes how you have to evaluate it entirely. The 2020 question, should I install Kubeflow?, has become the 2026 one, which piece of Kubeflow do I install, and at what cost?

For a platform that already has RKE2, vLLM, KServe, Flux and Volcano or Kueue, the honest answer is that Kubeflow as a whole duplicates almost everything you already have. KServe no longer lives under Kubeflow: it left, and since November 2025 it has been a CNCF project in its own right, so anyone serving models with KServe does not need Kubeflow to serve. The Trainer leans on Volcano and Kueue, it does not replace them. And the full manifest drags in Istio, Dex and cert-manager as hard dependencies that clash with what you already operate. The only thing that may be worth installing on its own is Pipelines, as an orchestrator for data and training flows, and perhaps the Trainer for its fine-tuning templates. The rest you either already have, or do not need.

The analogy: the hypermarket and the specialist shop

Kubeflow was born as a hypermarket: a single building with everything an ML team might need, from notebooks to the inference server, taking in pipelines and hyperparameter tuning along the way. It made sense in 2019, when assembling each piece separately was a project in itself and there were no standards.

By 2026 the hypermarket has been converted into a street of specialist shops. The butcher’s counter, KServe, went independent, opened its own premises and is doing better alone. The fishmonger’s, the Trainer, now depends on the central market next door, Volcano and Kueue, instead of having its own cold store. And the shared building still demands that you put in its wiring, its plumbing and its security system (Istio, cert-manager, Dex) even though you already have your own.

Anyone arriving today with a half-full fridge, which is the case for anyone with a platform already running, does not need the whole hypermarket. They need to know which shop to walk into for what they are missing, and to avoid buying twice what they already have at home.

What Kubeflow is in 2026

Kubeflow was created by Google in 2017 and graduated in the CNCF on 17 August 2026, reaching the top level alongside Kubernetes or Prometheus. The graduation figures are serious: more than 6,600 contributors from over a thousand organisations, and adopters such as Bloomberg, NVIDIA, Red Hat or Spotify.

The structural change is the one that matters for the decision. The full distribution, the Kubeflow Community Distribution, is numbered by year and month (the current one is 26.03, from March 2026, on a six-monthly cadence), but each component is by now a subproject with its own repository and its own cycle, usable independently. The distribution only packages them together. That means “installing Kubeflow” is no longer an atomic decision: it is a shopping list.

Component by component

Kubeflow Pipelines: the piece you may actually want

Pipelines orchestrates containerised ML workflows. They are written in Python with its SDK, which compiles to an intermediate YAML representation so they are portable, and underneath they run on Argo Workflows. It is the piece most likely to add value to an existing platform: a flow of “ingest, embeddings, fine-tuning, evaluation, deployment” is exactly what you need to operate RAG and adapters repeatably.

The small print has two parts. The first is that if you already use plain Argo Workflows, Pipelines adds a layer of SDK, interface and metadata on top of the same engine: it does not bring a new scheduler, it brings the typed component model and the runs interface. The second is that its metadata and lineage system, MLMD, is being removed from Pipelines itself, so it is not wise to build the platform’s traceability on that particular piece.

Kubeflow Trainer: it leans on your scheduler, it does not replace it

The old Training Operator, the one with PyTorchJob and TFJob, has been rewritten as Kubeflow Trainer, with a single TrainJob API that unifies all frameworks. Version 2.2 positions itself explicitly for distributed training and LLM fine-tuning, with support for PyTorch, DeepSpeed, HuggingFace and company.

A common expectation needs correcting here. The Trainer does not replace Volcano or Kueue: it leans on them. The TrainJob carries a podGroupPolicy field that automatically creates the Volcano PodGroup objects for gang scheduling, and it integrates with Kueue’s queues. If you already built that scheduler, as we saw in its post, the Trainer adds the TrainJob abstraction and some fine-tuning templates on top of what you already have, not a new scheduling layer. It may be worth it for the templates; not for the scheduler.

Katib: only half relevant

Katib does Kubernetes-native AutoML: hyperparameter optimisation with algorithms such as Bayesian optimisation, TPE or Hyperband, plus architecture search. In the LLM era it has repositioned itself, with official documentation for tuning fine-tuning hyperparameters and even for tuning RAG pipelines.

The practical reading is lukewarm. Architecture search is irrelevant for anyone doing LoRA and RAG. Hyperparameter optimisation can be useful for learning-rate or LoRA-rank sweeps, but it competes with doing it from the SDK of the QLoRA runbook itself or with Ray Tune, and it rarely justifies dragging in Katib for that alone.

KServe: the piece that is no longer in Kubeflow

This is the fact that settles half the evaluation. KServe left Kubeflow. It was born inside the project in 2019, was donated to the Linux Foundation in 2022, was renamed from KFServing to KServe and graduated from Kubeflow that same year, and since November 2025 it has been a CNCF incubating project in its own right. In the 2026 taxonomy it appears as an ecosystem project, external, not as a core component.

The consequence is direct: anyone already serving with KServe does not need Kubeflow to serve. And KServe is exactly where the LLM action is: its 0.15 release strengthened the vLLM backend and introduced an LLMInferenceService resource with disaggregated serving, prefix caching, per-variant autoscaling and OpenAI-compatible APIs. If your serving layer is vLLM on KServe, you already have that layer complete and Kubeflow adds nothing to it.

Model Registry, Notebooks and the rest

The Model Registry, now reorganised under the name Kubeflow Hub, is an index of models, versions and metadata. It stores pointers and states, not the bytes. Against MLflow, it lacks the experiment and metric tracking that is MLflow’s strength, to the point that the guides recommend combining them rather than substituting one for the other; and against an OCI registry, it does not package the binary, which still lives wherever you put it. It is still a young component, below version 1.0.

The Notebooks live in two versions, the stable 1 and a 2 redesigned on CRDs that is still in alpha. The Spark Operator remains in the core; Feast, by contrast, has left the core and is now an ecosystem project, a nuance worth being clear about before counting on it.

Multi-tenancy: powerful and tied to Istio

Isolation between teams is done with Profiles, a CRD that wraps a namespace and gives it RoleBindings, ServiceAccounts and Istio authorisation policies that validate an identity header derived from OIDC. It is a complete model, but it has an Achilles heel: it depends entirely on the Istio sidecar and on that header. If traffic bypasses the mesh or someone forges the header, isolation breaks, and all multi-tenant security ends up coupled to Istio, which clashes head-on if you already operate another mesh or no service mesh at all.

The cost of installing the whole thing

Here is the main reason not to install the full distribution on top of an existing platform. The 26.03 manifest drags in as hard dependencies: Istio, cert-manager, Dex for OIDC, OAuth2-Proxy, and Knative Serving and Eventing for KServe. It recommends 16 GB of RAM and 8 vCPU as a minimum, and the aggregate of all components comes to around 4.4 CPU cores and 12 GB of memory for the control plane alone.

The project itself acknowledges the problem. There is an open thread in its repository, bluntly titled “community feedback on Kubeflow’s complexity”, where it is admitted that Kubeflow practically forces a dedicated cluster because of the assumptions it makes about what is installed, that Istio and Dex should be swappable and not fixed dependencies, and that a lighter Helm that installs only what is needed is missing. For anyone already operating cert-manager, an ingress and GitOps with Flux, the full manifest does not coexist: it duplicates and tramples the base infrastructure.

Kubeflow against the alternatives

The field is wide and almost all the comparative literature is written by vendors with a product of their own, so it has to be read with tweezers. With that caution:

Plain Argo Workflows is the engine Pipelines uses underneath; if all you need is container graphs, Pipelines is overhead. MLflow is strong in experiment tracking and lightweight registry, installs in minutes, and does not orchestrate distributed training: it is complementary, and in fact we use it in the post on prompt versioning. Flyte is a typed orchestrator that many perceive as lighter to operate than full Kubeflow. And Ray on Kubernetes is the most serious competitor for LLM workloads, because with a single runtime it covers training, serving and tuning, overlapping at once with the Trainer, Katib and KServe.

Where Kubeflow wins is in breadth and in governance: a CNCF-graduated umbrella with adoption by large companies. Where it loses is in operational weight, in hard dependencies, and in the fact that its best pieces are already independent projects that do not require the rest.

The real fit with LLM workloads

The project has moved towards GenAI, and that has to be acknowledged: there is an SDK with LLM fine-tuning templates, the Trainer adds support for reinforcement post-training methods, and new agent-oriented pieces have appeared. It is not empty marketing, there is real work there.

But the question for a specific platform is not whether Kubeflow does LLM: it is which part I have not already duplicated. On a stack of RKE2, vLLM, KServe, Flux and Volcano or Kueue, the balance looks like this:

Component	Does it add anything new to your stack?
KServe	No: you already have it, and it is no longer Kubeflow’s
Trainer	Only the fine-tuning templates; you already have the scheduler
Katib	Marginal: LoRA HP sweeps, little more
Pipelines	Possibly yes, as a flow orchestrator, but it runs on Argo
Model Registry	Optional: competes with MLflow and with your OCI registry
Notebooks	Depends on what you use for notebooks today
Multi-tenancy	No, if you already isolate by namespace without tying everything to Istio

Operational traps and honest scepticism

Do not install the full manifest on a platform that already works. You are going to duplicate Istio, cert-manager and the rest, and fight a dependency war with no prize at the end. If something in Kubeflow interests you, install it as a standalone subproject.

KServe is not Kubeflow. It is the most common category error of 2026. If someone justifies building all of Kubeflow “to serve models”, you already know they have not checked that KServe left long ago and lives on its own.

The Trainer does not save you the scheduler. It needs Volcano or Kueue underneath. If you expected it to solve GPU gang scheduling by itself, it does not.

Multi-tenancy ties you to Istio. Before adopting Profiles, ask yourself whether you want security between teams to depend on one specific mesh. If you already isolate well by namespace and RBAC, you may not need that layer.

MLMD is on its way out. Do not build your pipelines’ traceability on the metadata system the project itself is retiring.

For an inference factory

The conclusion is comfortable to apply. Kubeflow in 2026 is not a yes-or-no decision, it is a menu you order from à la carte. On a platform that already serves with KServe, schedules with Volcano and deploys with Flux, the menu shrinks a great deal.

If you need to orchestrate data and training flows repeatably, and you do not want to build plain Argo Workflows with its ergonomics, Pipelines is the piece worth evaluating, installed standalone. If you do distributed fine-tuning frequently and want a uniform abstraction on top of your scheduler, the Trainer may save you some templates, knowing that it leans on the Volcano you already have. Everything else you either have, or a specialist piece covers better.

The expensive mistake would be installing the whole hypermarket to buy bread. Kubeflow stopped being that; treat it as what it is, a street of shops, and walk only into the one you are missing. The next piece in this run, Knative, is precisely one of those specialist shops, the one that decides whether switching off the GPU when nobody is using it pays off or costs you dear.

Sources

CNCF, CNCF announces Kubeflow’s graduation — https://www.cncf.io/announcements/2026/08/17/cncf-announces-kubeflows-graduation-solidifying-the-standard-for-cloud-native-ai-operations/
CNCF, Kubeflow unveils new cloud native innovations to supercharge AI — https://www.cncf.io/blog/2026/07/28/kubeflow-unveils-new-cloud-native-innovations-to-supercharge-ai/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
Kubeflow Docs, Introduction / components — https://www.kubeflow.org/docs/started/introduction/
Kubeflow Docs, Pipelines overview — https://www.kubeflow.org/docs/components/pipelines/overview/
Kubeflow Trainer Docs, Volcano gang scheduling and Kueue — https://trainer.kubeflow.org/en/latest/operator-guides/job-scheduling/volcano.html
Kubeflow Docs, Migrating to Kubeflow Trainer v2 — https://www.kubeflow.org/docs/components/trainer/operator-guides/migration/
Kubeflow Docs, Katib overview — https://www.kubeflow.org/docs/components/katib/overview/
Kubeflow Docs, Hyperparameter optimization for LLM fine-tuning — https://www.kubeflow.org/docs/components/katib/user-guides/llm-hp-optimization/
Kubeflow Docs, Model Registry overview — https://www.kubeflow.org/docs/components/model-registry/overview/
Kubeflow Docs, Multi-tenancy design — https://www.kubeflow.org/docs/concepts/multi-tenancy/design/
GitHub, kubeflow/manifests (dependencias y recursos) — https://github.com/kubeflow/manifests
GitHub, kubeflow/manifests #2451: community feedback on complexity — https://github.com/kubeflow/manifests/issues/2451
InfoQ, Kubeflow expands AI capabilities as CNCF graduation nears — https://www.infoq.com/news/2026/08/kubeflow/

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

Sun, 26 Jul 2026 20:00:00 +0200

This blog has devoted a good part of its 145 articles to what happens inside the inference process and another good part to what happens around it. What is missing is a thread that cuts across all of that and answers a question that takes thirty seconds to ask in an audit and does not get answered in three weeks in production: when the endpoint returns a token, where did that model come from, who signed it, and who proves that the binary running on the GPU is the one that was approved?

This series walks that chain backwards from the only visible point, the endpoint: the control plane and the API contract (this article), the registry and the distribution of the bytes (2/4), the signing and provenance that let you trust them (3/4), and the identity and isolation of whoever runs them (4/4). All of it with projects under the governance of CNCF, LF AI & Data or the Linux Foundation: a chain of trust that depends on a proprietary product is no better than a support contract.

TL;DR

A vLLM Deployment serves tokens perfectly well. What it does not give you is a contract: no model versioning, no rollout by percentage, no scale-to-zero, and no common surface for the thirty non-LLM artifacts of an inference factory.
KServe provides the vocabulary: InferenceService, ServingRuntime/ClusterServingRuntime, storage initializer and storageUri. It has been CNCF incubating since 29 September 2025, after seven years as KFServing and a spell in LF AI & Data. The cadence has accelerated: v0.18.0 on 29 April 2026 and v0.19.0 on 14 June 2026.
The LLM branch is recent and different: the LLMInferenceService CRD (serving.kserve.io/v1alpha1) arrived in v0.16 in November 2025, on top of llm-d, with spec.prefill, spec.worker, spec.parallelism and spec.router.
The Open Inference Protocol (V2) is the contract: /v2/health/*, /v2/models/<m>, /v2/models/<m>/infer, in REST and gRPC, implemented by Triton, OpenVINO Model Server, Seldon MLServer and AMD Inference Server. It is tensor-centric, and the LLM world standardised de facto on the OpenAI API: KServe serves both, with the OpenAI ones under the openai/ prefix.
The honest position: the portability of an LLM today comes from the OpenAI API, not from V2. Accepting the dual track is correct; pretending there is a single standard is not.
KServe is overkill if you serve one model on one node. It starts to pay off from the fifth heterogeneous artifact onwards, and it competes on a different axis from the LLM operators: contract and lifecycle versus engine and topology.

The analogy: the station master and the track gauge

An inference factory is like a railway station with several lines.

The station master does not drive any train. His job is the service timetable: which formation leaves from which platform, how many units are coupled each hour, when an engine is pulled out for maintenance and, when new rolling stock arrives, how to divert ten per cent of the passengers to the test train before committing the whole service. That is the control plane. KServe does not generate a single token, but it decides what runs, with what configuration, with how many replicas and with what traffic split. The trains are the engines (vLLM, SGLang, TensorRT-LLM, Triton, TEI), and a competent station master does not marry a manufacturer.

And then there is the track gauge: as long as everybody builds to the same one, any locomotive rolls on any line and changing supplier is a commercial decision, not a civil engineering project. That gauge is the Open Inference Protocol, the contract that makes a client written against Triton work against OpenVINO Model Server without touching a line.

The analogy has a twist that needs no explanation in Spain: the agreed gauge is not the gauge the new rolling stock runs on. The conventional network was built to Iberian gauge and the high-speed one to standard gauge, and the practical solution was not to convert the network but to install gauge changers at the boundary points. The same thing has happened in inference: V2 is the agreed gauge and the OpenAI API is the gauge everything generative runs on. What sustains interoperability is the changer, not a single standard. We will come back to it.

A `Deployment` serves tokens, but offers no contract

In vLLM on Kubernetes we built a service with Deployment, Service, HPA and an initContainer that pulls the weights down. It works, and for one model on one node it is the right answer. What it does not give you, and cannot give you without reinventing it every time:

A stable API contract. The endpoint is “whatever the image you put there exposes”: moving from vLLM to TensorRT-LLM changes routes, error format and metric names, and the clients find out in production.

Versioning with semantics. The model is a path inside a volume; knowing which version a pod is serving requires kubectl exec. There is no declarative field that says “this service serves artifact X” and can be audited from outside.

Progressive model rollout. A RollingUpdate replaces pods; it does not split traffic between two versions by percentage. For that you need an L7 gateway in front, like the one in canary, blue-green and shadow, or a control plane that emits it.

Scale-to-zero. The HPA does not go below one replica: for a model queried twelve times a day, that is an 80 GB GPU tied up twenty-four hours.

And, above all, a common surface for what is not an LLM. A real factory does not serve three LLMs: it serves two or three LLMs, four embedding models, a couple of rerankers, ticket classifiers, a PII detector, an OCR, industrial vision and three scikit-learn artifacts that nobody remembers training. Fifteen or thirty artifacts, each with its own runtime and its own way of saying “I am ready”. The cost is not in serving the first one: it is in making sure the thirtieth does not cost thirty times what the first one did.

KServe: from KFServing to a CNCF incubating project

KServe was born in 2019 as KFServing, inside Kubeflow, in a collaboration between Google, IBM, Bloomberg, NVIDIA and Seldon. In February 2022 it was donated to the LF AI & Data Foundation; in September 2022 it split from Kubeflow and was renamed. The relevant jump: on 29 September 2025 the CNCF accepted it directly at incubating level, without going through sandbox, with a public announcement on 11 November. The figures it declared then, more than 300 contributors, 19 maintainers, more than 30 adopting organisations (Bloomberg, Red Hat, Cloudera, Nutanix, SAP, NVIDIA), come from an interested party, but the list is verifiable and the 625 contributing organisations on the CNCF dashboard support the thesis of governance not captured by a vendor.

On cadence you have to be precise, because the cliché of “KServe is slow” no longer describes reality:

Version	Date	What it brought
v0.15.0	31 March 2025	`LocalModelCache`, multi-node inference, OpenAI-compatible embeddings API, Gateway API in raw mode, KEDA integration
v0.16.0	November 2025	`LLMInferenceService`; rename of `RawDeployment` to `Standard` and of `Serverless` to `KNative`; progressive rollout in raw mode
v0.18.0	29 April 2026	WVA + KEDA/HPA autoscaling for LLMISvc, `/v1/responses` route (OpenAI Responses API), `restricted` Pod Security Standards by default on LLMISvc, namespace-scoped `ModelCache`
v0.19.0	14 June 2026	Routing by model name, static LoRA adapters, balancing across heterogeneous GPUs, dual REST/gRPC routing in Standard mode, automatic migration to `InferencePool` v1

Two major releases in under two months, with the bulk of the changelog in the LLM branch. The project is not slow: it has pivoted from classic predictive serving to generative serving. The consequences run in both directions.

The negative one is called ModelMesh. It was the high-density mode (hundreds of small models sharing processes, with dynamic load and unload) and it was, on paper, the natural answer to the problem of thirty heterogeneous artifacts. Its last release was v0.12.0 on 9 July 2023, and the kserve/modelmesh-serving repository was archived read-only on 14 April 2026. Red Hat published specific guidance for migrating to Standard mode. If somebody proposes ModelMesh in a design today, it is a sign that the documentation they read is three years old.

Anatomy of the control plane

`InferenceService`: predictor, transformer, explainer

The core CRD is InferenceService (serving.kserve.io/v1beta1), and its contribution, the one that became the common vocabulary of the field, is to break an inference service down into three components each with its own contract:

predictor: mandatory. It declares modelFormat, optionally runtime, the storageUri the weights come from and the protocolVersion it exposes.
transformer: optional. Pre- and post-processing. That it is a separate component, rather than code inside the server, is what allows it to be scaled separately (it is usually CPU-bound while the predictor is GPU-bound) and versioned separately.
explainer: optional, and the one that has aged worst: the V2 protocol does not support the explain endpoint, which only exists in V1. Anyone who needs explainability with a standard contract has a real gap here.

All three share ComponentExtensionSpec: minReplicas, maxReplicas, scaleTarget, scaleMetric, containerConcurrency, timeout and canaryTrafficPercent.

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

A ServingRuntime is a pod template that knows how to serve one or several formats. It declares supportedModelFormats (with name, version, autoSelect and priority), protocolVersions (v1, v2) and containers, with args parameterised by templates in the {{.Name}} style that are substituted with metadata from the InferenceService.

Selection is what makes this scale. If the InferenceService names a runtime, the controller looks for it first in the namespace and then at cluster level; if it does not name one, it picks among those with autoSelect: true for that modelFormat, breaking ties by priority. That is what lets a platform publish a curated catalogue (“here, huggingface is served with this vLLM and these flags”) without the product teams having to know what --gpu-memory-utilization is. In multi-tenant setups, the ClusterServingRuntimes belong to the platform and the InferenceServices to the teams: the cleanest separation-of-duties line KServe offers.

The storage initializer and `storageUri`

The storageUri field triggers the storage initializer: an init container that downloads the artifact to /mnt/models before the server starts. Schemes supported as of July 2026: s3://, gs://, https://, pvc://, hf://, oci:// and git repositories; credentials are attached via a ServiceAccount with annotated secrets. That oci:// connects with the second article of the series: the weights can live in the same registry as the images, with the same access control and the same signing capability.

The deployment modes

This is where nearly everything operational is decided. It is selected with the serving.kserve.io/deploymentMode annotation.

	KNative (formerly Serverless)	Standard (formerly RawDeployment)	ModelMesh
Resources emitted	Knative `Service` + `Revision`, `VirtualService` with Istio	`Deployment`, `Service`, `HPA`, Gateway API `HTTPRoute`	Its own high-density runtime
Dependencies	Knative Serving + Istio (or an alternative gateway)	Kubernetes 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1 and a controller	Its own components
Autoscaling	KPA (concurrency / RPS)	HPA or KEDA	—
Scale-to-zero	Yes, with an activator that holds the request	No for HTTP requests	—
`canaryTrafficPercent`	Yes, by revision weights in Istio	Silently ignored (issue #5335, open since 2 April 2026)	—
Sidecars in the pod	`queue-proxy` (+ `istio-proxy`)	None	—
Status as of July 2026	Current	Current and the default mode of the LLM branch	Archived on 14 April 2026

That table has an uncomfortable reading: the two most quoted KServe capabilities, scale-to-zero and canary by percentage, live in the mode that drags Knative and Istio along, while the mode the project has pivoted towards is the other one.

The LLM branch: `LLMInferenceService`

Since v0.16 KServe has a separate CRD for LLMs: LLMInferenceService, in serving.kserve.io/v1alpha1. It is not an extension of the classic InferenceService and does not share its API version, and confusing the two is the first mistake made by anyone arriving from old documentation.

It is built on llm-d, the project that Red Hat, Google, IBM, CoreWeave and NVIDIA donated to the CNCF. The division of roles the project itself declares is clean: KServe is the control plane (lifecycle, scaling and operational governance) and llm-d contributes distributed scheduling, with GPU utilisation, queue depth, cache residency and SLA. In the analogy: llm-d decides which line each train takes; KServe decides which trains exist and when they are withdrawn.

The fields that matter: spec.model (URI, invocation name, scheduling criticality and LoRA adapters, with static adapter reconciliation from v0.19, which connects with multi-LoRA serving); spec.template (single node, or the decode pool); spec.worker (multi-node, which triggers LeaderWorkerSet); spec.prefill (the disaggregation we analysed in disaggregated serving turned into a declarative field); spec.parallelism (tensor, data and expert parallelism); and spec.router (gateway, route, scheduler), where KServe leans on the Gateway API Inference Extension, v1.5.0 of 19 April 2026, already GA, with automatic migration to InferencePool v1 from v0.19.

The endpoint it exposes is OpenAI-compatible: /v1/chat/completions with streaming, and since v0.18 also /v1/responses. That detail underpins the next section: the LLM branch of KServe does not speak V2, it speaks OpenAI.

For multi-node there is the kserve-huggingfaceserver-multinode runtime, based on vLLM over Ray, with workerSpec.tensorParallelSize and workerSpec.pipelineParallelSize. Its restrictions are hard: it requires a ReadWriteMany PVC, it only works in Standard mode, it does not support autoscaling and it requires exactly one head pod. A reference 4×H100 SXM node with NVLink needs none of this, because a 70B fits with tensor parallel 4 on a single node; you need it from there upwards.

The Open Inference Protocol: the contract itself

The routes and the payload

The Open Inference Protocol (historically the “V2 Inference Protocol”) lives in kserve/open-inference-protocol, with REST and gRPC specifications and versioning under SemVer 2.0. Its HTTP surface is deliberately small:

Resource	Verb	Route
Server metadata	GET	`/v2`
Liveness	GET	`/v2/health/live`
Server readiness	GET	`/v2/health/ready`
Model metadata	GET	`/v2/models/<model>[/versions/<v>]`
Model readiness	GET	`/v2/models/<model>[/versions/<v>]/ready`
Inference	POST	`/v2/models/<model>[/versions/<v>]/infer`

The payload is tensorial: id, a list of inputs (each with name, shape, datatype and data) and optionally the desired outputs. The types are BOOL, signed and unsigned integers from 8 to 64 bits, 16-, 32- and 64-bit floating point, and BYTES. In gRPC the same contract is GRPCInferenceService, with six RPCs: ServerLive, ServerReady, ModelReady, ServerMetadata, ModelMetadata and ModelInfer. For high-volume embeddings, the serialisation difference is not cosmetic.

Who actually implements it

The specification repository declares KServe, NVIDIA Triton, Seldon MLServer and Core v2, OpenVINO Model Server, AMD Inference Server and TorchServe as adopters. Not all entries are worth the same as of July 2026:

Triton implements it in a documented way (“Triton exposes HTTP/REST and gRPC endpoints based on the standard inference protocols proposed by the KServe project”), plus its own extensions. It is the de facto reference implementation.
OpenVINO Model Server exposes the KServe API over gRPC and REST, the TensorFlow Serving one and OpenAI-compatible endpoints: the clearest example of a server that speaks both track gauges.
TorchServe forces you to be honest. Its official documentation carries the warning: “This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches.” An adopter on zero maintenance is not an adopter: it is a row nobody has updated.

On the health of the standard, the most eloquent data point is not in any press release: the specification repository does not have a single published release, has accumulated something like thirty commits and seventy-five stars, and its governance consists of monthly meetings and a Slack channel. There is also a “vendor neutral” fork at open-inference/open-inference-protocol with more commits than the original and no traction whatsoever. V2 is a stable contract, widely implemented and barely evolving. For a serialisation format that can be a virtue; for a standard aspiring to cover the generative world, it is a death sentence.

The `generate` extension and why it did not win

The standard did try to cover generation: the specification contains a generate_rest.yaml with two routes, /v2/models/<m>/versions/<v>/generate and .../generate_stream, and a minimalist schema: a request with text_input and parameters (temperature, top_p, max_tokens defaulting to 20, stop, details); a response with text_output, model_name, model_version and, if requested, finish_reason and logprobs; streaming over server-sent events. It is clean, it is enough for a simple completion and nobody uses it: the KServe generative inference documentation, as of July 2026, lists four endpoints, and none of them is generate:

Task	Endpoint in KServe
Text generation	`openai/v1/completions`
Chat	`openai/v1/chat/completions`
Embeddings	`openai/v1/embeddings`
Reranking	`openai/v1/rerank`

The openai/ prefix exists, according to the documentation itself, “to avoid confusion with the V1 API protocol”, and it is changed with KSERVE_OPENAI_ROUTE_PREFIX. It is plumbing that says a lot: the in-house standard was relegated to a defensive prefix while the outside one takes all the traffic.

The tension, unvarnished

V2 solved the problem of 2021, and it solved it well. A tensorial, typed contract with metadata and health checks, in REST and gRPC, is what you need to serve a classifier, a vision model, a ranker or an embedding, and it is still the right answer for that half of the catalogue: swapping an embedding server from MLServer to OpenVINO Model Server without touching clients is the track gauge doing its job.

The LLM world standardised somewhere else, not by committee, but because the entire client ecosystem (SDKs, agent frameworks, gateways, evaluation) was written against the OpenAI API, and any engine that wanted to be used had to speak it. What that means for an architecture:

There is not one contract: there are two, and both are legitimate. Designing as if there were one does violence to half the catalogue.
LLM portability is superficial. It covers chat and completion well; as soon as you use detailed logprobs, guided decoding, priority headers, prefix caching parameters or tool calling with specific schemas, you leave the common subset and get tied back to the engine. It is the warning from structured output: what is portable is the envelope, not always the contents.
The gauge changer is the gateway. What makes the set switchable is the translation and routing layer of choosing an OSS LLM inference gateway and the router as an L7 gateway, not a single standard.
Crossing over loses the versioning. V2 has /versions/<v> in the route; the OpenAI API only has the model field. The version has to be encoded in the name (asistente-70b-2026-07) or managed in the control plane. That is the difference between being able to audit which version answered and not being able to.

Four manifests you can actually apply

Platform runtime for the blog’s reference node, 4×H100 SXM 80 GB with NVLink, with tensor parallel 4:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
 name: vllm-h100-tp4
spec:
 annotations:
 prometheus.kserve.io/port: "8080"
 prometheus.kserve.io/path: "/metrics"
 supportedModelFormats:
 - name: huggingface
 version: "1"
 autoSelect: true
 priority: 10
 protocolVersions:
 - v2
 containers:
 - name: kserve-container
 # ALWAYS pin by digest in production; the tag is only for readability
 image: vllm/vllm-openai:v0.11.0
 command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
 args:
 - --port=8080
 - --model=/mnt/models
 - --served-model-name={{.Name}}
 - --tensor-parallel-size=4
 - --max-model-len=32768
 - --gpu-memory-utilization=0.92
 - --enable-prefix-caching
 - --kv-cache-dtype=fp8
 resources:
 requests: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 limits: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 # the probe that stops the pod dying while the model loads
 startupProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 15
 failureThreshold: 40 # up to 10 minutes of loading
 readinessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 10
 livenessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 30

The InferenceService that consumes it, with KEDA on vLLM metrics:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 serving.kserve.io/deploymentMode: "Standard"
 serving.kserve.io/autoscalerClass: "keda"
spec:
 predictor:
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 # the OCI registry from article 2/4 of this series
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-07"
 autoScaling:
 metrics:
 - type: External
 external:
 metric:
 backend: "prometheus"
 serverAddress: "http://prometheus.monitoring.svc.cluster.local:9090"
 query: 'sum(vllm:num_requests_waiting{model_name="asistente-70b"})'
 target:
 type: Value
 value: "8"

The canary, with its small print:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 # WATCH OUT: canaryTrafficPercent only takes effect in KNative mode.
 # In Standard mode it is silently ignored (issue kserve/kserve#5335, open).
 serving.kserve.io/deploymentMode: "KNative"
spec:
 predictor:
 canaryTrafficPercent: 10
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-08"

KServe keeps three references in the status (LatestReadyRevision, LatestRolledoutRevision and PreviousRolledoutRevision): promoting means setting the percentage to 100, reverting means pinning traffic on the previous one, and a revision that does not become ready automatically receives zero traffic.

And the two calls. Against the V2 contract, for a classifier served with MLServer or Triton:

# metadata: which model it is, what inputs it expects, what types
curl -s http://clasificador.inferencia.example/v2/models/clasificador-tickets | jq .

# readiness of the specific model, not of the server
curl -s -o /dev/null -w '%{http_code}\n' \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/ready

curl -s -X POST \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/infer \
 -H 'Content-Type: application/json' \
 -d '{ "id": "req-4711",
 "inputs": [ { "name": "input-0", "shape": [1, 4], "datatype": "FP32",
 "data": [5.1, 3.5, 1.4, 0.2] } ],
 "outputs": [ { "name": "output-0" } ] }'

Against the same control plane, but on the OpenAI-compatible surface of the LLM:

curl -s -N -X POST \
 http://asistente-70b.inferencia.example/openai/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{ "model": "asistente-70b",
 "messages": [ {"role": "user",
 "content": "Summarise last night'\''s incident report."} ],
 "max_tokens": 256,
 "stream": true }'

Two contracts and two payload shapes, with the same InferenceService as the unit of governance. That is the value proposition of a control plane: the contract is chosen by the model and the management is single.

Autoscaling and the problem of zero

Three autoscalers coexist, and choosing the wrong one is the most common cause of disappointment. KPA, in KNative mode, scales by concurrency or RPS and is the only one that goes down to zero in a usable way, because Knative provides an activator that holds the request while the pod starts; its metric does not distinguish a 30-token request from an 8,000-token one. HPA, in Standard, scales by CPU or memory: for LLMs it is useless, because the CPU of a saturated vLLM pod and that of an idle one look too much alike. KEDA, in Standard, is the right option, the one we develop in autoscaling with KEDA: it is enabled with serving.kserve.io/autoscalerClass: "keda" and autoScaling.metrics accepts external metrics against Prometheus (vllm:num_requests_running or vllm:num_requests_waiting) or, with the OpenTelemetry add-on, pushed metrics. Since v0.18 there is also Workload Variant Autoscaling, which uses HPA or KEDA as a backend.

That leaves the serious problem, which no autoscaler solves: zero. A 70B model in fp8 is on the order of 70 GB to move into HBM, and that path, from disk to HBM, is measured in minutes: the first request after idling takes that long, or it fails.

Worse: the installation documentation is explicit as of July 2026: “Scale from Zero is currently not supported in Standard mode for HTTP requests”. The mode the project has pivoted towards, and the only one that supports KEDA, does not know how to come up from zero on an HTTP request: KEDA can go down to zero (default cooldown of 300 seconds), but the way back requires an activator that Standard does not have. Anyone who wants real zero with HTTP activation still needs Knative.

The levers, in order of effectiveness: do not scale the hot model to zero (minReplicas: 1 for anything with an SLA; zero is for the long tail); attack the load time, the only solution that is not a patch, with the techniques from speeding up cold start; use LocalModelCache, available since v0.15 and extended to LLMInferenceService in v0.19, to preload artifacts into a node PVC; and set correct probes, which is what comes next.

Operational traps

The probe that kills the pod while the model loads. The most frequent and the most expensive to diagnose, because the symptom is a CrashLoopBackOff with no error in the server logs. Until early 2026 the KServe vLLM templates used initialDelaySeconds of 120 to 300 seconds on liveness, with the result described in issue #5062 (12 February 2026, now resolved): the fast models waited too long and the large ones died before finishing loading. The fix applied: a startupProbe of up to ten minutes and probes with no initial delay. A legacy runtime has to be reviewed by hand. In KNative mode there is a worse variant (issue #3795, closed as not planned): what dies is not the model container but the queue-proxy, which does not respond OK within the deadline and takes the whole pod down with it.

canaryTrafficPercent silently ignored. It deserves its own trap because there is no error and no condition in the status to give it away: in Standard mode the field does nothing. The team believes it is serving 10 % with the new version and it is serving 100 %. While issue #5335 remains open, in Standard the canary is done with HTTPRoute weights by hand, or in the gateway.

Canary and scarce GPUs do not get along. A 10 % canary of a 70B with tensor parallel 4 costs a whole 4-GPU node, not 10 % of the resources, because the minimum granularity is a complete replica. During the window there are two versions occupying twice the hardware, or capacity has to be taken from the stable one. That is the difference between a microservice canary and a model canary, and the reason a shadow over duplicated traffic often pays off better.

storageUri and credentials. Storage initializer failures are always the same ones: a ServiceAccount without the annotated secret, a KMS-encrypted bucket with a role that has no decryption permission, an S3-compatible but not exactly S3 endpoint, or intermittent 403s, with a specific fix in v0.18. It is diagnosed in the init container logs, not in the server’s. And one standing recommendation: no downloading over https:// from the internet in production; that is precisely the problem of article 2/4.

The model size against the emptyDir. By default the artifact lands in an ephemeral volume on the node’s disk. Seventy gigabytes per replica fill a system disk with surprising ease, and then the kubelet evicts pods for disk pressure, including ones that have nothing to do with it. You have to set sizeLimit or use a dedicated pvc://. For multi-node the restriction is harder, with a mandatory ReadWriteMany PVC, which rules out a good part of local block storage and pushes you towards shared file systems, with the consequences covered in storage for AI.

The dependency on Knative and Istio. The real cost of KNative mode: two more control planes to version and correlate with Kubernetes, and two sidecars, queue-proxy and istio-proxy, in every pod, with their CPU and memory consumption and their network hop per request. On a platform that already has a mesh it is a marginal cost; on a cluster dedicated to inference it has to be justified by the capabilities you are actually going to use.

Observability: the name of the metric. The ServingRuntime declares prometheus.kserve.io/port and /path annotations, but the metrics that come out are the engine’s, not KServe’s: vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:gpu_cache_usage_perc. The control plane does not normalise names across runtimes, so a dashboard written against vLLM is no use for Triton, and correlating with DCGM is still your own work. v0.19 raises the HPA/KEDA scaling status into the service conditions and emits events on readiness transitions: exactly what was needed to alert on the control plane and not only on the engine.

Decision map: when KServe, and when it is overkill

The useful comparison is not “KServe versus OME, vLLM Production Stack, Dynamo or llm-d”. Those four are covered in LLM inference operators and compete on a different axis:

Axis	What gets decided	Who competes
Contract and control plane	Which CRD describes a service, which API clients see, how the model is versioned, how the rollout is done, how LLM and non-LLM coexist	KServe, Seldon Core v2
Engine and topology	Which engine runs, how prefill/decode is disaggregated, how routing is done with cache awareness, how multi-node tensor parallel is done	vLLM Production Stack, OME, NVIDIA Dynamo, llm-d

The axes are orthogonal, and LLMInferenceService proves it: KServe did not reimplement distributed scheduling, it leaned on llm-d. Choosing KServe is not choosing instead of an LLM operator: it is choosing the contract layer that operator lives on.

Choose KServe when the catalogue is heterogeneous, LLMs plus embeddings, rerankers, classifiers and vision, which is its decisive advantage and the one none of the four operators has: a service like the one in TEI in production fits into the same vocabulary as the LLM. Also when you need a contract decoupled from the engine because you have clients you do not control; when you need declarative, auditable governance (which artifact, from which origin and with which runtime, in an object versionable in git and subject to admission policies), the piece that makes the rest of this series possible; and when you want a catalogue of platform runtimes separated from the teams’ services.

KServe is overkill when you serve one model, on one node, with up to three replicas: the Deployment from vLLM on Kubernetes is the right answer. When your workload is LLM only and the priority is performance per GPU, because there the value is in the engine. When you are going to use neither scale-to-zero nor canary by revision, because what remains is a Deployment with more YAML. And when you already have an LLM operator and only serve LLMs: putting KServe on top for completeness is adding CRDs without adding decisions.

For an inference factory

First: separate the runtime catalogue from the service catalogue, and do it on day one. The ClusterServingRuntimes belong to the platform; the InferenceServices, to the teams. That boundary turns “serving the thirtieth model” into a five-line change instead of a negotiation about vLLM flags, and it fits with GitOps with Flux: the runtime is reviewed as platform code, the service as product configuration.

Second: accept the two contracts and put the gauge changer where it belongs. V2 for the tensorial world, the OpenAI API for the generative one, and the translation in the gateway, not in every client. And since the OpenAI API does not version in the route, encode the version in the model name and make that name the same identifier that appears in the registry, in the signature and in the traces. That is what makes a request traceable back to the artifact.

Third: measure start-up before promising elasticity. The time from the pod being scheduled to the readiness probe going green, with the real model and from the real origin, decides whether scale-to-zero is a FinOps lever or an incident machine. That number, and not the price of the GPU-hour, is the one to take into the cost conversation.

And here this piece falls short on purpose. KServe gives you the contract, the lifecycle and the rollout, but its whole trust model hangs off a text string: storageUri. The storage initializer downloads whatever is at that destination and puts it in front of the GPU without asking anything else; nothing in the InferenceService guarantees that the artifact is the one that was approved, that it has not changed since the last audit, or that the origin is legitimate. The next article goes there: where the model bytes come from, and why an OCI registry with ORAS, Harbor, MLflow or the Kubeflow Model Registry is the difference between distributing models and scattering them.

Sources

CNCF, KServe becomes a CNCF incubating project (11 de noviembre de 2025) — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
CNCF, KServe project page (incubating, aceptado el 29 de septiembre de 2025) — https://www.cncf.io/projects/kserve/
CNCF TOC, Project Moving Levels Checklist: KServe joining CNCF at Incubation level — https://github.com/cncf/toc/issues/1905
KServe, Releases (v0.15.0, v0.16.0, v0.18.0, v0.19.0) — https://github.com/kserve/kserve/releases
KServe, Inference Protocol V2 (Open Inference Protocol) — https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol
KServe, open-inference-protocol (especificación, adoptantes, gobernanza) — https://github.com/kserve/open-inference-protocol
KServe, open-inference-protocol — generate_rest.yaml — https://github.com/kserve/open-inference-protocol/blob/main/specification/protocol/generate_rest.yaml
open-inference, Vendor neutral fork of kserve/open-inference-protocol — https://github.com/open-inference/open-inference-protocol
KServe, Generative Inference — Runtime Overview (endpoints con prefijo openai/) — https://kserve.github.io/website/docs/model-serving/generative-inference/overview
KServe, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
KServe, Cloud-Native AI Inference at Scale using KServe and llm-d (5 de marzo de 2026) — https://kserve.github.io/website/blog/cloud-native-ai-inference-kserve-llm-d
KServe, Serving Runtime (ServingRuntime y ClusterServingRuntime) — https://kserve.github.io/website/docs/concepts/resources/servingruntime
KServe, Kubernetes Deployment Installation Guide (K8s 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1, sin scale-from-zero en Standard) — https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment
KServe, Canary Rollout Strategy — https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary
KServe, Autoscaling with KEDA — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/keda-autoscaler
KServe, Multi-node/Multi-GPU Inference (PVC RWX, sin autoescalado) — https://kserve.github.io/website/docs/model-serving/generative-inference/multi-node
KServe, Storage Options for Model Artifacts — https://kserve.github.io/website/docs/model-serving/storage/overview
KServe, issue #5335 — Support canaryTrafficPercent for RawDeployment mode via Gateway API HTTPRoute weights (abierta, 2 de abril de 2026) — https://github.com/kserve/kserve/issues/5335
KServe, issue #5062 — Add startupProbe to vLLM main containers (12 de febrero de 2026) — https://github.com/kserve/kserve/issues/5062
KServe, modelmesh-serving (última release v0.12.0 de julio de 2023; repositorio archivado el 14 de abril de 2026) — https://github.com/kserve/modelmesh-serving
Red Hat, Converting ModelMesh and Serverless InferenceServices to RawDeployment (Standard) Mode — https://access.redhat.com/articles/7134025
NVIDIA, Triton Inference Server — Inference Protocols and APIs — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inference_protocols.html
OpenVINO, What is OpenVINO Model Server (API KServe, TFS y endpoints compatibles con OpenAI) — https://docs.openvino.ai/2026/model-server/ovms_what_is_openvino_model_server.html
PyTorch, TorchServe — Notice: Limited Maintenance — https://docs.pytorch.org/serve/
Kubernetes SIG Network, Gateway API Inference Extension (v1.5.0, 19 de abril de 2026) — https://github.com/kubernetes-sigs/gateway-api-inference-extension

Model chain of trust (2/4): where the bytes live — registry, OCI artefacts and distribution to the GPU node

Sun, 26 Jul 2026 19:00:00 +0200

In the first article in the series we described the control plane: who declares that an inference endpoint exists, who reconciles it and what contract the clients speak. But an InferenceService is no more than a promise until somebody puts 140 GB of weights on the filesystem the container sees. This second article is about exactly that: where those bytes come from and by what route they reach the GPU’s memory.

It is a subject the blog has touched around the edges, in the cold start series, in data versioning with DVC and lakeFS, in the catalogue of OSS LLMOps tools, but never head on. It is time to, because almost every on-premise inference platform that gets audited shares the same original defect, and it is conceptual before it is about performance: they conflate three things that live in different places and fail in different ways.

TL;DR

Three planes, not one. Model registry (metadata, versions, lineage, promotion) ≠ artefact repository (the bytes, addressed by digest) ≠ deployment storage (what the pod mounts at /mnt/models). Mixing them is what breaks reproducibility.
The anti-pattern: a Hugging Face token in the pod and a download from the internet on every start-up. No immutability, no cache, no audit trail, no air-gap and cold starts measured in minutes.
The OCI artefact is today’s pragmatic answer: OCI Image Spec 1.1 (March 2024) brought artifactType, subject and the Referrers API; ORAS v1.3.0 (October 2025) added backup/restore for air-gap; ModelPack (CNCF sandbox since June 2025) standardises the mediaTypes for weights.
Kubernetes already knows how to mount OCI artefacts: the image volume source (KEP-4639) reached GA in v1.36 (22 April 2026). It is the most operationally relevant change of the year for model distribution.
The real bottleneck is almost never the GPU. 140 GB over a saturated 10 Gb/s link is about 112 s; over Gen4 NVMe about 20 s; over PCIe Gen5 about 2-3 s. Optimising the loader without fixing the network is optimising the wrong link.
A model registry (MLflow, Kubeflow Hub) does not store the bytes: it stores the record card and a pointer. If you were expecting it to solve your distribution, you bought the wrong tool.

The analogy

The simile is a serious historical archive, with three things a distracted visitor constantly confuses.

The catalogue: record cards with a shelfmark, author, date, provenance, conservation state and consultation restrictions. The card weighs grams and describes something that weighs kilos. The repository: the compact shelving in the basement where the physical boxes are, with access control, temperature control and an inventory that balances to the gram; it knows nothing about why a document matters, but it guarantees that box 4711 still contains exactly what it contained. And the reading room table: where the researcher has the document open in front of them. A temporary and expensive place, from which the document goes back to the repository when the session ends.

The model registry is the catalogue. The artefact repository is the repository. The GPU node with the model in HBM is the reading room table. Anyone who says “we have set up a model registry” and what they have set up is MLflow has set up the catalogue and left the repository unbuilt: the boxes are still in somebody’s car boot. Putting the model inside the container image is the equivalent of putting the document inside the table: every time the table changes, the document moves. And anyone who downloads from the internet on every start-up has no repository: they order the document by courier from another city every time a researcher sits down.

We will come back to the archive later, with the weeding that deletes boxes that are still referenced, the trolley that carries things between the basement and the reading room, and the certified reproduction sent to the isolated site.

Three planes that are not the same

Before discussing tools you have to fix the vocabulary, because a good part of the architecture discussions at this layer are really discussions about nomenclature.

Plane	What it stores	Unit	Question it answers	OSS examples
Model registry	Metadata, logical versions, lineage, promotion state, evaluation metrics	The model’s record card (KB)	Which version is approved for production and where did it come from?	MLflow, Kubeflow Hub, ClearML
Artefact repository	The bytes, addressed by digest, immutable	The blob (GB-TB)	What exactly are the bytes of that version?	Harbor, `distribution`, Zot, MinIO, lakeFS
Deployment storage	The copy the pod sees mounted	The volume (`/mnt/models`)	What is the inference process reading right now?	PVC, image volume, emptyDir + NVMe cache

The three layers have different life cycles. A record card lives forever (audit); a blob, for as long as some version references it (retention); a volume, for as long as the pod lives. Implemented as a single thing, it inherits the worst of the three: if the registry is also the deployment storage, deleting an old version takes down a pod in production; if the deployment storage is also the canonical reference, there is no way to answer what was served last Tuesday.

Hence the operational rule that gets written on the wall: the source of truth for the bytes is a digest, not a tag and not a path. Everything else (tags, aliases, PVC paths) is a mutable pointer that resolves to that digest at a given moment. Article 3/4 builds all of its provenance verification on that premise.

The anti-pattern we start from

This is, by a distance, the most widespread pattern in on-premise deployments that started as a pilot and stayed that way:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-anti-patron
spec:
 template:
 spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.11.0
 args: ["--model", "meta-llama/Llama-3.3-70B-Instruct"]
 env:
 - name: HF_TOKEN
 valueFrom:
 secretKeyRef:
 name: hf-token
 key: token
 resources:
 limits:
 nvidia.com/gpu: "4"

It works and it serves tokens. And it is unacceptable on a sovereign platform for five independent reasons, any one of which would be enough:

An external dependency on the critical start-up path. A Deployment that cannot recover from a failure without outbound internet access carries a third-party dependency that nobody has declared in the SLA.
Zero immutability. meta-llama/Llama-3.3-70B-Instruct is a mutable reference, and a repository’s default branch changes (tokenizer fixes, chat templates). Two replicas of the same Deployment started a week apart can serve different bytes, and nothing in the cluster says so.
Zero coordinated cache and zero audit trail. Every pod downloads its own copy: eight replicas of a 140 GB model is 1.1 TB of outbound traffic per rolling update. And no internal record remains of what was downloaded, with which digest or who authorised it; faced with an ENS or ISO 42001 auditor, the answer is “we got it off the internet”.
Incompatible with air-gap. The whole design collapses when the platform has to be replicated in an isolated environment, which is the defence and healthcare case dealt with in regulatory compliance infrastructure.
Cold start measured in minutes. Downloading 140 GB over a 1 Gb/s outbound link (125 MB/s effective, being generous) is about 19 minutes of transfer alone, before touching the disk or the GPU; with 10 Gb/s dedicated and saturated, about 112 seconds. In a load-driven scaling event, that is service latency.

The blog’s cold start series explores how to trim the disk-to-HBM leg. What this article adds is that if the first leg is the internet, the rest of the optimisation is irrelevant.

The model as an OCI artefact

The pragmatic answer is boring, and that is why it works: the model is stored in the same registry where the containers are already stored. There is already access control, cross-site replication, scanning, quotas, audit and a team that knows how to operate it.

What OCI 1.1 made possible

For years, putting things that were not container images into an OCI registry was a bodge. The OCI Image Spec 1.1 and Distribution Spec 1.1 specifications, published on 13 March 2024, turned it into a first-class use case with three pieces:

artifactType in the manifest: it explicitly declares “this is not a runnable image, it is an artefact of type X”. Previously this was encoded by abusing the config mediaType, which is why even today there are registries that show a model as a broken image.
subject: a manifest can declare that it refers to another manifest. That is what lets you hang a signature, an SBOM or an attestation off a model without modifying it and without changing its digest.
Referrers API (GET /v2/<repo>/referrers/<digest>): the reverse query, “give me everything that points at this digest”, with a tag-based fallback for registries that do not implement it yet.

Article 3/4 builds signing, provenance and AIBOM on top of these three pieces. Without them, verifying a model forces you to invent a convention of your own.

ORAS: the generic client

ORAS (OCI Registry As Storage) is the Swiss army knife for getting arbitrary artefacts in and out of an OCI registry. Version v1.3.0 was published on 6 October 2025, conformant with distribution-spec v1.1.1, and added three things relevant to models: backup/restore to a directory or tarball (key for air-gap), management of multi-platform indexes and --format for structured output.

One data point to calibrate maturity: ORAS entered the CNCF sandbox on 13 July 2021 and as of this article is still in sandbox. Five years at the entry level does not invalidate the tool, which is stable and the de facto reference, but it says something about the project’s governance that is worth weighing before turning it into a critical dependency with no plan B.

Pushing a model with ORAS, with explicit content types:

oras push registro.interno/modelos/llama-3.3-70b:2026.07.1 \
 --artifact-type application/vnd.cncf.model.manifest.v1+json \
 --annotation "org.opencontainers.image.created=2026-07-20T09:00:00Z" \
 --annotation "es.ejemplo.modelo.precision=bf16" \
 --annotation "es.ejemplo.modelo.origen=hf:meta-llama/Llama-3.3-70B-Instruct" \
 config.json:application/vnd.cncf.model.weight.config.v1.raw \
 tokenizer.json:application/vnd.cncf.model.weight.config.v1.raw \
 model-00001-of-00030.safetensors:application/vnd.cncf.model.weight.v1.raw \
 model-00002-of-00030.safetensors:application/vnd.cncf.model.weight.v1.raw

# Retrieve it, with a local blob cache so nothing already present is re-downloaded
export ORAS_CACHE=/var/cache/oras
oras pull registro.interno/modelos/llama-3.3-70b:2026.07.1 --output /srv/modelos/llama-70b
oras manifest fetch registro.interno/modelos/llama-3.3-70b:2026.07.1 --descriptor

ModelPack: the missing convention

ORAS lets you invent the mediaTypes, with the predictable result that every organisation invents its own and nothing interoperates. ModelPack is the attempt to standardise them, accepted into the CNCF sandbox in June 2025. On top of OCI Image Manifest 1.1 it defines:

artifactType: application/vnd.cncf.model.manifest.v1+json
config: application/vnd.cncf.model.config.v1+json
weight layers: application/vnd.cncf.model.weight.v1.raw (and the .tar, .tar+gzip, .tar+zstd variants)
weight configuration, documentation, code and dataset layers following the same pattern

Its reference CLI is modctl, which works with a declarative Modelfile:

NAME llama-3.3-70b-instruct
ARCH transformer
FAMILY llama
FORMAT safetensors
PARAMSIZE 70
PRECISION bf16
CONFIG config.json
CONFIG tokenizer.json
MODEL *.safetensors
DOC *.md

modctl build -t registro.interno/modelos/llama-3.3-70b:2026.07.1 -f Modelfile .
modctl push registro.interno/modelos/llama-3.3-70b:2026.07.1
modctl pull registro.interno/modelos/llama-3.3-70b:2026.07.1 \
 --extract-dir /srv/modelos/llama-70b --extract-from-remote

The alternative with more traction among users is KitOps (ModelKit), with Docker-like ergonomics and native integration with Hugging Face. As of this article there is no convergence: ModelPack aims at the enterprise case and at integration with runtimes and registries; KitOps and Docker Model Runner, at developer ergonomics. Picking one today is a bet; using artifactType and explicit annotations instead of home-grown conventions is not.

How a 140 GB model is sliced up

A large model is 20-40 safetensors shards plus half a dozen small configuration files, and the layering strategy matters:

One layer per shard, uncompressed. Weights in bf16 or fp8 are incompressible in practice: gzip over safetensors burns CPU to save 2-3 %, and forces decompression on the node, adding a full pass through CPU and disk on the critical path. The right type is .raw.
The small files in their own layer. config.json, tokenizer.json and the templates change far more often than the weights. If they go in the same layer as a 5 GB shard, every tokenizer fix invalidates 5 GB of cache on every node.
Layer size between 2 and 8 GB. Large enough not to pay per-blob overhead, small enough to parallelise and so that a network failure does not force you to retry 40 GB.

In practice three things hurt. The upload uses the distribution-spec’s chunked upload, with historically uneven interoperability between implementations, so pushing 140 GB is an operation measured in tens of minutes that demands a decent link and retries. Deduplication is by layer digest, so two versions that differ in one shard share the rest only if those blobs are byte-for-byte identical, and any repackaging that alters metadata breaks it. And the default values of reverse proxies (timeouts, maximum body size, upload session lifetime) are calibrated for 1 GB images, not for 8 GB blobs.

Harbor as the on-premise repository

Harbor is the reference OSS registry for on-premise: a CNCF graduated project since June 2020, with a healthy release cycle (the 2.15 series came out on 20 March 2026 and patch 2.15.2 on 2 July 2026) and support for the last three minors for about nine months each. What matters for models:

Cross-site replication. Push or pull rules, filtered by repository, tag and resource type, with manual, scheduled or event-driven triggering. For models, scheduled and within a window, with a label marking which versions get replicated (see traps).

Proxy cache. Here you have to be precise, because a lot of misinformation circulates: Harbor’s proxy cache works against OCI registries (Harbor, Docker Hub, a distribution registry, ECR, ACR, GCR, Quay and GHCR) and not against Hugging Face, which does not expose an OCI distribution API. If the idea was “I put Harbor in front of Hugging Face and that is it”, it does not work that way. The real options are (a) an ingestion task that pulls from Hugging Face in a staging environment and pushes to Harbor as an OCI artefact, the recommended pattern because it introduces an explicit control point, or (b) a P2P accelerator with native support for model repositories, such as Dragonfly. What the proxy cache does solve is mirroring the runtime base images (vLLM, TGI, TEI), which is no small thing: it creates a 7-day retention by default and the project is read-only.

Per-project quotas. In bytes, via the UI or the API:

curl -u admin:REDACTED -X PUT "https://registro.interno/api/v2.0/quotas/7" \
 -H "Content-Type: application/json" \
 -d '{"hard": {"storage": 21990232555520}}'

Those 21990232555520 bytes are 20 TiB, tight sizing, not generous, for a dozen models with three live versions each. Two warnings: the quota is checked when the manifest arrives, that is after uploading the blobs, so a push that exceeds it has already consumed disk and network; and the accounting does not always reflect deduplication of blobs shared between projects.

Immutable tags. Per-project rules that prevent overwriting a tag matching a pattern. For models it is not optional: a rule making everything under modelos/** with tag pattern 2* (date-based versions) immutable eliminates the entire class of “the tag changed underneath us” incidents.

Retention and GC. Per-project retention rules (last N versions, or those from the last N days) that mark artefacts for deletion, and a separate garbage collector that frees the unreferenced blobs. They are two different things, and deleting in the UI does not free disk: the weeding runs separately. Running GC with a concurrent push in flight is the classic recipe for deleting blobs that were about to be referenced.

Scanning. Trivy is integrated, with one caveat: scanning a weights artefact detects nothing relevant about the model. It is useful for the runtime images, which do execute code. Detecting dangerous formats, a PyTorch pickle with arbitrary code instead of safetensors, requires specific tools, and that is the subject of article 3/4.

Model registries proper

Now the catalogue, with the uncomfortable claim up front: a model registry does not solve distribution.

MLflow

MLflow (LF AI & Data) is the de facto catalogue standard, with a mature 3.x series; 3.13.0 was published in June 2026, focusing on RBAC, an administration UI and trace retention. Its data model: registered model (logical name) → model version (incrementing integer) → aliases and tags.

The important conceptual change: stages (Staging, Production, Archived) have been deprecated since MLflow 2.9.0 and are replaced by aliases and tags, because a fixed enumeration is too inflexible to express real MLOps flows. An alias is a named mutable pointer:

from mlflow import MlflowClient

client = MlflowClient()
client.set_registered_model_alias("llama-3.3-70b-soberano", "champion", 7)
client.set_model_version_tag("llama-3.3-70b-soberano", "7",
 "estado_validacion", "aprobado_ens_medio")
client.set_model_version_tag("llama-3.3-70b-soberano", "7",
 "oci_digest", "sha256:9f2c...")

Consumption happens through the alias URI: models:/llama-3.3-70b-soberano@champion.

What MLflow gives you: lineage back to the run that produced the model, evaluation metrics attached to the version, a shared promotion language and an API to hang approvals off. What it does not give you: it is not a CDN and it does not manage distribution to nodes, and its artifact store (S3/MinIO/NFS) provides neither immutability by digest, nor Referrers, nor cross-site replication. The healthy pattern is the one in the snippet: the catalogue stores the OCI digest as a tag on the version and points at the repository, instead of trying to be the repository.

Kubeflow Hub (formerly Model Registry)

The Kubeflow Model Registry was renamed Kubeflow Hub and unified two functions: the registry itself and a federated Model Catalog that discovers external models (YAML, Hugging Face), with lineage across data, code and models, metadata and state-based promotion.

The honest assessment: as of this article the component is still in the 0.3.x series and marked Alpha under Kubeflow’s versioning policy, with limited support. Interesting if you already run the full Kubeflow and want a single console; it is not yet the piece on which to build model governance for a sovereign platform in production. The rename in the middle of 0.3.x is, moreover, a sign that the API is still moving.

Comparison table by function

Function	OCI registry (Harbor + ORAS/ModelPack)	MLflow Registry	Kubeflow Hub	lakeFS / DVC
Stores the bytes	Yes, by digest, immutable	No (pointer to an artifact store)	No (pointer)	Yes (data and versions)
Lineage to data/code	Only via annotations	Yes, to the run	Yes, explicit	Yes, for datasets
Rich metadata	Annotations and JSON config	Yes, typed	Yes, typed	Partial
Promotion / approval	By convention (tags, labels)	Aliases and tags	States	No
Signing and attestations	Yes, native (`subject` + Referrers)	No	No	No
Cross-site replication	Yes, native	No	No	Partial
Standard API	OCI Distribution 1.1	Its own REST	Its own REST	S3/Git-like
Air-gap	Yes (`oras backup`/`restore`, `skopeo`)	Manual	Manual	Manual
Maintainer	CNCF (Harbor graduated; ORAS and ModelPack sandbox)	LF AI & Data	Kubeflow (CNCF)	Commercial OSS
Real maturity	High (registry), medium (model convention)	High	Alpha	High

The thesis of the table: the columns are complementary, not alternatives. The OCI registry is the repository; MLflow, the catalogue; lakeFS or DVC version the dataset that everything originates from, as detailed in data versioning with DVC and lakeFS.

The hot path: from the registry to HBM

Image volumes: the change of the year

The image volume source (KEP-4639) mounts an OCI image or artefact directly as a read-only volume in a pod: no initContainer, no copying, no PVC. Its track record: alpha in v1.31 (August 2024), beta in v1.33 (April 2025) with subPath support, beta by default in v1.35 and GA in v1.36, released on 22 April 2026.

apiVersion: v1
kind: Pod
metadata:
 name: vllm-modelo-oci
spec:
 containers:
 - name: vllm
 image: registro.interno/runtime/vllm:v0.11.0
 args: ["--model", "/mnt/models", "--served-model-name", "llama-3.3-70b"]
 volumeMounts:
 - name: pesos
 mountPath: /mnt/models
 readOnly: true
 resources:
 limits:
 nvidia.com/gpu: "4"
 volumes:
 - name: pesos
 image:
 reference: registro.interno/modelos/llama-3.3-70b@sha256:9f2c...
 pullPolicy: IfNotPresent

Two details that change operations: the reference is by digest, so immutability stops depending on anyone’s discipline; and pullPolicy: IfNotPresent makes the node’s runtime cache the blobs in its local store, so the second replica on that node starts without touching the network. It is the coordinated cache the anti-pattern was missing.

KServe: `storageUri` and modelcars

In KServe there are two routes: the classic one, storageUri with a storage initializer that copies from S3, PVC, HTTP or GCS into an emptyDir; and the modelcar, with the oci:// scheme:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: llama-70b-soberano
spec:
 predictor:
 model:
 modelFormat:
 name: huggingface
 storageUri: oci://registro.interno/modelos/llama-3.3-70b:2026.07.1
 resources:
 limits:
 nvidia.com/gpu: "4"

You have to understand the modelcar’s internal mechanism before depending on it: KServe sets shareProcessNamespace: true on the pod, starts a sidecar container with the model image and creates a symbolic link from /mnt/models to that process’s root filesystem through /proc. The runtime reads the weights without copying them, which is the saving being sought. Two warnings: it is not enabled by default (you have to turn on enableModelcar in the inferenceservice-config ConfigMap) and the latest tag forces pullPolicy: Always, cancelling out the cache. With image volumes now GA, the modelcar remains the route for clusters that are not yet on 1.36.

Loading fast: where each technique attacks

You have to separate two problems that get confused all the time: getting the bytes to the node (network) and getting them into HBM (disk → CPU → PCIe → GPU).

For the first, Dragonfly is the mature piece: it graduated in CNCF on 14 January 2026, distributes weights at the scale of hundreds of terabytes to hundreds of nodes “in minutes”, cuts the bandwidth consumed at the origin by up to 90 %, and makes use of between 70 % and 80 % of each node’s bandwidth. Its v2.5.0 (30 June 2026) added direct download from model repositories (dfget hf://...), P2P acceleration of Git LFS and a dragonfly-injector that injects P2P capability via a webhook without rebuilding images. Its subproject Nydus supplies the lazy-loading image format. There is a very common misplaced expectation here: for model weights lazy loading helps little, because an inference engine reads all the weights almost immediately; the real benefit is for fat runtimes and for models where only part is read.

For the second, the numbers published by the NVIDIA Run:ai Model Streamer on Llama-3-8B (15 GB, safetensors, a node with one A10G) are the cleanest reference available:

Source	Safetensors loader	Run:ai Model Streamer	Tensorizer
SSD gp3	47.99 s	14.34 s (concurrency 16)	16.11 s (16 workers)
SSD io2	47.0 s	7.53 s (concurrency 8)	10.36 s (8 workers)
S3 object storage	not supported	4.88 s (concurrency 32)	37.36 s (16 workers)

And the total time until vLLM is ready to serve: 66.13 s with the standard loader over gp3 versus 35.08 s with the streamer. Two critical readings. First: these are the project’s own benchmarks, not an independent measurement, on a 15 GB model on a modest GPU; extrapolating linearly to 140 GB on 4×H100 is a leap nobody has published. Second: the gain comes from saturating the storage medium with concurrency. If the bottleneck is an NFS at 1 Gb/s, no loader fixes it. The detail of this leg is in speeding up cold start with tensorizer and in from disk to HBM.

As the diagram shows, on a reference node of 4×H100 SXM 80 GB with NVLink each leg is between 5 and 10 times faster than the previous one. The conclusion is boring: the money is in having the model already on the node’s NVMe before the pod starts.

LoRA adapters: when the artefact weighs megabytes

Everything above assumes artefacts of tens or hundreds of gigabytes. A LoRA adapter breaks that assumption: for a 70B base, a rank-16 adapter over the attention projections weighs tens or a few hundred megabytes, three orders of magnitude less, and that changes the whole strategy.

With artefacts like that, the local cache and P2P stop mattering: the download is instant at any reasonable bandwidth. What starts to matter is cardinality and rate of change. A multi-tenant platform like the one described in multi-LoRA serving can have hundreds of adapters, each with its owner and its access policy, over a handful of bases. The practical consequences:

The adapter is an OCI artefact in its own right, with its own repository and an immutable tag. But the registry then has thousands of small repositories instead of tens of large ones, which stresses metadata, listings and granular RBAC, not disk.
It must declare its base. An adapter with no immutable reference to the digest of the base model it was trained on is a time bomb: applied over another version, it degrades quality silently. It is the natural use case for subject: the adapter refers to the base, and the Referrers API answers “which adapters exist for this digest”.
Loading is dynamic, not at start-up. vLLM allows hot loading of adapters with VLLM_ALLOW_RUNTIME_LORA_UPDATING and the /v1/load_lora_adapter endpoint, or declaratively with the LoRA resolver plugins (VLLM_PLUGIN_LORA_RESOLVERS), which resolve an adapter by name from a directory. The clean pattern on Kubernetes is to mount an image volume with the adapter and let the filesystem resolver discover it.
Versioning becomes the dominant problem. With 300 live adapters, knowing which adapter serves tenant X, over which base and trained with which dataset, only has an answer if the catalogue exists. Here a model registry does beat convention over tags.

Air-gap and sovereignty: promotion without losing traceability

The strict sovereign case: a connected environment where you ingest and validate, and an isolated environment where you serve, with no network route between them. Promotion crosses a data diode or a physical medium, and traceability has to survive the journey. It is the archive’s certified reproduction: the document, its record card and the seal certifying that the copy is faithful all travel together.

# 1. In the connected environment: export the artefact with EVERYTHING that hangs off it
oras backup \
 --output /transfer/llama-3.3-70b-2026.07.1.tar \
 --include-referrers \
 registro-dmz.interno/modelos/llama-3.3-70b:2026.07.1

# 2. Verify the exact digest before crossing
oras manifest fetch --descriptor \
 registro-dmz.interno/modelos/llama-3.3-70b:2026.07.1 > /transfer/descriptor.json
sha256sum /transfer/llama-3.3-70b-2026.07.1.tar > /transfer/SHA256SUMS

# 3. Physical crossing or data diode

# 4. In the isolated environment: restore preserving digests and referrers
oras restore \
 --input /transfer/llama-3.3-70b-2026.07.1.tar \
 registro-aislado.interno/modelos/llama-3.3-70b

The --include-referrers flag is what makes this worth anything: without it the weights cross, but the signature, the AIBOM and the attestations stay behind, and the isolated environment receives a model indistinguishable from one downloaded by hand. oras backup is marked experimental in v1.3.0, which has to be accepted explicitly before turning it into a procedure; the consolidated alternative for pure copying is skopeo copy with the oci-archive format, at the cost of managing the referrers separately.

What must travel attached to the artefact so that article 3/4 can verify it: the digest of the artefact and of its blobs; the provenance of the origin (the repository and the specific revision, not the model’s name, plus who ingested it and when); the identity of the process that packaged it (pipeline, commit, run), the raw material for an SLSA attestation; the cross-references to the evaluation dataset and to the base model if it is a derivative; and the promotion state with its approver, which in an isolated environment cannot be queried against the connected environment’s MLflow. All of it fits in OCI annotations and in artefacts referred to via subject. The rule: the isolated environment must be able to verify without calling anyone; if verification needs an external query, it is not air-gap.

Decision map

An OCI registry and a convention are enough when you have fewer than twenty models or so, a single team decides what gets promoted, lineage back to training is not a regulatory requirement (typical if you consume third-party models) and you already operate Harbor. Here a model registry adds one more console to maintain and no answer that immutable tags and annotations were not already giving. It is the majority case in pure inference.

A model registry is needed if you train or fine-tune and need auditable lineage from model to dataset and to code, the continuous fine-tuning cycle; if several teams compete to promote versions and an explicit approval flow is needed; if you have to answer an ISO 42001 or EU AI Act auditor about which version was in production on a given date; or if you manage tens of adapters with different owners.

The registry becomes valueless bureaucracy when the model is registered after being deployed (the catalogue as an administrative act, not as a gate); when the promotion state is not connected to any technical control and an unapproved version can be deployed; or when a model registry and an OCI registry coexist with two truths and no digest joining them. The last one is the most common failure and the most expensive: two versioning systems that diverge in silence.

Operational traps

Mutable tags and latest in production. Beyond style, a mutable tag turns any audit into guesswork and lets two replicas of the same Deployment serve different bytes. On top of that, in KServe latest forces pullPolicy: Always and cancels the node cache, so you pay in cold start what you lose in traceability. Rule: immutable tags by project policy and deployment by digest.

GC that deletes referenced layers. The archive’s weeding. The garbage collector identifies blobs that no manifest references, and the window between “I have uploaded the blobs” and “I have uploaded the manifest” is exactly where a concurrent GC can take them away. In Harbor this has historically shown up in _uploads and in discrepancies between the space the UI claims to have freed and the real figure. Mitigation: GC in a window with pushes blocked, and do not trust the first pass.

The registry’s disk and the cost of versioning. Brutal arithmetic: 140 GB per version with three live versions is 420 GB; twelve models like that, 5 TB; with six-version retention and two replicated sites, 20 TB. And it is expensive-class storage if the registry lives on replicated block. Deduplication helps little, because the weights change completely between versions. The retention policy has to be designed before filling the registry, and models must be separated from runtime images by project, because their retentions have nothing to do with each other.

Replication that saturates the inter-site link. An event-driven rule on a model repository moves 140 GB every time someone pushes a version: on a shared 1 Gb/s link, almost twenty minutes flat out in the middle of the working day. Schedule it in a window, limit bandwidth, filter by label which versions get replicated and use P2P inside each site.

Putting the model inside the runtime image. Tempting because “that way there is only one artefact”. The result is a 145 GB image that has to be rebuilt and redistributed every time a runtime CVE is patched, with the security life cycle coupled to the model’s. Separated: a small, patchable runtime and a large, stable model.

For an inference factory

Three actionable things for anyone running an on-premise inference factory, not a training one.

First: cut the internet dependency at start-up, this week. You do not need a platform project: an ingestion task that pulls each model once, pushes it to Harbor as an OCI artefact with an immutable tag and provenance annotations, plus switching the deployments to reference by digest, is enough. The HF_TOKEN disappears from the pod and moves to the ingestion pipeline, which is where it makes sense and where it fits with secrets hardening.

Second: decide consciously whether you need a catalogue. If you only consume third-party models and do not train, probably not: the OCI registry with a convention gives 90 % of the value at 10 % of the operational cost. If you train, fine-tune or manage per-tenant adapters, set up MLflow and store the OCI digest as a tag on each version, so that catalogue and repository cannot diverge.

Third: move the bottleneck to the right place. Before investing in load accelerators, measure the three legs. If the model arrives over the network on every start-up, the work is a local NVMe cache, image volumes with pullPolicy: IfNotPresent and, with many nodes, P2P. Only when the model is already on the node’s disk does it make sense to wrestle with loader concurrency.

So much for where the bytes come from and by what route they arrive. What remains is what we have taken for granted all along: why trust those bytes. A digest guarantees integrity, that nobody has changed them, but not provenance: it does not say who produced them, with what data or whether somebody with authority certified that they could be served. The third article in the series builds that answer with Sigstore, SLSA, in-toto and AIBOM, hanging precisely off the subject and the Referrers API that we have left ready here.

Sources

Open Container Initiative, OCI Image and Distribution Specs v1.1 Releases (13 March 2024) — https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/
CNCF, Announcing ORAS v1.3.0: Elevating artifact and registry management workflows (6 October 2025) — https://www.cncf.io/blog/2025/10/06/announcing-oras-v1-3-0-elevating-artifact-and-registry-management-workflows/
CNCF, ORAS project page (sandbox since 13 July 2021) — https://www.cncf.io/projects/oras/
ORAS, oras backup (experimental) — https://oras.land/docs/commands/oras_backup
ModelPack, CNCF ModelPack Specification — model spec (artifactType and mediaTypes) — https://github.com/modelpack/model-spec
ModelPack, modctl — getting started (Modelfile and commands) — https://github.com/modelpack/modctl/blob/main/docs/getting-started.md
CNCF, Cloud Native Computing Foundation announces Harbor graduation (23 June 2020) — https://www.cncf.io/announcements/2020/06/23/cloud-native-computing-foundation-announces-harbor-graduation/
Harbor, Configure proxy cache (supported registries) — https://goharbor.io/docs/2.6.0/administration/configure-proxy-cache/
OneUptime, How to configure Harbor project quotas for storage limits (9 February 2026) — https://oneuptime.com/blog/post/2026-02-09-harbor-project-quotas-storage/view
endoflife.date, Harbor releases (2.15.2 on 2 July 2026; 2.15 on 20 March 2026) — https://endoflife.date/harbor
VMware Cloud Foundation Blog, Using Harbor as an AI Model Registry (3 March 2026) — https://blogs.vmware.com/cloud-foundation/2026/03/03/using-harbor-as-an-ai-model-registry/
MLflow, Model Registry workflows — stages deprecated since MLflow 2.9.0 — https://mlflow.org/docs/latest/ml/model-registry/workflow/
MLflow, Release 3.13.0 highlights — https://mlflow.org/releases/3.13.0/
Kubeflow, Hub (formerly Model Registry) — overview — https://www.kubeflow.org/docs/components/hub/overview/
kubernetes/enhancements, KEP-4639: OCI VolumeSource (alpha 1.31, beta 1.33/1.35, GA 1.36) — https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4639-oci-volume-source/README.md
PerfectScale, Kubernetes v1.36 release: OCI artifact volume support reaches GA (22 April 2026) — https://www.perfectscale.io/blog/kubernetes-v1-36-release
KServe, OCI storage (modelcar) — https://kserve.github.io/website/docs/model-serving/storage/providers/oci
CNCF, CNCF announces Dragonfly’s graduation (14 January 2026) — https://www.cncf.io/announcements/2026/01/14/cloud-native-computing-foundation-announces-dragonflys-graduation/
CNCF, Dragonfly v2.5.0 is released (30 June 2026) — https://www.cncf.io/blog/2026/06/30/dragonfly-v2-5-0-is-released/
NVIDIA Run:ai, Model Streamer benchmarks — https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/benchmarks.md
vLLM, LoRA adapters (dynamic loading and resolver plugins) — https://docs.vllm.ai/en/stable/features/lora/

LLM inference operators on Kubernetes: OME, vLLM Production Stack, NVIDIA Dynamo and llm-d

Mon, 18 May 2026 17:00:00 +0200

TL;DR

Serving an LLM in production is not running a binary: it is coordinating a model (tens of gigabytes that take minutes to load), a runtime (vLLM, SGLang, TensorRT-LLM with a hundred flags), heterogeneous GPUs (NVLink, MIG, PCIe), prefill and decode that live better apart, a KV cache that wants offloading to colder tiers, intelligent routing that exploits prefix caching, and autoscaling that reacts to metrics other than CPU%. A flat Kubernetes Deployment only covers the first 20% of that. The other 80% is covered by LLM inference operators, of which four matter in 2026: OME (LMSYS, July 2025, multi-engine with a focus on SGLang), vLLM Production Stack (a curated Helm chart from vLLM itself with LMCache for tiered KV), NVIDIA Dynamo (the official successor to Triton, multi-engine, with its own Grove scheduler) and llm-d (a CNCF donation of March 2026 by Red Hat + Google + IBM + CoreWeave + NVIDIA, built on vLLM, focused on distributed scale). Behind all four sits KServe, the CNCF parent operator that normalised the InferenceService concept and that several of them lean on. This article walks through the full hierarchy, gives a decision map and teaches you not to get lost when someone drops seven acronyms in the first meeting.

This article closes the LLM inference series. The previous ones were KV cache: the working memory that holds up LLM inference, vLLM on Kubernetes: the LLM inference piece that actually scales and PagedAttention from the inside and the state of the art of the KV cache in 2026. There we explained what happens inside one inference process. Here we explain how many inference processes are coordinated through Kubernetes.

The analogy: from `init.d` to systemd to operators

Anyone with 20 years of sysadmin behind them will recognise the pattern. Decades ago, starting a service on Linux was a shell script in /etc/init.d/: start, stop, status, reloaded by hand. When services got more complex, with dependencies between them, monitoring, restart on failure, per-user slots, it became obvious that a script was not enough. Along came systemd, which turned “a service” into a declarative unit with dependencies, resources, restart policy, sockets and timers. The script did not disappear; it moved up one level of abstraction.

Kubernetes made the same move for distributed services. A Deployment declares “I want N replicas of this container”; a Service declares “these replicas are exposed like this”; an Ingress declares “this HTTP traffic comes in here”. The controller translates the declaration into real state and keeps the system converged.

Serving LLMs in 2024 was the equivalent of /etc/init.d/: every team wrote its own Deployment/Service/HPA with custom scripts for model loading, session draining and GPU handling. We covered it in the vLLM on Kubernetes article: it can be done, and it does work, but it is repetitive, fragile and nobody is extracting the right abstractions. Serving LLMs in 2026 has gone through the same transition as services: the equivalent of systemd has appeared, the inference operators, which normalise the abstractions and let the engineer declare what matters: “this model, with this runtime, this scalable, with this routing policy”.

There are four relevant operators in 2026 and a fifth common ancestor. Let us take them in order.

Why an operator, and not just a Deployment

Listing what an inference operator adds over a flat Deployment is the best way to understand what problem it solves:

The model as a first-class citizen. In a Deployment, the model is “the thing you download in an initContainer and mount as a volume”. In an operator, the model is a CustomResource with metadata (origin, fingerprint, licence, GPU requirements). It can be shared between InferenceServices, versioned, replicated to multiple nodes. It is the difference between “a file” and “a managed artifact”.

The runtime as a first-class citizen. The same goes for the runtime (vLLM/SGLang/TRT-LLM): it is not “a Docker image with flags”; it is a ServingRuntime that declares which args it accepts, which metrics it exports, which deployment types it supports (single-node, multi-node TP, PD-disag). Changing runtime is changing a reference, not rewriting every manifest.

Declarative composition. An InferenceService (the core CRD of KServe and its descendants) references a model and a runtime, declares the scaling policy, wires up observability and configures routing. The controller composes all the pieces: Deployment(s), Service, HPA, possibly a LeaderWorkerSet, a KEDA ScaledObject, a Gateway API HTTPRoute. You declare intent; the operator emits the 8 derived resources.

Operational prefill–decode disaggregation. As we saw in the PagedAttention article, separating prefill and decode into distinct pools can give 7× goodput. Modelling that with flat Deployments is feasible, but it requires coordinating two sets of pods, a transport to move the KV cache and conditional routing. An operator models it as a single InferenceService with two sub-pools.

Autoscaling on LLM metrics. The standard HPA does not understand vllm:num_requests_waiting. An operator integrates KEDA or Prometheus Adapter automatically and exposes the right metrics as knobs on the CRD.

Multi-tenancy. Multiple models on the same cluster, with quotas, priorities and fairness. One Deployment per model scaling independently is fine up to the fifth model; beyond that, coordinating GPUs between tenants becomes operationally hostile.

Model lifecycle. Weights on a shared PVC, warm-up of the first pod, rolling updates with maxUnavailable: 0, draining of active sessions, integrated observability. Things that with a flat Deployment have to be reinvented by every team.

If your workload is one model, one node, up to three replicas, a flat Deployment is enough and an operator is overkill. If your workload is two or more models, serious scaling, disaggregation or multi-tenancy, an operator stops being optional.

KServe: the common ancestor

Before the four new ones, we have to mention KServe, the parent operator from which all the rest conceptually descend. It was born as KFServing inside the Kubeflow project in 2019, was renamed KServe when it became independent in 2021, and in 2025 was accepted into the CNCF as a project incubating towards graduation.

KServe’s conceptual contribution is the InferenceService CRD, which has become the field’s common vocabulary: a declarative K8s object that joins a model (origin + metadata) with a predictor (runtime + resources) and produces a ready HTTP service. Under the hood, the controller emits Deployments, Services, HorizontalPodAutoscalers, Knative Services if you go serverless, Istio VirtualServices if you do traffic splitting.

KServe was designed in a pre-LLM era: its first use cases were traditional scikit-learn, TensorFlow and PyTorch models served as simple REST APIs. That gives it strengths (it is mature, with 6 years in production at Bloomberg, JPMorgan and others) and weaknesses (it was not designed to manage multi-node tensor parallel, prefill–decode disaggregation, or the patterns specific to LLMs).

The way the ecosystem has reacted is elegant: the new LLM operators inherit from or take inspiration from InferenceService but extend the API with LLM-specific primitives. OME is the clearest example: it uses the name InferenceService and the idea of “model + runtime → service”, but adds BaseModel, a ServingRuntime with LLM-aware flags, and deployment modes (PD-disag, multi-node) that KServe does not cover natively.

OME (Open Model Engine)

OME was published by the LMSYS team in July 2025 (announced on their blog). It is an operator that understands SGLang deeply (that is its first-class runtime) but also supports vLLM, TensorRT-LLM and Triton.

The CRD hierarchy

OME models the domain with four main CRDs:

BaseModel and ClusterBaseModel: the model itself. It defines origin (Hugging Face, S3, URL), fingerprint and metadata. The Cluster* version is global; BaseModel is namespaced. It lets multiple InferenceService objects reference the same model without duplicating the download.
FineTunedWeight: LoRA adapters or finetuned weights served on top of a BaseModel. Critical for multi-tenant setups where each customer has their own finetune.
ServingRuntime and ClusterServingRuntime: the runtime (vLLM, SGLang, and so on) with its configuration. It declares which args it accepts, which metrics it exports, which deployment modes it supports.
InferenceService: the central declarative piece, joining BaseModel + ServingRuntime + infrastructure.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
 name: llama3-70b-prod
 namespace: inference
spec:
 model:
 name: meta-llama-3-70b-instruct  # reference to a BaseModel
 runtime:
 name: sglang-h100  # reference to a ServingRuntime
 deploymentMode: PrefillDecodeDisaggregated  # standard | PD | MultiNode | Serverless
 prefill:
 minReplicas: 2
 maxReplicas: 8
 resources:
 requests:
 nvidia.com/gpu: 4
 decode:
 minReplicas: 4
 maxReplicas: 16
 resources:
 requests:
 nvidia.com/gpu: 1
 router:
 type: cache-aware  # SGLang router with cache awareness
 autoscaling:
 metricSource: keda
 metrics:
 - type: prometheus
 metricName: vllm_requests_waiting
 threshold: "10"

That is what the operator takes as input. The output is roughly 8 derived resources that would be horrible to declare by hand: two LeaderWorkerSets (one per prefill/decode pool), two Services, a Deployment for the router, KEDA ScaledObjects for each pool, a Gateway API HTTPRoute, and a PriorityClass that hooks into Kueue for gang scheduling.

The four deployment modes

OME materialises the InferenceService differently depending on deploymentMode:

Standard: one Deployment with N replicas; the classic. For small or single-GPU models.
PrefillDecodeDisaggregated: two coordinated pools; the SGLang router routes between them.
MultiNode: tensor parallel across multiple nodes via LeaderWorkerSet, with NCCL/InfiniBand. For models above 70B where a single node is not enough.
Serverless: Knative-style scale-to-zero. For sporadic workloads where the cost of keeping GPUs powered up does not pay off. The trade-off: the first request pays the model’s cold start cost (minutes).

Integration with the K8s ecosystem

OME does not invent primitives where they already exist. It leans on:

Kueue for gang scheduling: all the pods of a tensor parallel group have to start at once or not at all; Kueue guarantees it.
LeaderWorkerSet (LWS) for multi-node: workers join the leader’s Ray cluster, with an atomic lifecycle (one going down restarts the group).
KEDA for autoscaling on LLM-specific Prometheus metrics (queue depth, GPU cache usage, TTFT p95).
Gateway API and its Inference Extension for advanced routing (model-aware, prefix-aware, weighted canary).

The consequence: OME feels “idiomatically Kubernetes”. It does not introduce new concepts where they are not needed; it uses standard primitives and concentrates on what is specific to the LLM domain.

When to choose it

OME is the natural choice if SGLang is your main runtime and/or if you come from the KServe ecosystem and want an idiomatic evolution of it. It is mature but relatively young (a year old at the time of writing); expect rough edges on advanced features.

vLLM Production Stack

vLLM Production Stack is the official project from vLLM itself for production on Kubernetes. Its philosophy is the opposite of OME’s: instead of an operator with new CRDs, it is a curated Helm chart that deploys a coherent set of pieces.

The three pieces

The stack has three components:

Serving engines: vLLM pods, configured with the flags we have been seeing throughout the series (--enable-prefix-caching, --kv-cache-dtype fp8, and so on). The Helm chart lets you declare them as a list; it deploys the underlying Deployments and Services.
Request router: a proxy in front of the engines that decides which one each request goes to. It supports several policies:
- Round-robin: trivial, for a baseline.
- Session-based: pins each session to one replica to keep its KV cache.
- Prefix-aware: detects shared prefixes between requests and routes them to the replica that already has them cached.
- KV-aware: sees each replica’s gpu_cache_usage_perc and avoids the saturated ones.
- Disaggregated-prefill with native LMCache: separates prefill and decode, with LMCache as the transport for the KV cache between the two.
Observability stack: Prometheus + Grafana with ready-made dashboards. It measures TTFT, TBT (Time-Between-Tokens), throughput, queue depth and GPU memory.

LMCache and tiered KV

One of the most interesting pieces the stack brings is LMCache, which adds a KV cache with multiple tiers: GPU HBM as L1, CPU RAM as L2, local disk as L3, and optionally remote storage as L4. When a KV cache block does not fit in HBM, instead of evicting and recomputing it, LMCache drops it to a lower tier. For workloads with shared prefixes and multi-turn conversations, the saving is huge.

LMCache integrates as a sidecar to the engines and as part of the transport in disaggregated-prefill. Production Stack ships it enabled by default in its Helm chart.

A typical manifest (values.yaml)

servingEngineSpec:
 modelSpec:
 - name: llama3-8b
 repository: vllm/vllm-openai
 tag: v0.6.3
 modelURL: meta-llama/Meta-Llama-3-8B-Instruct
 replicaCount: 3
 requestCPU: 4
 requestMemory: 16Gi
 requestGPU: 1
 vllmConfig:
 enablePrefixCaching: true
 kvCacheDtype: fp8
 maxModelLen: 32768
 enableChunkedPrefill: true

routerSpec:
 routingLogic: prefix-aware  # round-robin | session | prefix-aware | kv-aware
 sessionKey: x-user-id  # when routingLogic=session

cacheserverSpec:
 enabled: true # LMCache for tiered KV
 storageBackends:
 - cpu
 - disk  # offload to local disk

observabilitySpec:
 prometheus:
 enabled: true
 grafana:
 enabled: true
 dashboards:
 - vllm-engine-metrics
 - lmcache-metrics

This is declarative but these are not CRDs: they are Helm chart values. The difference with OME is not semantic (both start from declaration) but operational: with Helm, changes go through helm upgrade; with CRDs, they go through kubectl apply. For teams already living in GitOps with Argo CD or Flux, both approaches integrate cleanly, but the flows are different.

When to choose it

If vLLM is your only runtime and you want the closest thing to “the happy path the project recommends”, this one. It is the productionised version maintained by the same people who write the engine. The downsides: it ties you to vLLM (it is not generic) and it does not solve some advanced cases such as multi-tenancy with strict quotas or gang scheduling, where OME or full-fledged operators are better.

NVIDIA Dynamo

NVIDIA Dynamo is the official successor to Triton Inference Server, announced at GTC 2025 and merged into the brand as Dynamo-Triton in March of that year. Triton had spent years as the most-used inference engine in “serious” NVIDIA infrastructures; Dynamo is what NVIDIA believes the new generation needs.

What it is exactly

Dynamo is a distributed inference framework, not exactly a Kubernetes operator. It has its own runtime (it can run engines), a scheduler (Grove), intelligent routing, multi-tier KV cache management and disaggregation. It supports SGLang, TensorRT-LLM and vLLM as engines, but the engines are executed by Dynamo, not the other way round: the model is “Dynamo manages, the engine executes”.

On Kubernetes, Dynamo is deployed via its own operator plus CRDs, normalised with the K8s integration NVIDIA formalised at the end of 2025 (covered by this InfoQ piece). The CRDs are product-specific: they define a DynamoCluster, a topology of prefill/decode workers and a routing policy.

The four contributions

Dynamo is sold on four pillars, with figures reported by NVIDIA:

Disaggregated serving built in, with its own scheduler.
Smart routing based on cache state: if a worker already has most of a prompt cached, the request goes there.
Multi-tier KV cache: analogous to LMCache, with HBM/RAM/SSD/NVMe.
Autoscaling integrated with Dynamo’s scheduler.

The marketing figure: up to 30× more throughput than legacy Triton on the same hardware. With all the caution a vendor benchmark deserves.

Grove: its own scheduler

A controversial decision in Dynamo is not to rely 100% on the Kubernetes scheduler and instead bring its own scheduler, called Grove, that understands GPU topologies. Grove decides which worker runs on which physical GPU, which interconnects (NVLink/InfiniBand) are relevant, and how to distribute tensor parallel across nodes. That gives it more control than the standard kube-scheduler.

Operationally: if your cluster is “pure Kubernetes” with kube-scheduler and heterogeneous workloads (not only LLMs), Grove adds one more component to operate. If your cluster is dedicated to LLM inference and there is already a team dedicated to running it, Grove gives you more levers.

When to choose it

Dynamo makes sense if:

Your infrastructure is NVIDIA-heavy (Hopper, Blackwell, GB200) and you want to exploit the latest in TensorRT-LLM with the familiar Triton integration, modernised.
You were already a Triton user for legacy inference (vision, recommendation) and want to keep the ecosystem.
You have a dedicated SRE team for inference and Grove’s extra operational complexity is not a problem.

It is the vendor-specific option of the quartet. In exchange it gives you NVIDIA’s support and first-rate integration with their hardware. If your organisation is already fighting NVIDIA for GPUs, they may even call you offering help with Dynamo.

llm-d

llm-d is the youngest and the most “political” of the four. In March 2026, at KubeCon Europe Amsterdam, Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA announced the joint donation of the project to the CNCF as a Sandbox project, with backing from AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley and the University of Chicago. An explicit coalition for vendor neutrality.

Philosophy

llm-d positions itself as the vendor-neutral “Kubernetes blueprint” for distributed inference. It is not a runtime; it is a system that sits on top of vLLM (the default engine) and orchestrates the control plane.

The primitives the project puts on the table:

Intelligent routing with prefix-cache awareness and load-aware balancing.
Tiered KV cache with offload to CPU and disk for multi-turn.
Prefill/decode disaggregation over fast interconnects.
Wide expert parallelism for serving very large Mixture-of-Experts (MoE) models, a critical pattern popularised by DeepSeek-V3 and Mixtral, where the experts live on different GPUs and tokens have to be routed to the right expert.

Numbers

The v0.5 release validates ~3.1k tok/s per decode GPU on B200, and up to 50k output tok/s on a 16×16 B200 prefill/decode topology. The most interesting benchmark: an order-of-magnitude reduction in TTFT against a round-robin baseline. In other words, intelligent routing is worth what they say it is.

CNCF and the future

Donating to the CNCF as a Sandbox project means neutral governance: no vendor is in charge. For an organisation wary of being tied to a single supplier, llm-d is probably the safest medium-term bet. The price: like any Sandbox project, it is not yet “boring” in the way vLLM is. There is API churn, features that move around, documentation that lags the code.

When to choose it

llm-d makes sense if:

You want multi-vendor portability without ties to NVIDIA, Red Hat or Google.
Your workload includes large MoE models (DeepSeek-V3, Mixtral 8x22B, Llama 4 Behemoth if the size is confirmed), where wide expert parallelism is decisive.
Your organisation is already comfortable with CNCF Sandbox (projects under active evolution, not yet a stable 1.0).
You want to bet on the project that will probably be the de facto standard in 2-3 years.

The common ancestor is still there: KServe

It is worth reconnecting before the comparison: KServe is still alive and heavily used in organisations that serve both LLMs and traditional models (scikit-learn, XGBoost, PyTorch CV). Its InferenceService is generic enough to serve any model, including vLLM or SGLang as a ServingRuntime. What it does not do well is the LLM-specific part: disaggregation, multi-node tensor parallel, routing with KV cache awareness. If your organisation already has KServe in production for other models, adding an LLM-specific operator alongside it (OME, vLLM Stack or llm-d) is reasonable. Fighting it all out from pure KServe is not.

Decision map

Dimension	OME	vLLM Prod Stack	NVIDIA Dynamo	llm-d
Philosophy	Classic K8s-idiomatic operator	Curated Helm chart	Framework with its own scheduler	Vendor-neutral CNCF blueprint
Own CRDs	Yes (BaseModel, ServingRuntime, InferenceService…)	No (Helm values)	Yes (DynamoCluster)	Yes (KServe-derived + extensions)
Primary runtime	SGLang (first class), also vLLM/TRT-LLM/Triton	vLLM only	TensorRT-LLM (first class), also SGLang/vLLM	vLLM (first class)
PD-disaggregation	Yes, declarative	Yes, with LMCache	Yes, own scheduler	Yes, native
Multi-node TP	Yes, via LWS	Limited	Yes, via Grove	Yes, via LWS and MoE EP
Multi-model in a cluster	Yes, mature multi-tenant	Yes (list of models in values)	Yes	Yes
Multi-LoRA	Yes, first class (FineTunedWeight CRD)	Limited	Yes	On the roadmap
Tiered KV cache	Via LMCache (external integration)	Native LMCache	Own multi-tier	Yes, native
Intelligent routing	Cache-aware via the SGLang router	Prefix-aware / KV-aware / session-based	Own smart routing	Prefix-cache + load-aware
GPU scheduler	kube-scheduler + Kueue	kube-scheduler	Grove (its own)	kube-scheduler + Kueue
Hardware	NVIDIA, AMD ROCm, Intel	NVIDIA, AMD ROCm	NVIDIA only (with emphasis)	NVIDIA, AMD, Intel — neutral
Maturity (mid-2026)	Young, evolving	Stable	Stable, vendor-driven	CNCF Sandbox, evolving fast
Governance	LMSYS (academic-industrial)	vLLM project (academic)	NVIDIA (vendor)	CNCF (neutral)
Learning curve	Medium (4 new CRDs)	Low (familiar Helm values)	Medium-high (Grove + own CRDs)	Medium (similar to extended KServe)

When to choose each one

Choose OME if:

SGLang is your main engine.
You need multi-LoRA serving in production.
The hierarchical abstraction (BaseModel → ServingRuntime → InferenceService) fits you and you come from, or live alongside, KServe.
You have an appetite for a young and very active project.

Choose vLLM Production Stack if:

vLLM is your only engine and you want to align with what the project recommends.
Your team already lives in Helm and does not want to learn new CRDs.
LMCache + advanced routing inside a single Helm chart is exactly what you need.
Your scale is medium (tens of replicas), not extreme.

Choose NVIDIA Dynamo if:

Your infrastructure is NVIDIA-heavy and you want the most optimised path for Hopper/Blackwell.
You already ran Triton for legacy inference and the transition is natural.
You accept vendor lock-in in exchange for direct NVIDIA support.
Your organisation has a dedicated SRE team for inference.

Choose llm-d if:

You want to bet on the future CNCF standard, neutral between vendors.
Your workload includes large MoE models with wide expert parallelism.
You operate multi-cloud or multi-hardware and portability is valuable.
You accept the immaturity of a Sandbox project in exchange for the bet on the future.

Choose pure KServe if:

You already serve non-LLM models and want to unify; LLMs are a minority of your workload.
You need the most conservative and mature use case.
You accept that advanced LLM features (disaggregation, MoE EP, smart routing) are yours to add with external pieces.

Concrete scenarios

Scenario A — Small startup, 1-2 models, 1-3 GPU nodes. You probably do not need an operator. Deployment + Service + HPA with KEDA metrics, as in the vLLM on Kubernetes article. When you grow to 5+ models, reassess.

Scenario B — Mid-sized company, 5-15 models, internal multi-tenant. vLLM Production Stack or OME are the reasonable options. Production Stack if vLLM is all you are going to use; OME if you want runtime flexibility and idiomatic CRDs.

Scenario C — Internal corporate platform or an external service for end customers. llm-d or Dynamo. llm-d if you value vendor neutrality; Dynamo if you live on NVIDIA infrastructure and want the path they recommend.

Scenario D — Mixed cluster of LLMs plus traditional models. KServe as the base, with an LLM operator alongside (OME is the most natural given its conceptual kinship).

Common traps

“I’ll start with pure KServe because it’s mature”. For mid-sized LLMs and up, pure KServe leaves a lot of optimisation on the table. The sensible route is KServe as the base if you live alongside other models, but with an LLM-specific operator next to it.

“I’ll wire everything by hand to understand it”. Reasonable in a PoC, suicidal in production. There are 8 derived resources per model. Multiply by 10 models. You are writing 80 YAMLs and maintaining them. Use an operator.

“I’ll pick the one I like best and pivot if I’m wrong”. Pivoting between operators is not free: although the InferenceService abstraction is converging, the details (how LoRA is modelled, how routing is configured, how metrics are exposed) vary. Migrating from OME to Dynamo is a project of weeks, not days.

“I’ll go with Dynamo because it’s NVIDIA’s and therefore better”. Only if your organisation is already aligned with its operational philosophy (its own scheduler, acceptable vendor lock-in). For many cases, vLLM Production Stack or llm-d give 95% of the value with less friction.

“Helm chart vs operator is a technical decision”. It is a cultural and operational decision. If your team ships via Argo CD with Helm values in Git, Production Stack fits without friction. If your team lives in direct kubectl apply -f and the idea of operators feels natural, OME or llm-d.

What we have not covered

Mooncake: the KV cache system shared between instances that Kimi/Moonshot runs in production across hundreds of millions of queries. It is a primitive (not a complete operator), but it integrates as a cache tier with several of the above.
Ray Serve LLM: Anyscale’s offering, on Kubernetes through KubeRay. More tied to the Ray ecosystem than to native K8s CRDs. Useful if Ray is already part of your infrastructure.
Fireworks AI, Modular MAX: commercial platforms with similar primitives, but hosted. They are not K8s operators; they are competitors at another layer.
Gateway API Inference Extension: the sigwg proposal to extend Gateway API with LLM primitives (model-aware routing, sticky sessions, fairness). In 2026 it is in alpha; the operators above are already starting to support it. When it matures, routing will stop being each operator’s problem and become part of the Kubernetes standard.
A generic inference observability stack: Prometheus + Grafana is standardising around the vllm:* metrics we covered in the vLLM article. There is an OpenTelemetry effort for LLMs (gen-ai semantic conventions) that is probably the next link in the chain.

Closing the series

This series of four articles has walked through LLM inference in production from the bottom up:

KV cache: the working memory that holds up LLM inference — why each token consumes VRAM, and how much.
vLLM on Kubernetes: the LLM inference piece that actually scales — how a model is served in production with a serious Deployment.
PagedAttention from the inside and the state of the art of the KV cache in 2026 — what happens inside the engine at block level, and what has arrived since.
This one — how many models are orchestrated in a cluster.

If you have got this far, you have the vocabulary and the map to sit in a meeting where five people throw acronyms around and place each one correctly. And, more importantly, to start making reasoned decisions about where to begin.

References

Operators and projects covered:

OME — Open Model Engine (GitHub) — LMSYS operator for LLM serving with SGLang/vLLM/TRT-LLM/Triton.
Introducing OME (LMSYS Blog, jul 2025) — announcement and architecture.
vLLM Production Stack (GitHub) — official vLLM Helm chart for K8s.
vLLM Production Stack docs — installation and configuration.
LMCache (GitHub) — KV cache with tiers.
NVIDIA Dynamo — successor to Triton.
NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges (InfoQ, dic 2025) — K8s integration.
llm-d (GitHub) — CNCF Sandbox project.
IBM, Red Hat, and Google donated llm-d to CNCF (The New Stack) — KubeCon EU 2026 announcement.
Red Hat bets big on Kubernetes inference with llm-d (SiliconANGLE, mar 2026) — coverage of the announcement.

Ancestors and primitives:

KServe (site) and KServe joins CNCF (The New Stack).
Kueue — gang scheduling.
LeaderWorkerSet — coordinated workloads such as multi-pod tensor parallel.
KEDA — autoscaling on external metrics.
Gateway API — the successor to Ingress.

Analysis and perspectives:

Building Efficient LLM Inference with the Cloud Native Quartet: KServe, vLLM, llm-d, and WG Serving (Jimmy Song) — an integrating view.
Complete Guide to llm-d CNCF Sandbox (DEV Community) — an operational walkthrough.
Previous articles on this blog: KV cache, vLLM on Kubernetes, PagedAttention deep dive.

Kserve on lo0 — Tech Blog

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

TL;DR

The analogy: the restaurant that switches off the kitchen

How Knative scales

KPA versus HPA

Why an LLM cold start is not starting a pod

The techniques that shorten the wait

Knative and KServe: the two modes

The economics

Operational traps and honest scepticism

For an inference factory

See also

Sources

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

TL;DR

The analogy: the hypermarket and the specialist shop

What Kubeflow is in 2026

Component by component

Kubeflow Pipelines: the piece you may actually want

Kubeflow Trainer: it leans on your scheduler, it does not replace it

Katib: only half relevant

KServe: the piece that is no longer in Kubeflow

Model Registry, Notebooks and the rest

Multi-tenancy: powerful and tied to Istio

The cost of installing the whole thing

Kubeflow against the alternatives

The real fit with LLM workloads

Operational traps and honest scepticism

For an inference factory

See also

Sources

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

TL;DR

The analogy: the station master and the track gauge

A Deployment serves tokens, but offers no contract

KServe: from KFServing to a CNCF incubating project

Anatomy of the control plane

InferenceService: predictor, transformer, explainer

ServingRuntime and ClusterServingRuntime: the catalogue of trains

The storage initializer and storageUri

The deployment modes

The LLM branch: LLMInferenceService

The Open Inference Protocol: the contract itself

The routes and the payload

Who actually implements it

The generate extension and why it did not win

The tension, unvarnished

Four manifests you can actually apply

Autoscaling and the problem of zero

Operational traps

Decision map: when KServe, and when it is overkill

For an inference factory

See also

Sources

Model chain of trust (2/4): where the bytes live — registry, OCI artefacts and distribution to the GPU node

TL;DR

The analogy

Three planes that are not the same

The anti-pattern we start from

The model as an OCI artefact

What OCI 1.1 made possible

ORAS: the generic client

ModelPack: the missing convention

How a 140 GB model is sliced up

Harbor as the on-premise repository

Model registries proper

MLflow

Kubeflow Hub (formerly Model Registry)

Comparison table by function

The hot path: from the registry to HBM

Image volumes: the change of the year

KServe: storageUri and modelcars

Loading fast: where each technique attacks

LoRA adapters: when the artefact weighs megabytes

Air-gap and sovereignty: promotion without losing traceability

Decision map

Operational traps

For an inference factory

See also

A `Deployment` serves tokens, but offers no contract

`InferenceService`: predictor, transformer, explainer

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

The storage initializer and `storageUri`

The LLM branch: `LLMInferenceService`

The `generate` extension and why it did not win

KServe: `storageUri` and modelcars

The analogy: from `init.d` to systemd to operators