Llm-Inference on lo0 — Tech Blog

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

Sun, 26 Jul 2026 20:00:00 +0200

This blog has devoted a good part of its 145 articles to what happens inside the inference process and another good part to what happens around it. What is missing is a thread that cuts across all of that and answers a question that takes thirty seconds to ask in an audit and does not get answered in three weeks in production: when the endpoint returns a token, where did that model come from, who signed it, and who proves that the binary running on the GPU is the one that was approved?

This series walks that chain backwards from the only visible point, the endpoint: the control plane and the API contract (this article), the registry and the distribution of the bytes (2/4), the signing and provenance that let you trust them (3/4), and the identity and isolation of whoever runs them (4/4). All of it with projects under the governance of CNCF, LF AI & Data or the Linux Foundation: a chain of trust that depends on a proprietary product is no better than a support contract.

TL;DR

A vLLM Deployment serves tokens perfectly well. What it does not give you is a contract: no model versioning, no rollout by percentage, no scale-to-zero, and no common surface for the thirty non-LLM artifacts of an inference factory.
KServe provides the vocabulary: InferenceService, ServingRuntime/ClusterServingRuntime, storage initializer and storageUri. It has been CNCF incubating since 29 September 2025, after seven years as KFServing and a spell in LF AI & Data. The cadence has accelerated: v0.18.0 on 29 April 2026 and v0.19.0 on 14 June 2026.
The LLM branch is recent and different: the LLMInferenceService CRD (serving.kserve.io/v1alpha1) arrived in v0.16 in November 2025, on top of llm-d, with spec.prefill, spec.worker, spec.parallelism and spec.router.
The Open Inference Protocol (V2) is the contract: /v2/health/*, /v2/models/<m>, /v2/models/<m>/infer, in REST and gRPC, implemented by Triton, OpenVINO Model Server, Seldon MLServer and AMD Inference Server. It is tensor-centric, and the LLM world standardised de facto on the OpenAI API: KServe serves both, with the OpenAI ones under the openai/ prefix.
The honest position: the portability of an LLM today comes from the OpenAI API, not from V2. Accepting the dual track is correct; pretending there is a single standard is not.
KServe is overkill if you serve one model on one node. It starts to pay off from the fifth heterogeneous artifact onwards, and it competes on a different axis from the LLM operators: contract and lifecycle versus engine and topology.

The analogy: the station master and the track gauge

An inference factory is like a railway station with several lines.

The station master does not drive any train. His job is the service timetable: which formation leaves from which platform, how many units are coupled each hour, when an engine is pulled out for maintenance and, when new rolling stock arrives, how to divert ten per cent of the passengers to the test train before committing the whole service. That is the control plane. KServe does not generate a single token, but it decides what runs, with what configuration, with how many replicas and with what traffic split. The trains are the engines (vLLM, SGLang, TensorRT-LLM, Triton, TEI), and a competent station master does not marry a manufacturer.

And then there is the track gauge: as long as everybody builds to the same one, any locomotive rolls on any line and changing supplier is a commercial decision, not a civil engineering project. That gauge is the Open Inference Protocol, the contract that makes a client written against Triton work against OpenVINO Model Server without touching a line.

The analogy has a twist that needs no explanation in Spain: the agreed gauge is not the gauge the new rolling stock runs on. The conventional network was built to Iberian gauge and the high-speed one to standard gauge, and the practical solution was not to convert the network but to install gauge changers at the boundary points. The same thing has happened in inference: V2 is the agreed gauge and the OpenAI API is the gauge everything generative runs on. What sustains interoperability is the changer, not a single standard. We will come back to it.

A `Deployment` serves tokens, but offers no contract

In vLLM on Kubernetes we built a service with Deployment, Service, HPA and an initContainer that pulls the weights down. It works, and for one model on one node it is the right answer. What it does not give you, and cannot give you without reinventing it every time:

A stable API contract. The endpoint is “whatever the image you put there exposes”: moving from vLLM to TensorRT-LLM changes routes, error format and metric names, and the clients find out in production.

Versioning with semantics. The model is a path inside a volume; knowing which version a pod is serving requires kubectl exec. There is no declarative field that says “this service serves artifact X” and can be audited from outside.

Progressive model rollout. A RollingUpdate replaces pods; it does not split traffic between two versions by percentage. For that you need an L7 gateway in front, like the one in canary, blue-green and shadow, or a control plane that emits it.

Scale-to-zero. The HPA does not go below one replica: for a model queried twelve times a day, that is an 80 GB GPU tied up twenty-four hours.

And, above all, a common surface for what is not an LLM. A real factory does not serve three LLMs: it serves two or three LLMs, four embedding models, a couple of rerankers, ticket classifiers, a PII detector, an OCR, industrial vision and three scikit-learn artifacts that nobody remembers training. Fifteen or thirty artifacts, each with its own runtime and its own way of saying “I am ready”. The cost is not in serving the first one: it is in making sure the thirtieth does not cost thirty times what the first one did.

KServe: from KFServing to a CNCF incubating project

KServe was born in 2019 as KFServing, inside Kubeflow, in a collaboration between Google, IBM, Bloomberg, NVIDIA and Seldon. In February 2022 it was donated to the LF AI & Data Foundation; in September 2022 it split from Kubeflow and was renamed. The relevant jump: on 29 September 2025 the CNCF accepted it directly at incubating level, without going through sandbox, with a public announcement on 11 November. The figures it declared then, more than 300 contributors, 19 maintainers, more than 30 adopting organisations (Bloomberg, Red Hat, Cloudera, Nutanix, SAP, NVIDIA), come from an interested party, but the list is verifiable and the 625 contributing organisations on the CNCF dashboard support the thesis of governance not captured by a vendor.

On cadence you have to be precise, because the cliché of “KServe is slow” no longer describes reality:

Version	Date	What it brought
v0.15.0	31 March 2025	`LocalModelCache`, multi-node inference, OpenAI-compatible embeddings API, Gateway API in raw mode, KEDA integration
v0.16.0	November 2025	`LLMInferenceService`; rename of `RawDeployment` to `Standard` and of `Serverless` to `KNative`; progressive rollout in raw mode
v0.18.0	29 April 2026	WVA + KEDA/HPA autoscaling for LLMISvc, `/v1/responses` route (OpenAI Responses API), `restricted` Pod Security Standards by default on LLMISvc, namespace-scoped `ModelCache`
v0.19.0	14 June 2026	Routing by model name, static LoRA adapters, balancing across heterogeneous GPUs, dual REST/gRPC routing in Standard mode, automatic migration to `InferencePool` v1

Two major releases in under two months, with the bulk of the changelog in the LLM branch. The project is not slow: it has pivoted from classic predictive serving to generative serving. The consequences run in both directions.

The negative one is called ModelMesh. It was the high-density mode (hundreds of small models sharing processes, with dynamic load and unload) and it was, on paper, the natural answer to the problem of thirty heterogeneous artifacts. Its last release was v0.12.0 on 9 July 2023, and the kserve/modelmesh-serving repository was archived read-only on 14 April 2026. Red Hat published specific guidance for migrating to Standard mode. If somebody proposes ModelMesh in a design today, it is a sign that the documentation they read is three years old.

Anatomy of the control plane

`InferenceService`: predictor, transformer, explainer

The core CRD is InferenceService (serving.kserve.io/v1beta1), and its contribution, the one that became the common vocabulary of the field, is to break an inference service down into three components each with its own contract:

predictor: mandatory. It declares modelFormat, optionally runtime, the storageUri the weights come from and the protocolVersion it exposes.
transformer: optional. Pre- and post-processing. That it is a separate component, rather than code inside the server, is what allows it to be scaled separately (it is usually CPU-bound while the predictor is GPU-bound) and versioned separately.
explainer: optional, and the one that has aged worst: the V2 protocol does not support the explain endpoint, which only exists in V1. Anyone who needs explainability with a standard contract has a real gap here.

All three share ComponentExtensionSpec: minReplicas, maxReplicas, scaleTarget, scaleMetric, containerConcurrency, timeout and canaryTrafficPercent.

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

A ServingRuntime is a pod template that knows how to serve one or several formats. It declares supportedModelFormats (with name, version, autoSelect and priority), protocolVersions (v1, v2) and containers, with args parameterised by templates in the {{.Name}} style that are substituted with metadata from the InferenceService.

Selection is what makes this scale. If the InferenceService names a runtime, the controller looks for it first in the namespace and then at cluster level; if it does not name one, it picks among those with autoSelect: true for that modelFormat, breaking ties by priority. That is what lets a platform publish a curated catalogue (“here, huggingface is served with this vLLM and these flags”) without the product teams having to know what --gpu-memory-utilization is. In multi-tenant setups, the ClusterServingRuntimes belong to the platform and the InferenceServices to the teams: the cleanest separation-of-duties line KServe offers.

The storage initializer and `storageUri`

The storageUri field triggers the storage initializer: an init container that downloads the artifact to /mnt/models before the server starts. Schemes supported as of July 2026: s3://, gs://, https://, pvc://, hf://, oci:// and git repositories; credentials are attached via a ServiceAccount with annotated secrets. That oci:// connects with the second article of the series: the weights can live in the same registry as the images, with the same access control and the same signing capability.

The deployment modes

This is where nearly everything operational is decided. It is selected with the serving.kserve.io/deploymentMode annotation.

	KNative (formerly Serverless)	Standard (formerly RawDeployment)	ModelMesh
Resources emitted	Knative `Service` + `Revision`, `VirtualService` with Istio	`Deployment`, `Service`, `HPA`, Gateway API `HTTPRoute`	Its own high-density runtime
Dependencies	Knative Serving + Istio (or an alternative gateway)	Kubernetes 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1 and a controller	Its own components
Autoscaling	KPA (concurrency / RPS)	HPA or KEDA	—
Scale-to-zero	Yes, with an activator that holds the request	No for HTTP requests	—
`canaryTrafficPercent`	Yes, by revision weights in Istio	Silently ignored (issue #5335, open since 2 April 2026)	—
Sidecars in the pod	`queue-proxy` (+ `istio-proxy`)	None	—
Status as of July 2026	Current	Current and the default mode of the LLM branch	Archived on 14 April 2026

That table has an uncomfortable reading: the two most quoted KServe capabilities, scale-to-zero and canary by percentage, live in the mode that drags Knative and Istio along, while the mode the project has pivoted towards is the other one.

The LLM branch: `LLMInferenceService`

Since v0.16 KServe has a separate CRD for LLMs: LLMInferenceService, in serving.kserve.io/v1alpha1. It is not an extension of the classic InferenceService and does not share its API version, and confusing the two is the first mistake made by anyone arriving from old documentation.

It is built on llm-d, the project that Red Hat, Google, IBM, CoreWeave and NVIDIA donated to the CNCF. The division of roles the project itself declares is clean: KServe is the control plane (lifecycle, scaling and operational governance) and llm-d contributes distributed scheduling, with GPU utilisation, queue depth, cache residency and SLA. In the analogy: llm-d decides which line each train takes; KServe decides which trains exist and when they are withdrawn.

The fields that matter: spec.model (URI, invocation name, scheduling criticality and LoRA adapters, with static adapter reconciliation from v0.19, which connects with multi-LoRA serving); spec.template (single node, or the decode pool); spec.worker (multi-node, which triggers LeaderWorkerSet); spec.prefill (the disaggregation we analysed in disaggregated serving turned into a declarative field); spec.parallelism (tensor, data and expert parallelism); and spec.router (gateway, route, scheduler), where KServe leans on the Gateway API Inference Extension, v1.5.0 of 19 April 2026, already GA, with automatic migration to InferencePool v1 from v0.19.

The endpoint it exposes is OpenAI-compatible: /v1/chat/completions with streaming, and since v0.18 also /v1/responses. That detail underpins the next section: the LLM branch of KServe does not speak V2, it speaks OpenAI.

For multi-node there is the kserve-huggingfaceserver-multinode runtime, based on vLLM over Ray, with workerSpec.tensorParallelSize and workerSpec.pipelineParallelSize. Its restrictions are hard: it requires a ReadWriteMany PVC, it only works in Standard mode, it does not support autoscaling and it requires exactly one head pod. A reference 4×H100 SXM node with NVLink needs none of this, because a 70B fits with tensor parallel 4 on a single node; you need it from there upwards.

The Open Inference Protocol: the contract itself

The routes and the payload

The Open Inference Protocol (historically the “V2 Inference Protocol”) lives in kserve/open-inference-protocol, with REST and gRPC specifications and versioning under SemVer 2.0. Its HTTP surface is deliberately small:

Resource	Verb	Route
Server metadata	GET	`/v2`
Liveness	GET	`/v2/health/live`
Server readiness	GET	`/v2/health/ready`
Model metadata	GET	`/v2/models/<model>[/versions/<v>]`
Model readiness	GET	`/v2/models/<model>[/versions/<v>]/ready`
Inference	POST	`/v2/models/<model>[/versions/<v>]/infer`

The payload is tensorial: id, a list of inputs (each with name, shape, datatype and data) and optionally the desired outputs. The types are BOOL, signed and unsigned integers from 8 to 64 bits, 16-, 32- and 64-bit floating point, and BYTES. In gRPC the same contract is GRPCInferenceService, with six RPCs: ServerLive, ServerReady, ModelReady, ServerMetadata, ModelMetadata and ModelInfer. For high-volume embeddings, the serialisation difference is not cosmetic.

Who actually implements it

The specification repository declares KServe, NVIDIA Triton, Seldon MLServer and Core v2, OpenVINO Model Server, AMD Inference Server and TorchServe as adopters. Not all entries are worth the same as of July 2026:

Triton implements it in a documented way (“Triton exposes HTTP/REST and gRPC endpoints based on the standard inference protocols proposed by the KServe project”), plus its own extensions. It is the de facto reference implementation.
OpenVINO Model Server exposes the KServe API over gRPC and REST, the TensorFlow Serving one and OpenAI-compatible endpoints: the clearest example of a server that speaks both track gauges.
TorchServe forces you to be honest. Its official documentation carries the warning: “This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches.” An adopter on zero maintenance is not an adopter: it is a row nobody has updated.

On the health of the standard, the most eloquent data point is not in any press release: the specification repository does not have a single published release, has accumulated something like thirty commits and seventy-five stars, and its governance consists of monthly meetings and a Slack channel. There is also a “vendor neutral” fork at open-inference/open-inference-protocol with more commits than the original and no traction whatsoever. V2 is a stable contract, widely implemented and barely evolving. For a serialisation format that can be a virtue; for a standard aspiring to cover the generative world, it is a death sentence.

The `generate` extension and why it did not win

The standard did try to cover generation: the specification contains a generate_rest.yaml with two routes, /v2/models/<m>/versions/<v>/generate and .../generate_stream, and a minimalist schema: a request with text_input and parameters (temperature, top_p, max_tokens defaulting to 20, stop, details); a response with text_output, model_name, model_version and, if requested, finish_reason and logprobs; streaming over server-sent events. It is clean, it is enough for a simple completion and nobody uses it: the KServe generative inference documentation, as of July 2026, lists four endpoints, and none of them is generate:

Task	Endpoint in KServe
Text generation	`openai/v1/completions`
Chat	`openai/v1/chat/completions`
Embeddings	`openai/v1/embeddings`
Reranking	`openai/v1/rerank`

The openai/ prefix exists, according to the documentation itself, “to avoid confusion with the V1 API protocol”, and it is changed with KSERVE_OPENAI_ROUTE_PREFIX. It is plumbing that says a lot: the in-house standard was relegated to a defensive prefix while the outside one takes all the traffic.

The tension, unvarnished

V2 solved the problem of 2021, and it solved it well. A tensorial, typed contract with metadata and health checks, in REST and gRPC, is what you need to serve a classifier, a vision model, a ranker or an embedding, and it is still the right answer for that half of the catalogue: swapping an embedding server from MLServer to OpenVINO Model Server without touching clients is the track gauge doing its job.

The LLM world standardised somewhere else, not by committee, but because the entire client ecosystem (SDKs, agent frameworks, gateways, evaluation) was written against the OpenAI API, and any engine that wanted to be used had to speak it. What that means for an architecture:

There is not one contract: there are two, and both are legitimate. Designing as if there were one does violence to half the catalogue.
LLM portability is superficial. It covers chat and completion well; as soon as you use detailed logprobs, guided decoding, priority headers, prefix caching parameters or tool calling with specific schemas, you leave the common subset and get tied back to the engine. It is the warning from structured output: what is portable is the envelope, not always the contents.
The gauge changer is the gateway. What makes the set switchable is the translation and routing layer of choosing an OSS LLM inference gateway and the router as an L7 gateway, not a single standard.
Crossing over loses the versioning. V2 has /versions/<v> in the route; the OpenAI API only has the model field. The version has to be encoded in the name (asistente-70b-2026-07) or managed in the control plane. That is the difference between being able to audit which version answered and not being able to.

Four manifests you can actually apply

Platform runtime for the blog’s reference node, 4×H100 SXM 80 GB with NVLink, with tensor parallel 4:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
 name: vllm-h100-tp4
spec:
 annotations:
 prometheus.kserve.io/port: "8080"
 prometheus.kserve.io/path: "/metrics"
 supportedModelFormats:
 - name: huggingface
 version: "1"
 autoSelect: true
 priority: 10
 protocolVersions:
 - v2
 containers:
 - name: kserve-container
 # ALWAYS pin by digest in production; the tag is only for readability
 image: vllm/vllm-openai:v0.11.0
 command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
 args:
 - --port=8080
 - --model=/mnt/models
 - --served-model-name={{.Name}}
 - --tensor-parallel-size=4
 - --max-model-len=32768
 - --gpu-memory-utilization=0.92
 - --enable-prefix-caching
 - --kv-cache-dtype=fp8
 resources:
 requests: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 limits: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 # the probe that stops the pod dying while the model loads
 startupProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 15
 failureThreshold: 40 # up to 10 minutes of loading
 readinessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 10
 livenessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 30

The InferenceService that consumes it, with KEDA on vLLM metrics:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 serving.kserve.io/deploymentMode: "Standard"
 serving.kserve.io/autoscalerClass: "keda"
spec:
 predictor:
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 # the OCI registry from article 2/4 of this series
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-07"
 autoScaling:
 metrics:
 - type: External
 external:
 metric:
 backend: "prometheus"
 serverAddress: "http://prometheus.monitoring.svc.cluster.local:9090"
 query: 'sum(vllm:num_requests_waiting{model_name="asistente-70b"})'
 target:
 type: Value
 value: "8"

The canary, with its small print:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 # WATCH OUT: canaryTrafficPercent only takes effect in KNative mode.
 # In Standard mode it is silently ignored (issue kserve/kserve#5335, open).
 serving.kserve.io/deploymentMode: "KNative"
spec:
 predictor:
 canaryTrafficPercent: 10
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-08"

KServe keeps three references in the status (LatestReadyRevision, LatestRolledoutRevision and PreviousRolledoutRevision): promoting means setting the percentage to 100, reverting means pinning traffic on the previous one, and a revision that does not become ready automatically receives zero traffic.

And the two calls. Against the V2 contract, for a classifier served with MLServer or Triton:

# metadata: which model it is, what inputs it expects, what types
curl -s http://clasificador.inferencia.example/v2/models/clasificador-tickets | jq .

# readiness of the specific model, not of the server
curl -s -o /dev/null -w '%{http_code}\n' \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/ready

curl -s -X POST \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/infer \
 -H 'Content-Type: application/json' \
 -d '{ "id": "req-4711",
 "inputs": [ { "name": "input-0", "shape": [1, 4], "datatype": "FP32",
 "data": [5.1, 3.5, 1.4, 0.2] } ],
 "outputs": [ { "name": "output-0" } ] }'

Against the same control plane, but on the OpenAI-compatible surface of the LLM:

curl -s -N -X POST \
 http://asistente-70b.inferencia.example/openai/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{ "model": "asistente-70b",
 "messages": [ {"role": "user",
 "content": "Summarise last night'\''s incident report."} ],
 "max_tokens": 256,
 "stream": true }'

Two contracts and two payload shapes, with the same InferenceService as the unit of governance. That is the value proposition of a control plane: the contract is chosen by the model and the management is single.

Autoscaling and the problem of zero

Three autoscalers coexist, and choosing the wrong one is the most common cause of disappointment. KPA, in KNative mode, scales by concurrency or RPS and is the only one that goes down to zero in a usable way, because Knative provides an activator that holds the request while the pod starts; its metric does not distinguish a 30-token request from an 8,000-token one. HPA, in Standard, scales by CPU or memory: for LLMs it is useless, because the CPU of a saturated vLLM pod and that of an idle one look too much alike. KEDA, in Standard, is the right option, the one we develop in autoscaling with KEDA: it is enabled with serving.kserve.io/autoscalerClass: "keda" and autoScaling.metrics accepts external metrics against Prometheus (vllm:num_requests_running or vllm:num_requests_waiting) or, with the OpenTelemetry add-on, pushed metrics. Since v0.18 there is also Workload Variant Autoscaling, which uses HPA or KEDA as a backend.

That leaves the serious problem, which no autoscaler solves: zero. A 70B model in fp8 is on the order of 70 GB to move into HBM, and that path, from disk to HBM, is measured in minutes: the first request after idling takes that long, or it fails.

Worse: the installation documentation is explicit as of July 2026: “Scale from Zero is currently not supported in Standard mode for HTTP requests”. The mode the project has pivoted towards, and the only one that supports KEDA, does not know how to come up from zero on an HTTP request: KEDA can go down to zero (default cooldown of 300 seconds), but the way back requires an activator that Standard does not have. Anyone who wants real zero with HTTP activation still needs Knative.

The levers, in order of effectiveness: do not scale the hot model to zero (minReplicas: 1 for anything with an SLA; zero is for the long tail); attack the load time, the only solution that is not a patch, with the techniques from speeding up cold start; use LocalModelCache, available since v0.15 and extended to LLMInferenceService in v0.19, to preload artifacts into a node PVC; and set correct probes, which is what comes next.

Operational traps

The probe that kills the pod while the model loads. The most frequent and the most expensive to diagnose, because the symptom is a CrashLoopBackOff with no error in the server logs. Until early 2026 the KServe vLLM templates used initialDelaySeconds of 120 to 300 seconds on liveness, with the result described in issue #5062 (12 February 2026, now resolved): the fast models waited too long and the large ones died before finishing loading. The fix applied: a startupProbe of up to ten minutes and probes with no initial delay. A legacy runtime has to be reviewed by hand. In KNative mode there is a worse variant (issue #3795, closed as not planned): what dies is not the model container but the queue-proxy, which does not respond OK within the deadline and takes the whole pod down with it.

canaryTrafficPercent silently ignored. It deserves its own trap because there is no error and no condition in the status to give it away: in Standard mode the field does nothing. The team believes it is serving 10 % with the new version and it is serving 100 %. While issue #5335 remains open, in Standard the canary is done with HTTPRoute weights by hand, or in the gateway.

Canary and scarce GPUs do not get along. A 10 % canary of a 70B with tensor parallel 4 costs a whole 4-GPU node, not 10 % of the resources, because the minimum granularity is a complete replica. During the window there are two versions occupying twice the hardware, or capacity has to be taken from the stable one. That is the difference between a microservice canary and a model canary, and the reason a shadow over duplicated traffic often pays off better.

storageUri and credentials. Storage initializer failures are always the same ones: a ServiceAccount without the annotated secret, a KMS-encrypted bucket with a role that has no decryption permission, an S3-compatible but not exactly S3 endpoint, or intermittent 403s, with a specific fix in v0.18. It is diagnosed in the init container logs, not in the server’s. And one standing recommendation: no downloading over https:// from the internet in production; that is precisely the problem of article 2/4.

The model size against the emptyDir. By default the artifact lands in an ephemeral volume on the node’s disk. Seventy gigabytes per replica fill a system disk with surprising ease, and then the kubelet evicts pods for disk pressure, including ones that have nothing to do with it. You have to set sizeLimit or use a dedicated pvc://. For multi-node the restriction is harder, with a mandatory ReadWriteMany PVC, which rules out a good part of local block storage and pushes you towards shared file systems, with the consequences covered in storage for AI.

The dependency on Knative and Istio. The real cost of KNative mode: two more control planes to version and correlate with Kubernetes, and two sidecars, queue-proxy and istio-proxy, in every pod, with their CPU and memory consumption and their network hop per request. On a platform that already has a mesh it is a marginal cost; on a cluster dedicated to inference it has to be justified by the capabilities you are actually going to use.

Observability: the name of the metric. The ServingRuntime declares prometheus.kserve.io/port and /path annotations, but the metrics that come out are the engine’s, not KServe’s: vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:gpu_cache_usage_perc. The control plane does not normalise names across runtimes, so a dashboard written against vLLM is no use for Triton, and correlating with DCGM is still your own work. v0.19 raises the HPA/KEDA scaling status into the service conditions and emits events on readiness transitions: exactly what was needed to alert on the control plane and not only on the engine.

Decision map: when KServe, and when it is overkill

The useful comparison is not “KServe versus OME, vLLM Production Stack, Dynamo or llm-d”. Those four are covered in LLM inference operators and compete on a different axis:

Axis	What gets decided	Who competes
Contract and control plane	Which CRD describes a service, which API clients see, how the model is versioned, how the rollout is done, how LLM and non-LLM coexist	KServe, Seldon Core v2
Engine and topology	Which engine runs, how prefill/decode is disaggregated, how routing is done with cache awareness, how multi-node tensor parallel is done	vLLM Production Stack, OME, NVIDIA Dynamo, llm-d

The axes are orthogonal, and LLMInferenceService proves it: KServe did not reimplement distributed scheduling, it leaned on llm-d. Choosing KServe is not choosing instead of an LLM operator: it is choosing the contract layer that operator lives on.

Choose KServe when the catalogue is heterogeneous, LLMs plus embeddings, rerankers, classifiers and vision, which is its decisive advantage and the one none of the four operators has: a service like the one in TEI in production fits into the same vocabulary as the LLM. Also when you need a contract decoupled from the engine because you have clients you do not control; when you need declarative, auditable governance (which artifact, from which origin and with which runtime, in an object versionable in git and subject to admission policies), the piece that makes the rest of this series possible; and when you want a catalogue of platform runtimes separated from the teams’ services.

KServe is overkill when you serve one model, on one node, with up to three replicas: the Deployment from vLLM on Kubernetes is the right answer. When your workload is LLM only and the priority is performance per GPU, because there the value is in the engine. When you are going to use neither scale-to-zero nor canary by revision, because what remains is a Deployment with more YAML. And when you already have an LLM operator and only serve LLMs: putting KServe on top for completeness is adding CRDs without adding decisions.

For an inference factory

First: separate the runtime catalogue from the service catalogue, and do it on day one. The ClusterServingRuntimes belong to the platform; the InferenceServices, to the teams. That boundary turns “serving the thirtieth model” into a five-line change instead of a negotiation about vLLM flags, and it fits with GitOps with Flux: the runtime is reviewed as platform code, the service as product configuration.

Second: accept the two contracts and put the gauge changer where it belongs. V2 for the tensorial world, the OpenAI API for the generative one, and the translation in the gateway, not in every client. And since the OpenAI API does not version in the route, encode the version in the model name and make that name the same identifier that appears in the registry, in the signature and in the traces. That is what makes a request traceable back to the artifact.

Third: measure start-up before promising elasticity. The time from the pod being scheduled to the readiness probe going green, with the real model and from the real origin, decides whether scale-to-zero is a FinOps lever or an incident machine. That number, and not the price of the GPU-hour, is the one to take into the cost conversation.

And here this piece falls short on purpose. KServe gives you the contract, the lifecycle and the rollout, but its whole trust model hangs off a text string: storageUri. The storage initializer downloads whatever is at that destination and puts it in front of the GPU without asking anything else; nothing in the InferenceService guarantees that the artifact is the one that was approved, that it has not changed since the last audit, or that the origin is legitimate. The next article goes there: where the model bytes come from, and why an OCI registry with ORAS, Harbor, MLflow or the Kubeflow Model Registry is the difference between distributing models and scattering them.

Sources

CNCF, KServe becomes a CNCF incubating project (11 de noviembre de 2025) — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
CNCF, KServe project page (incubating, aceptado el 29 de septiembre de 2025) — https://www.cncf.io/projects/kserve/
CNCF TOC, Project Moving Levels Checklist: KServe joining CNCF at Incubation level — https://github.com/cncf/toc/issues/1905
KServe, Releases (v0.15.0, v0.16.0, v0.18.0, v0.19.0) — https://github.com/kserve/kserve/releases
KServe, Inference Protocol V2 (Open Inference Protocol) — https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol
KServe, open-inference-protocol (especificación, adoptantes, gobernanza) — https://github.com/kserve/open-inference-protocol
KServe, open-inference-protocol — generate_rest.yaml — https://github.com/kserve/open-inference-protocol/blob/main/specification/protocol/generate_rest.yaml
open-inference, Vendor neutral fork of kserve/open-inference-protocol — https://github.com/open-inference/open-inference-protocol
KServe, Generative Inference — Runtime Overview (endpoints con prefijo openai/) — https://kserve.github.io/website/docs/model-serving/generative-inference/overview
KServe, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
KServe, Cloud-Native AI Inference at Scale using KServe and llm-d (5 de marzo de 2026) — https://kserve.github.io/website/blog/cloud-native-ai-inference-kserve-llm-d
KServe, Serving Runtime (ServingRuntime y ClusterServingRuntime) — https://kserve.github.io/website/docs/concepts/resources/servingruntime
KServe, Kubernetes Deployment Installation Guide (K8s 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1, sin scale-from-zero en Standard) — https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment
KServe, Canary Rollout Strategy — https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary
KServe, Autoscaling with KEDA — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/keda-autoscaler
KServe, Multi-node/Multi-GPU Inference (PVC RWX, sin autoescalado) — https://kserve.github.io/website/docs/model-serving/generative-inference/multi-node
KServe, Storage Options for Model Artifacts — https://kserve.github.io/website/docs/model-serving/storage/overview
KServe, issue #5335 — Support canaryTrafficPercent for RawDeployment mode via Gateway API HTTPRoute weights (abierta, 2 de abril de 2026) — https://github.com/kserve/kserve/issues/5335
KServe, issue #5062 — Add startupProbe to vLLM main containers (12 de febrero de 2026) — https://github.com/kserve/kserve/issues/5062
KServe, modelmesh-serving (última release v0.12.0 de julio de 2023; repositorio archivado el 14 de abril de 2026) — https://github.com/kserve/modelmesh-serving
Red Hat, Converting ModelMesh and Serverless InferenceServices to RawDeployment (Standard) Mode — https://access.redhat.com/articles/7134025
NVIDIA, Triton Inference Server — Inference Protocols and APIs — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inference_protocols.html
OpenVINO, What is OpenVINO Model Server (API KServe, TFS y endpoints compatibles con OpenAI) — https://docs.openvino.ai/2026/model-server/ovms_what_is_openvino_model_server.html
PyTorch, TorchServe — Notice: Limited Maintenance — https://docs.pytorch.org/serve/
Kubernetes SIG Network, Gateway API Inference Extension (v1.5.0, 19 de abril de 2026) — https://github.com/kubernetes-sigs/gateway-api-inference-extension

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

Thu, 21 May 2026 07:15:00 +0200

TL;DR

Having an expensive GPU cluster and many different workloads that want to use it is not an infrastructure problem: it is an internal product problem. What separates “we have a cluster” from “we have an inference platform” are four layers the market consolidated in 2026: a gateway layer that centralises authentication, routing and policies (LiteLLM, Portkey, Kong AI Gateway); a GPU isolation model appropriate to the tenants’ profile (MIG hardware isolation for untrusted multi-tenant, MPS for processes from the same team, time-slicing only for dev); a quota and rate limiting system with budgets per tenant/team/project (LiteLLM does this in its core at team/user/api-key level with descriptive 429s); and a multi-tenant observability plane that enables real cost attribution (showback as an intermediate step, chargeback as the destination), per-tenant tracing and differentiated dashboards. Applied to a typical mid-scale GPU cluster (one node with 4-8 H100 SXM and NVLink, a common place to start in production), this translates into concrete decisions: with ~640 GB of aggregate VRAM across 8 GPUs and two typical production models (a large 70B+ model with tensor parallel and a replicated mid-sized model), the cluster serves between tens and low hundreds of simultaneous sessions depending on the mix; GPU isolation is usually resolved with MIG on smaller workloads and per-model dedication on large ones; and the platform’s success metric is effective utilisation, which in typical production sits at 30-40% and where the reasonable optimisation target is to push it to 60-70% without degrading SLAs.

This is the fifth post in the MLOps for LLMs series. It is the most operationally oriented one and cuts across several pipeline stages (Deploy + Observe plus cross-cutting concerns). The “you are here” marker flags the two active stages because the notion of a multi-tenant platform does not live in just one.

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

When a platform team buys expensive GPU hardware and starts building inference, the first version is almost always single-purpose: one model, one client, one target latency. It works. When the second team turns up asking for the same resource, that single purpose becomes internal politics: how many replicas do we give them? What do we do if the SLAs clash? Who pays for the tokens of team B’s experiment? And when the third one arrives, what was an SRE project becomes an internal product project.

The distinction is not technical, it is one of framing. A cluster is infrastructure. A platform is a service with customers, contracts and success metrics. The change of framing implies:

Identifiable customers (tenants), not anonymous users.
Contracts (latency SLA, guaranteed throughput, available models), not “whatever we manage”.
Success metrics that are not technical but product ones: adoption, satisfaction, cost per query per tenant, time to the first “hello world”.

This post walks through how that change of framing is carried out. It lands it on a mid-scale cluster (4-8 H100 SXM with NVLink in a single node), the usual configuration when starting with serious LLM inference; but the principles generalise to any topology, from a single node with two GPUs to multi-node clusters with InfiniBand.

The four layers of a multi-tenant inference platform

The canonical architecture that settled in 2026 has four layers that any serious multi-tenant platform implements, from the outside in:

Each layer solves a concrete problem. Let us take them one by one.

Layer 1 — AI Gateway: the single front door

The AI Gateway is the component your tenants see. It is an HTTP/gRPC API compatible with OpenAI (typically /v1/chat/completions, /v1/embeddings, /v1/models) that centralises everything that happens before touching the inference backends.

Why centralise

Without a gateway, tenants connect directly to vLLM or whatever the model is. Every change (rotating an endpoint, adding a model, changing credentials, applying a policy) means notifying every tenant. Every tenant has its own retry logic, its own logging, its own auth model. It is unworkable from the third customer onwards.

With a gateway, the change is made in one place. Tenants have a stable URL and a set of credentials; the rest is the gateway’s problem.

The three dominant options in 2026

LiteLLM is the most popular OSS option, Python-first, deployed as a proxy. It supports 100+ providers (OpenAI, Anthropic, Bedrock, self-hosted vLLM, Ollama, and so on) behind a unified OpenAI-compatible API. It has a native multi-tenant hierarchy with Organizations → Teams → Users → API Keys, each level with its own budget. The Apache 2.0 version covers the basics; RBAC, SSO, audit logs and team-level enforcement require the paid Enterprise version. Deployed on K8s with the official Helm chart.

Portkey is the most mature commercial / SaaS option. A single control plane that enforces budgets, quotas, permissions and compliance. Real-time spending tracking with alerting. RBAC, audit, workspaces and SSO included. The trade-off: dependency on an external service and a per-request pricing model.

Kong AI Gateway is the option for organisations that already have Kong as their API gateway. An AI plug-in on top of the existing Kong gateway, integrated with its plugin, consumer and rate-limit model. If your platform team already runs Kong, it is the lowest friction.

When to choose each one

Situation	Gateway
Pure OSS, self-hosted, Python-first team	LiteLLM
You need RBAC, SSO, audit log out of the box, and have budget	Portkey
You already run Kong as a corporate API gateway	Kong AI Gateway
Greenfield enterprise with strict compliance	Portkey (probably)
Mid-sized OSS-first company without regulated compliance	LiteLLM (typically)

The minimum the gateway has to do

Whichever option you pick, this is what any serious deployment must enforce:

Auth and identity: every request carries an API key resolvable to a tenant + user + team.
Routing by model: the tenant asks for model: "gpt-4o"; the gateway decides whether it goes to OpenAI, to Azure OpenAI, or to your vLLM with Qwen3 32B (a cheaper fallback), according to policy.
Rate limiting: RPS per tenant, TPM (tokens per minute), concurrency limits.
Caching of identical responses: 5-30% of RAG queries are repeats; caching saves latency and cost.
OTel emission: every call produces a span with gen_ai.* semantic conventions and tenant_id as an attribute. Covered in the Evals post and MCP observability.
Failover: if vLLM goes down, the gateway redirects to the OpenAI API. If OpenAI rate-limits, the gateway falls back to Anthropic. Configurable policy.

Example of a multi-tenant LiteLLM configuration

# litellm-config.yaml — simplified example
model_list:
 - model_name: llama-3-70b
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-llama3-70b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: qwen3-32b
 litellm_params:
 model: openai/qwen3-32b
 api_base: http://vllm-qwen3-32b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: gpt-4o
 litellm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

router_settings:
 routing_strategy: usage-based-routing-v2
 fallbacks:
 - llama-3-70b: [qwen3-32b, gpt-4o]  # if vLLM goes down, fall back to the external one

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL  # Postgres for budgets/keys

# Hierarchy: Organizations → Teams → Users → API Keys
# Created via the API, not in static YAML

Creating a team with a budget:

curl -X POST http://litellm/team/new \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_alias": "soporte-chat",
 "max_budget": 500, # 500 USD/month
 "budget_duration": "30d",
 "tpm_limit": 100000, # 100K tokens/min
 "rpm_limit": 1000, # 1000 requests/min
 "models": ["llama-3-70b", "qwen3-32b"] # access to these
 }'

And the team’s API key:

curl -X POST http://litellm/key/generate \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_id": "<team-id>",
 "duration": "30d",
 "metadata": {"environment": "production", "app": "support-bot"}
 }'

That API key is what the tenant uses. Every request made with it consumes from the team’s budget. When it runs out, LiteLLM returns HTTP 429 with a description.

Layer 2 — Policy & Quota Plane: what each tenant can do

The gateway is where enforcement happens. Policy is what gets enforced. Five axes of multi-tenant policy:

Technical quotas

TPM (tokens per minute): the hard consumption limit. For a Llama 3 70B at TP=5, ~3000 sustained output tokens/s = 180K aggregate TPM. If you have 10 tenants, assign 18K each as the ceiling.
RPS / RPM: load control, not consumption control. A 4K-token session counts as one request; so does a batch of 100 mini-completions. Useful against abuse.
Concurrency: how many simultaneous active requests per tenant. Important for latency SLAs: 100 RPS with concurrency=50 works out at 2 seconds per request.

Economic budgets

Monthly per tenant: a hard cap in USD.
Daily and hourly: soft caps to avoid a runaway in a single day.
Per project / API key: fine granularity within a single tenant.

LiteLLM has a max_budget field at every level of the hierarchy (organization, team, user, api key). Budgets are inherited and constrained downwards.

Model whitelist and blacklist

Tenants with critical workloads → only stable models (llama-3-70b, gpt-4o). Research tenants → access to experimental models as well.

Priority classes

Not all requests are equal. Three typical classes:

Guaranteed: workloads with an SLA, latency respected even under pressure.
Best-effort: normal workloads without a strict SLA.
Spot: batches that can wait, evictable if a guaranteed one arrives.

The Token Management in Multi-Tenant AI Inference Platforms paper (2026) formalises this with a model of token pools per priority class that has started to be adopted in production. It maintains guaranteed P99 latency for guaranteed workloads even under overload, with selective throttling of spot.

Admission control

Before accepting a request: is there capacity? If not, return 429 immediately instead of queueing and degrading everyone. It is the most underrated operational discipline. A cluster with admission control done properly has predictable latency; without it, catastrophic degradation when the peak arrives.

The typical pattern in 2026

# Conceptual policy for a "soporte-chat" tenant
tenant: soporte-chat
quotas:
 tpm: 50000
 rpm: 500
 max_concurrency: 30
budget:
 monthly_usd: 800
 alert_thresholds: [0.5, 0.8, 0.95] # warn when you get there
models_allowed:
 - llama-3-70b
 - qwen3-32b
priority: guaranteed
fallback_on_overload:
 - qwen3-32b  # if guaranteed fills up, fall back
 - gpt-4o-mini  # last resort, external model

Layer 3 — Isolation Plane: isolating workloads physically

This is the technically densest layer. You have a node with several H100 SXM GPUs interconnected by NVLink. How do you partition them between tenants?

MIG (Multi-Instance GPU) is the strongest isolation. It partitions the GPU into up to 7 instances with physically separate HBM memory and dedicated compute units (SMs). Tenants in different MIG instances cannot touch each other: one workload does not consume memory another needs, one does not degrade another’s throughput. Hardware isolation. Available on A100, H100, B100 and B200.

MPS (Multi-Process Service) is soft. Several processes share the GPU concurrently and NVIDIA distributes SMs according to usage. Good performance if all the processes are yours and you trust them. Worse for multi-tenant setups between customers who do not know each other, because a noisy process can degrade the others.

Time-slicing is the simplest: the GPU is assigned alternately, slot by slot, to different processes. Much worse latency (waits between slots); not recommended for production workloads with an SLA.

The choice for multi-tenant in 2026

According to the enterprise adoption survey: 80% use MIG for untrusted multi-tenant (different customers who do not know each other) and MPS for trusted environments (processes from the same team) where you want to maximise throughput. Time-slicing is only used in dev/staging so that every developer can touch a GPU without the cost of exclusivity.

An important limitation of MIG: it isolates compute and HBM memory, but the PCIe path remains shared. For PCIe-bound workloads (a lot of host↔device traffic), tenants in different MIG instances can still affect each other. For LLM inference the main path is HBM, so this is rarely a problem. But it is worth knowing.

MIG partitions on the H100

An H100 (80GB HBM3) can be partitioned into fixed profiles:

Profile	SM	Memory	Max instances per GPU
1g.10gb	14	10 GB	7
1g.20gb	14	20 GB	4
2g.20gb	28	20 GB	3
3g.40gb	42	40 GB	2
7g.80gb	98	80 GB	1 (the whole GPU)

For a mid-scale cluster with NVLink, MIG has a fundamental problem: when you partition with MIG, NVLink between GPUs is disabled. An H100 in MIG mode does not take part in multi-GPU tensor parallel. If you are going to serve a large model with tensor parallel (Llama 3 70B with TP=4 or TP=8, for instance), those GPUs have to be whole, without MIG.

That defines the architectural decision. There are two main approaches:

Approach A — A shared large model with quotas at the gateway

All the node’s GPUs serve a single large model with tensor parallel spanning the whole node. All tenants share that instance. Isolation happens at the gateway layer (quotas, rate limiting) and the policy layer (priority classes). The cluster’s kernel is one enormous vLLM instance with --max-num-seqs=128 or similar; internally, vLLM shares GPU time between the active requests with continuous batching.

Advantages: you use all the GPUs to the full, NVLink stays active, better KV cache utilisation. Disadvantages: soft isolation. A tenant that saturates does not degrade the others directly (vLLM batches), but it does compete for batch slots. You need serious priority classes.

Approach B — Dedicating GPUs per model / tenant

You split the GPUs into pools dedicated to different models. Examples on an 8-GPU node:

4 GPUs: a large 70B model with TP=4.
2 GPUs: a mid-sized 32B model replicated (2 independent instances) for tenants with a strict SLA.
2 GPUs: miscellaneous workloads (smaller models, experimentation).

Advantages: physical isolation between critical models / tenants. Disadvantages: worse aggregate utilisation; some GPUs idle while others saturate.

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

If you have small workloads (4B, 7B models), you can use MIG on 1-2 GPUs to serve them and dedicate the rest to tensor parallel for the large model. It combines strong isolation for the small workloads with full use of NVLink for the large model.

The operational choice: start with A, move up to C if you need to

In most deployments, Approach A (a shared large model plus quotas) is the right starting point. Utilisation is better, operation is simpler, and the gateway’s soft isolation works for reasonable workloads.

When there is a tenant with a strict SLA that cannot tolerate competing with others, you move to Approach B for that tenant in particular (dedicating GPUs to an instance of the model just for them), keeping the rest of the cluster shared.

Approach C is for when you have 10+ tenants with very heterogeneous profiles.

Isolation at the Kubernetes level

Independently of GPU isolation, pod-level isolation is applied in K8s:

Namespaces per tenant: tenant-soporte, tenant-legal, and so on.
ResourceQuotas and LimitRanges: CPU/memory limits per namespace.
NetworkPolicies: tenant A cannot talk to tenant B’s namespaces.
K8s PriorityClasses: classes with a numeric value defining preemption order if a more critical pod arrives.
PodDisruptionBudgets: how many pods of each deployment can go down simultaneously.

Layer 4 — Observability Plane: seeing what happens per tenant

The fourth layer: observability with a tenant dimension. Without it you cannot do cost attribution, you cannot debug incidents for a single tenant, you cannot show dashboards to stakeholders.

The four mandatory properties

1. tenant_id on every span. The AI gateway resolves the API key and attributes a tenant_id. That ID is propagated via params._meta or OTel headers to every downstream component (vLLM, retrieval, MCP servers, tools). Any span in any system carries that label. It is what lets you reconstruct tenant-specific traces.

2. Metrics labelled per tenant. gen_ai.usage.input_tokens{tenant="soporte-chat"} or equivalents. Prometheus, Grafana, groupable by tenant.

3. Real cost attribution. The sum of tokens × cost/token per tenant gives the cost. For self-hosted vLLM, the cost is per GPU hour plus the proportional share of tokens (you can compute an equivalent cost per 1k tokens).

4. An immutable audit log. Every API key used, every model invoked, every quota change, every budget exceeded. For compliance.

Showback vs chargeback

An important FinOps distinction that gained clarity in 2026:

Showback: visibility without consequence. “Support team, you have consumed 623 USD this month on LLMs”. Information, not an invoice. It lets you spot abuse without penalising anyone before the team understands what is going on.

Chargeback: the cost is charged to the team’s budget. When it runs out, it runs out. It changes behaviour.

The practice that works: 6-18 months on showback while tags are calibrated, misattributions are identified and teams are trained. Then chargeback, once the numbers are credible. Launching chargeback on day 1 when the costs are still dirty creates an immediate political fight; launching showback prepares the ground so that chargeback lands in an orderly way.

Only 14% of organisations have chargeback active according to a recent survey, which indicates this is still mostly showback in real production.

Tools

Kubecost: cost allocation per namespace, deployment and pod on Kubernetes. For the cost of the shared GPU, allocate proportionally to the tokens consumed per tenant.
Finout: a FinOps platform that combines cloud bills and LLM API costs in a unified view with virtual tagging.
Langfuse: already covered. Cost tracking per trace, groupable by user or session metadata.
LiteLLM native tracking: LiteLLM’s master DB keeps running spend per team, user and API key, accessible via the API or the UI.

The minimum multi-tenant dashboard

Any platform should have:

Summary per tenant: monthly spend, current RPS, TPM consumed, % of budget spent, active sessions.
Top users within each tenant (for detecting internal abuse).
p95 latency per tenant: SLA tracking.
429 / 503 errors: how many requests are being rate-limited or rejected because of overload.
Cost trend: the monthly trajectory with a projection.
Drift per tenant (from the post-tracing series): if a tenant starts getting worse results, raise an alert.

Sizing on mid-scale GPU clusters: concrete decisions

Let us get down to hardware. As a reference we take a node with N H100 SXM (between 4 and 8) with NVLink/NVSwitch, 80 GB HBM3 each. That gives between 320 GB and 640 GB of aggregate VRAM. Inter-GPU connectivity is 900 GB/s (NVLink 4) or 600 GB/s (NVLink 3) depending on generation. HBM bandwidth per GPU is 3.35 TB/s.

Default decisions

Start with Approach A: all the node’s GPUs serving a single large 70B model in BF16 with tensor parallel = N. Expected real capacity (computed for a standard 8-GPU HGX node as an example; it scales roughly linearly with N):

Model VRAM (70B BF16): ~140 GB (≈ 17.5 GB/GPU at TP=8).
vLLM overhead + activations VRAM: ~10 GB/GPU.
VRAM free for KV cache: ~52 GB/GPU. On an 8-GPU node that is ~416 GB aggregate; on a 4-GPU one, ~210 GB.
With --kv-cache-dtype=fp8 and a 70B GQA model: ~320 KB/token.
Aggregate cache capacity (8-GPU node): ~1.3M tokens to distribute between simultaneous sessions.

That translates into throughput and concurrency (indicative figures for an 8-GPU node):

Simultaneous sessions	Average context per session	Aggregate throughput (tokens/s)
32	16K	~5000
64	8K	~8000
128	4K	~12000

Typical latencies: TTFT ~150ms at low traffic, TPOT ~15-20 ms/tok. Under high concurrency, TTFT rises to ~500ms if the queue is saturated.

An example tenant scheme

A cluster with 4 tenants and a research pool:

Tenant	TPM cap	RPM cap	Concurrency	Budget	Priority	Models
Support chat	80K	800	50	1500 USD/month	Guaranteed	llama-3-70b, qwen3-32b
Legal RAG	30K	200	15	600 USD/month	Guaranteed	llama-3-70b
Code agent	50K	300	25	1200 USD/month	Best-effort	llama-3-70b, qwen-coder
Data extr. batch	40K	1000	40	400 USD/month	Spot	llama-3-70b, qwen3-32b
Research / notebooks	10K	100	5	200 USD/month	Spot	all

TPM total: 210K. Aggregate cluster capacity: ~180K sustained TPM. It is overcommitted by about 15%, assuming not every tenant hits the ceiling at the same time. That is normal and desirable; if they all do it at once, the priority classes degrade in an orderly way.

When to add hardware

Signals that the node has become too small:

Sustained p95 TTFT > 500 ms during peak hours → the queue is building up.
vllm:num_requests_waiting constantly > 20 → admission control starting to reject.
Sustained GPU utilisation > 80% at critical hours without dropping in quiet hours → there is no headroom.
429 rate on guaranteed tenants > 1% → the platform is breaking its SLA in production.

When several of these hold, the natural next step is to add another HGX node with internal NVLink and stand up a second vLLM instance of the same model. The gateway load-balances between the two instances. Aggregate throughput doubles; latency stays the same.

Common operational traps

A gateway without auth: a backdoor into the cluster

Your vLLM sits on a ClusterIP Service and the main app talks to it. Some tenant discovers the endpoint directly and hits it without going through the gateway. Quotas and costs are silently evaded. Use a strict NetworkPolicy: only the gateway can talk to the vLLM Services; the rest of the cluster cannot.

MIG and NVLink are incompatible

You enable MIG on a GPU thinking you will get isolation plus multi-GPU; you discover that MIG disables NVLink. Any large model with TP becomes unusable. Decide MIG vs NVLink globally per cluster, not per individual GPU.

Quotas pinned to the cluster’s ceiling

You add up the TPM of all the tenants and it comes to exactly the cluster’s capacity. When two tenants peak at the same time, both wait or one gets rejected. An overcommit of 10-20% is healthy (it assumes they do not all peak at once); more than that is dangerous.

No multi-tenant observability from day 1

You launch with quotas and isolation but without tenant_id on spans. Three months later, your CFO asks “how much does the support agent cost versus the legal one?” and you cannot answer. OTel with a mandatory tenant_id from the first version, even if there are no dashboards yet; having the data is worth more than having perfect dashboards with no data.

Showback that never reaches chargeback

You have been on showback for 18 months, the teams know the numbers, nobody changes behaviour. Without the pressure of real chargeback, the incentive dilutes. Set an explicit calendar for the transition to chargeback, with an owner and a deadline.

Non-whitelisted models eating the budget

A team discovers that LiteLLM has gpt-4o configured. They use it without permission. The budget burns on an external API when the idea was to use the cheap self-hosted one. Keep an explicit per-team whitelist of accessible models.

Badly calibrated priority classes

Everyone declares themselves “guaranteed”. At the first peak, there is nothing left to degrade and everything suffers. Priority classes only for critical cases, with justification. The majority should be best-effort.

No failover from the gateway

Your vLLM goes down. The gateway has no fallback configured and returns 503 to every tenant. Configure a fallback to another model, ideally an external one (OpenAI) for guaranteed workloads, even if it costs more per hour. Availability is worth more than cost per hour.

An operational roadmap for getting started

If you start from scratch with an empty GPU node, the minimum order is as follows. Each milestone is a day’s work with slack, not a tight one:

Day 1-2 — Base K8s infrastructure. NVIDIA GPU Operator + nvidia-device-plugin + dcgm-exporter + cluster-default NetworkPolicies. Validation: a basic pod with nvidia.com/gpu: 1 gets scheduled.

Day 3 — vLLM with a large model and tensor parallel across the whole node. The vLLM Production Stack Helm chart (or bare vLLM manifests). Model weights on a shared PVC (CephFS or NFS). Validation: a curl request against the internal Service responds.

Day 4 — AI Gateway: LiteLLM. Helm chart, Postgres for budgets, master key, a first model_list pointing at vLLM. Validation: an OpenAI-compatible request through LiteLLM responds with the same content as vLLM directly.

Day 5 — Basic multi-tenancy. Create teams, API keys, budgets, model whitelists. Test with two teams. Validation: the second team, using the model it does not have whitelisted, gets a 403.

Day 6 — Minimum observability. Prometheus + Grafana scraping vLLM and LiteLLM. A dashboard with TTFT, TPOT, throughput, num_requests_waiting and budget_consumed_per_team. Validation: visible in Grafana with real data.

Day 7-8 — Pilot customer. One real tenant (ideally a controlled internal one) starts using it. Measure real latencies, discover the first operational incidents.

Day 9-10 — Tuning. Adjust --max-num-seqs, --gpu-memory-utilization, priority classes and quotas based on what the pilot taught you.

Day 11-14 — Onboarding the second tenant plus iteration. Repeat. Every new tenant onboarded reveals new cases.

After two weeks you have an operational platform with two real tenants and data to decide whether it is ready for more. The line of progress from here on is horizontal (more tenants) until you saturate; from then on, vertical (more hardware).

What we have not covered (upcoming posts)

Continuous fine-tuning in production (post 6, already decided): LoRA/QLoRA/DPO, dataset curation, eval gates, A/B versioning with real traffic between model versions.
Constitutional AI and alignment at runtime: an option still on the table.
Edge LLMs: when an H100 cluster is too expensive for a specific workload, distilled models running on NPUs or consumer GPUs.
GPU networking deep dive: NCCL, InfiniBand, GPUDirect, RDMA. For multi-node clusters with cross-host tensor parallel.

References

Multi-tenancy and GPU isolation:

Multitenant GPU Infrastructure: 4 Powerful Design Rules — survey of enterprise patterns.
Run Multiple LLMs on One GPU: MIG, Time-Slicing, and MPS Guide (Spheron).
A Practical Guide to GPU Partitioning with MIG (Medium).
GPU Partitioning for AI Workloads: NVIDIA MIG with SUSE Virtualization (KubeCon EU 2026).
Predictable LLM Serving on GPU Clusters (arxiv 2508.20274).
Token Management in Multi-Tenant AI Inference Platforms (arxiv 2603.00356) — paper on priority + admission control.

AI Gateways:

Multi-tenant FinOps:

Cross-references:

Previous posts in series 4: MLOps for LLMs overview, RAG over Kafka, The six-stage pipeline, PostgreSQL + Qdrant.
Relevant posts from the inference series: vLLM on Kubernetes — the multi-GPU HGX node scenario we develop here. LLM operators on K8s — the vLLM Production Stack and OME that the gateway can direct.
Observability: Evals, MCP observability, eBPF + drift.

LLM inference operators on Kubernetes: OME, vLLM Production Stack, NVIDIA Dynamo and llm-d

Mon, 18 May 2026 17:00:00 +0200

TL;DR

Serving an LLM in production is not running a binary: it is coordinating a model (tens of gigabytes that take minutes to load), a runtime (vLLM, SGLang, TensorRT-LLM with a hundred flags), heterogeneous GPUs (NVLink, MIG, PCIe), prefill and decode that live better apart, a KV cache that wants offloading to colder tiers, intelligent routing that exploits prefix caching, and autoscaling that reacts to metrics other than CPU%. A flat Kubernetes Deployment only covers the first 20% of that. The other 80% is covered by LLM inference operators, of which four matter in 2026: OME (LMSYS, July 2025, multi-engine with a focus on SGLang), vLLM Production Stack (a curated Helm chart from vLLM itself with LMCache for tiered KV), NVIDIA Dynamo (the official successor to Triton, multi-engine, with its own Grove scheduler) and llm-d (a CNCF donation of March 2026 by Red Hat + Google + IBM + CoreWeave + NVIDIA, built on vLLM, focused on distributed scale). Behind all four sits KServe, the CNCF parent operator that normalised the InferenceService concept and that several of them lean on. This article walks through the full hierarchy, gives a decision map and teaches you not to get lost when someone drops seven acronyms in the first meeting.

This article closes the LLM inference series. The previous ones were KV cache: the working memory that holds up LLM inference, vLLM on Kubernetes: the LLM inference piece that actually scales and PagedAttention from the inside and the state of the art of the KV cache in 2026. There we explained what happens inside one inference process. Here we explain how many inference processes are coordinated through Kubernetes.

The analogy: from `init.d` to systemd to operators

Anyone with 20 years of sysadmin behind them will recognise the pattern. Decades ago, starting a service on Linux was a shell script in /etc/init.d/: start, stop, status, reloaded by hand. When services got more complex, with dependencies between them, monitoring, restart on failure, per-user slots, it became obvious that a script was not enough. Along came systemd, which turned “a service” into a declarative unit with dependencies, resources, restart policy, sockets and timers. The script did not disappear; it moved up one level of abstraction.

Kubernetes made the same move for distributed services. A Deployment declares “I want N replicas of this container”; a Service declares “these replicas are exposed like this”; an Ingress declares “this HTTP traffic comes in here”. The controller translates the declaration into real state and keeps the system converged.

Serving LLMs in 2024 was the equivalent of /etc/init.d/: every team wrote its own Deployment/Service/HPA with custom scripts for model loading, session draining and GPU handling. We covered it in the vLLM on Kubernetes article: it can be done, and it does work, but it is repetitive, fragile and nobody is extracting the right abstractions. Serving LLMs in 2026 has gone through the same transition as services: the equivalent of systemd has appeared, the inference operators, which normalise the abstractions and let the engineer declare what matters: “this model, with this runtime, this scalable, with this routing policy”.

There are four relevant operators in 2026 and a fifth common ancestor. Let us take them in order.

Why an operator, and not just a Deployment

Listing what an inference operator adds over a flat Deployment is the best way to understand what problem it solves:

The model as a first-class citizen. In a Deployment, the model is “the thing you download in an initContainer and mount as a volume”. In an operator, the model is a CustomResource with metadata (origin, fingerprint, licence, GPU requirements). It can be shared between InferenceServices, versioned, replicated to multiple nodes. It is the difference between “a file” and “a managed artifact”.

The runtime as a first-class citizen. The same goes for the runtime (vLLM/SGLang/TRT-LLM): it is not “a Docker image with flags”; it is a ServingRuntime that declares which args it accepts, which metrics it exports, which deployment types it supports (single-node, multi-node TP, PD-disag). Changing runtime is changing a reference, not rewriting every manifest.

Declarative composition. An InferenceService (the core CRD of KServe and its descendants) references a model and a runtime, declares the scaling policy, wires up observability and configures routing. The controller composes all the pieces: Deployment(s), Service, HPA, possibly a LeaderWorkerSet, a KEDA ScaledObject, a Gateway API HTTPRoute. You declare intent; the operator emits the 8 derived resources.

Operational prefill–decode disaggregation. As we saw in the PagedAttention article, separating prefill and decode into distinct pools can give 7× goodput. Modelling that with flat Deployments is feasible, but it requires coordinating two sets of pods, a transport to move the KV cache and conditional routing. An operator models it as a single InferenceService with two sub-pools.

Autoscaling on LLM metrics. The standard HPA does not understand vllm:num_requests_waiting. An operator integrates KEDA or Prometheus Adapter automatically and exposes the right metrics as knobs on the CRD.

Multi-tenancy. Multiple models on the same cluster, with quotas, priorities and fairness. One Deployment per model scaling independently is fine up to the fifth model; beyond that, coordinating GPUs between tenants becomes operationally hostile.

Model lifecycle. Weights on a shared PVC, warm-up of the first pod, rolling updates with maxUnavailable: 0, draining of active sessions, integrated observability. Things that with a flat Deployment have to be reinvented by every team.

If your workload is one model, one node, up to three replicas, a flat Deployment is enough and an operator is overkill. If your workload is two or more models, serious scaling, disaggregation or multi-tenancy, an operator stops being optional.

KServe: the common ancestor

Before the four new ones, we have to mention KServe, the parent operator from which all the rest conceptually descend. It was born as KFServing inside the Kubeflow project in 2019, was renamed KServe when it became independent in 2021, and in 2025 was accepted into the CNCF as a project incubating towards graduation.

KServe’s conceptual contribution is the InferenceService CRD, which has become the field’s common vocabulary: a declarative K8s object that joins a model (origin + metadata) with a predictor (runtime + resources) and produces a ready HTTP service. Under the hood, the controller emits Deployments, Services, HorizontalPodAutoscalers, Knative Services if you go serverless, Istio VirtualServices if you do traffic splitting.

KServe was designed in a pre-LLM era: its first use cases were traditional scikit-learn, TensorFlow and PyTorch models served as simple REST APIs. That gives it strengths (it is mature, with 6 years in production at Bloomberg, JPMorgan and others) and weaknesses (it was not designed to manage multi-node tensor parallel, prefill–decode disaggregation, or the patterns specific to LLMs).

The way the ecosystem has reacted is elegant: the new LLM operators inherit from or take inspiration from InferenceService but extend the API with LLM-specific primitives. OME is the clearest example: it uses the name InferenceService and the idea of “model + runtime → service”, but adds BaseModel, a ServingRuntime with LLM-aware flags, and deployment modes (PD-disag, multi-node) that KServe does not cover natively.

OME (Open Model Engine)

OME was published by the LMSYS team in July 2025 (announced on their blog). It is an operator that understands SGLang deeply (that is its first-class runtime) but also supports vLLM, TensorRT-LLM and Triton.

The CRD hierarchy

OME models the domain with four main CRDs:

BaseModel and ClusterBaseModel: the model itself. It defines origin (Hugging Face, S3, URL), fingerprint and metadata. The Cluster* version is global; BaseModel is namespaced. It lets multiple InferenceService objects reference the same model without duplicating the download.
FineTunedWeight: LoRA adapters or finetuned weights served on top of a BaseModel. Critical for multi-tenant setups where each customer has their own finetune.
ServingRuntime and ClusterServingRuntime: the runtime (vLLM, SGLang, and so on) with its configuration. It declares which args it accepts, which metrics it exports, which deployment modes it supports.
InferenceService: the central declarative piece, joining BaseModel + ServingRuntime + infrastructure.

apiVersion: ome.io/v1beta1
kind: InferenceService
metadata:
 name: llama3-70b-prod
 namespace: inference
spec:
 model:
 name: meta-llama-3-70b-instruct  # reference to a BaseModel
 runtime:
 name: sglang-h100  # reference to a ServingRuntime
 deploymentMode: PrefillDecodeDisaggregated  # standard | PD | MultiNode | Serverless
 prefill:
 minReplicas: 2
 maxReplicas: 8
 resources:
 requests:
 nvidia.com/gpu: 4
 decode:
 minReplicas: 4
 maxReplicas: 16
 resources:
 requests:
 nvidia.com/gpu: 1
 router:
 type: cache-aware  # SGLang router with cache awareness
 autoscaling:
 metricSource: keda
 metrics:
 - type: prometheus
 metricName: vllm_requests_waiting
 threshold: "10"

That is what the operator takes as input. The output is roughly 8 derived resources that would be horrible to declare by hand: two LeaderWorkerSets (one per prefill/decode pool), two Services, a Deployment for the router, KEDA ScaledObjects for each pool, a Gateway API HTTPRoute, and a PriorityClass that hooks into Kueue for gang scheduling.

The four deployment modes

OME materialises the InferenceService differently depending on deploymentMode:

Standard: one Deployment with N replicas; the classic. For small or single-GPU models.
PrefillDecodeDisaggregated: two coordinated pools; the SGLang router routes between them.
MultiNode: tensor parallel across multiple nodes via LeaderWorkerSet, with NCCL/InfiniBand. For models above 70B where a single node is not enough.
Serverless: Knative-style scale-to-zero. For sporadic workloads where the cost of keeping GPUs powered up does not pay off. The trade-off: the first request pays the model’s cold start cost (minutes).

Integration with the K8s ecosystem

OME does not invent primitives where they already exist. It leans on:

Kueue for gang scheduling: all the pods of a tensor parallel group have to start at once or not at all; Kueue guarantees it.
LeaderWorkerSet (LWS) for multi-node: workers join the leader’s Ray cluster, with an atomic lifecycle (one going down restarts the group).
KEDA for autoscaling on LLM-specific Prometheus metrics (queue depth, GPU cache usage, TTFT p95).
Gateway API and its Inference Extension for advanced routing (model-aware, prefix-aware, weighted canary).

The consequence: OME feels “idiomatically Kubernetes”. It does not introduce new concepts where they are not needed; it uses standard primitives and concentrates on what is specific to the LLM domain.

When to choose it

OME is the natural choice if SGLang is your main runtime and/or if you come from the KServe ecosystem and want an idiomatic evolution of it. It is mature but relatively young (a year old at the time of writing); expect rough edges on advanced features.

vLLM Production Stack

vLLM Production Stack is the official project from vLLM itself for production on Kubernetes. Its philosophy is the opposite of OME’s: instead of an operator with new CRDs, it is a curated Helm chart that deploys a coherent set of pieces.

The three pieces

The stack has three components:

Serving engines: vLLM pods, configured with the flags we have been seeing throughout the series (--enable-prefix-caching, --kv-cache-dtype fp8, and so on). The Helm chart lets you declare them as a list; it deploys the underlying Deployments and Services.
Request router: a proxy in front of the engines that decides which one each request goes to. It supports several policies:
- Round-robin: trivial, for a baseline.
- Session-based: pins each session to one replica to keep its KV cache.
- Prefix-aware: detects shared prefixes between requests and routes them to the replica that already has them cached.
- KV-aware: sees each replica’s gpu_cache_usage_perc and avoids the saturated ones.
- Disaggregated-prefill with native LMCache: separates prefill and decode, with LMCache as the transport for the KV cache between the two.
Observability stack: Prometheus + Grafana with ready-made dashboards. It measures TTFT, TBT (Time-Between-Tokens), throughput, queue depth and GPU memory.

LMCache and tiered KV

One of the most interesting pieces the stack brings is LMCache, which adds a KV cache with multiple tiers: GPU HBM as L1, CPU RAM as L2, local disk as L3, and optionally remote storage as L4. When a KV cache block does not fit in HBM, instead of evicting and recomputing it, LMCache drops it to a lower tier. For workloads with shared prefixes and multi-turn conversations, the saving is huge.

LMCache integrates as a sidecar to the engines and as part of the transport in disaggregated-prefill. Production Stack ships it enabled by default in its Helm chart.

A typical manifest (values.yaml)

servingEngineSpec:
 modelSpec:
 - name: llama3-8b
 repository: vllm/vllm-openai
 tag: v0.6.3
 modelURL: meta-llama/Meta-Llama-3-8B-Instruct
 replicaCount: 3
 requestCPU: 4
 requestMemory: 16Gi
 requestGPU: 1
 vllmConfig:
 enablePrefixCaching: true
 kvCacheDtype: fp8
 maxModelLen: 32768
 enableChunkedPrefill: true

routerSpec:
 routingLogic: prefix-aware  # round-robin | session | prefix-aware | kv-aware
 sessionKey: x-user-id  # when routingLogic=session

cacheserverSpec:
 enabled: true # LMCache for tiered KV
 storageBackends:
 - cpu
 - disk  # offload to local disk

observabilitySpec:
 prometheus:
 enabled: true
 grafana:
 enabled: true
 dashboards:
 - vllm-engine-metrics
 - lmcache-metrics

This is declarative but these are not CRDs: they are Helm chart values. The difference with OME is not semantic (both start from declaration) but operational: with Helm, changes go through helm upgrade; with CRDs, they go through kubectl apply. For teams already living in GitOps with Argo CD or Flux, both approaches integrate cleanly, but the flows are different.

When to choose it

If vLLM is your only runtime and you want the closest thing to “the happy path the project recommends”, this one. It is the productionised version maintained by the same people who write the engine. The downsides: it ties you to vLLM (it is not generic) and it does not solve some advanced cases such as multi-tenancy with strict quotas or gang scheduling, where OME or full-fledged operators are better.

NVIDIA Dynamo

NVIDIA Dynamo is the official successor to Triton Inference Server, announced at GTC 2025 and merged into the brand as Dynamo-Triton in March of that year. Triton had spent years as the most-used inference engine in “serious” NVIDIA infrastructures; Dynamo is what NVIDIA believes the new generation needs.

What it is exactly

Dynamo is a distributed inference framework, not exactly a Kubernetes operator. It has its own runtime (it can run engines), a scheduler (Grove), intelligent routing, multi-tier KV cache management and disaggregation. It supports SGLang, TensorRT-LLM and vLLM as engines, but the engines are executed by Dynamo, not the other way round: the model is “Dynamo manages, the engine executes”.

On Kubernetes, Dynamo is deployed via its own operator plus CRDs, normalised with the K8s integration NVIDIA formalised at the end of 2025 (covered by this InfoQ piece). The CRDs are product-specific: they define a DynamoCluster, a topology of prefill/decode workers and a routing policy.

The four contributions

Dynamo is sold on four pillars, with figures reported by NVIDIA:

Disaggregated serving built in, with its own scheduler.
Smart routing based on cache state: if a worker already has most of a prompt cached, the request goes there.
Multi-tier KV cache: analogous to LMCache, with HBM/RAM/SSD/NVMe.
Autoscaling integrated with Dynamo’s scheduler.

The marketing figure: up to 30× more throughput than legacy Triton on the same hardware. With all the caution a vendor benchmark deserves.

Grove: its own scheduler

A controversial decision in Dynamo is not to rely 100% on the Kubernetes scheduler and instead bring its own scheduler, called Grove, that understands GPU topologies. Grove decides which worker runs on which physical GPU, which interconnects (NVLink/InfiniBand) are relevant, and how to distribute tensor parallel across nodes. That gives it more control than the standard kube-scheduler.

Operationally: if your cluster is “pure Kubernetes” with kube-scheduler and heterogeneous workloads (not only LLMs), Grove adds one more component to operate. If your cluster is dedicated to LLM inference and there is already a team dedicated to running it, Grove gives you more levers.

When to choose it

Dynamo makes sense if:

Your infrastructure is NVIDIA-heavy (Hopper, Blackwell, GB200) and you want to exploit the latest in TensorRT-LLM with the familiar Triton integration, modernised.
You were already a Triton user for legacy inference (vision, recommendation) and want to keep the ecosystem.
You have a dedicated SRE team for inference and Grove’s extra operational complexity is not a problem.

It is the vendor-specific option of the quartet. In exchange it gives you NVIDIA’s support and first-rate integration with their hardware. If your organisation is already fighting NVIDIA for GPUs, they may even call you offering help with Dynamo.

llm-d

llm-d is the youngest and the most “political” of the four. In March 2026, at KubeCon Europe Amsterdam, Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA announced the joint donation of the project to the CNCF as a Sandbox project, with backing from AMD, Cisco, Hugging Face, Intel, Lambda, Mistral AI, UC Berkeley and the University of Chicago. An explicit coalition for vendor neutrality.

Philosophy

llm-d positions itself as the vendor-neutral “Kubernetes blueprint” for distributed inference. It is not a runtime; it is a system that sits on top of vLLM (the default engine) and orchestrates the control plane.

The primitives the project puts on the table:

Intelligent routing with prefix-cache awareness and load-aware balancing.
Tiered KV cache with offload to CPU and disk for multi-turn.
Prefill/decode disaggregation over fast interconnects.
Wide expert parallelism for serving very large Mixture-of-Experts (MoE) models, a critical pattern popularised by DeepSeek-V3 and Mixtral, where the experts live on different GPUs and tokens have to be routed to the right expert.

Numbers

The v0.5 release validates ~3.1k tok/s per decode GPU on B200, and up to 50k output tok/s on a 16×16 B200 prefill/decode topology. The most interesting benchmark: an order-of-magnitude reduction in TTFT against a round-robin baseline. In other words, intelligent routing is worth what they say it is.

CNCF and the future

Donating to the CNCF as a Sandbox project means neutral governance: no vendor is in charge. For an organisation wary of being tied to a single supplier, llm-d is probably the safest medium-term bet. The price: like any Sandbox project, it is not yet “boring” in the way vLLM is. There is API churn, features that move around, documentation that lags the code.

When to choose it

llm-d makes sense if:

You want multi-vendor portability without ties to NVIDIA, Red Hat or Google.
Your workload includes large MoE models (DeepSeek-V3, Mixtral 8x22B, Llama 4 Behemoth if the size is confirmed), where wide expert parallelism is decisive.
Your organisation is already comfortable with CNCF Sandbox (projects under active evolution, not yet a stable 1.0).
You want to bet on the project that will probably be the de facto standard in 2-3 years.

The common ancestor is still there: KServe

It is worth reconnecting before the comparison: KServe is still alive and heavily used in organisations that serve both LLMs and traditional models (scikit-learn, XGBoost, PyTorch CV). Its InferenceService is generic enough to serve any model, including vLLM or SGLang as a ServingRuntime. What it does not do well is the LLM-specific part: disaggregation, multi-node tensor parallel, routing with KV cache awareness. If your organisation already has KServe in production for other models, adding an LLM-specific operator alongside it (OME, vLLM Stack or llm-d) is reasonable. Fighting it all out from pure KServe is not.

Decision map

Dimension	OME	vLLM Prod Stack	NVIDIA Dynamo	llm-d
Philosophy	Classic K8s-idiomatic operator	Curated Helm chart	Framework with its own scheduler	Vendor-neutral CNCF blueprint
Own CRDs	Yes (BaseModel, ServingRuntime, InferenceService…)	No (Helm values)	Yes (DynamoCluster)	Yes (KServe-derived + extensions)
Primary runtime	SGLang (first class), also vLLM/TRT-LLM/Triton	vLLM only	TensorRT-LLM (first class), also SGLang/vLLM	vLLM (first class)
PD-disaggregation	Yes, declarative	Yes, with LMCache	Yes, own scheduler	Yes, native
Multi-node TP	Yes, via LWS	Limited	Yes, via Grove	Yes, via LWS and MoE EP
Multi-model in a cluster	Yes, mature multi-tenant	Yes (list of models in values)	Yes	Yes
Multi-LoRA	Yes, first class (FineTunedWeight CRD)	Limited	Yes	On the roadmap
Tiered KV cache	Via LMCache (external integration)	Native LMCache	Own multi-tier	Yes, native
Intelligent routing	Cache-aware via the SGLang router	Prefix-aware / KV-aware / session-based	Own smart routing	Prefix-cache + load-aware
GPU scheduler	kube-scheduler + Kueue	kube-scheduler	Grove (its own)	kube-scheduler + Kueue
Hardware	NVIDIA, AMD ROCm, Intel	NVIDIA, AMD ROCm	NVIDIA only (with emphasis)	NVIDIA, AMD, Intel — neutral
Maturity (mid-2026)	Young, evolving	Stable	Stable, vendor-driven	CNCF Sandbox, evolving fast
Governance	LMSYS (academic-industrial)	vLLM project (academic)	NVIDIA (vendor)	CNCF (neutral)
Learning curve	Medium (4 new CRDs)	Low (familiar Helm values)	Medium-high (Grove + own CRDs)	Medium (similar to extended KServe)

When to choose each one

Choose OME if:

SGLang is your main engine.
You need multi-LoRA serving in production.
The hierarchical abstraction (BaseModel → ServingRuntime → InferenceService) fits you and you come from, or live alongside, KServe.
You have an appetite for a young and very active project.

Choose vLLM Production Stack if:

vLLM is your only engine and you want to align with what the project recommends.
Your team already lives in Helm and does not want to learn new CRDs.
LMCache + advanced routing inside a single Helm chart is exactly what you need.
Your scale is medium (tens of replicas), not extreme.

Choose NVIDIA Dynamo if:

Your infrastructure is NVIDIA-heavy and you want the most optimised path for Hopper/Blackwell.
You already ran Triton for legacy inference and the transition is natural.
You accept vendor lock-in in exchange for direct NVIDIA support.
Your organisation has a dedicated SRE team for inference.

Choose llm-d if:

You want to bet on the future CNCF standard, neutral between vendors.
Your workload includes large MoE models with wide expert parallelism.
You operate multi-cloud or multi-hardware and portability is valuable.
You accept the immaturity of a Sandbox project in exchange for the bet on the future.

Choose pure KServe if:

You already serve non-LLM models and want to unify; LLMs are a minority of your workload.
You need the most conservative and mature use case.
You accept that advanced LLM features (disaggregation, MoE EP, smart routing) are yours to add with external pieces.

Concrete scenarios

Scenario A — Small startup, 1-2 models, 1-3 GPU nodes. You probably do not need an operator. Deployment + Service + HPA with KEDA metrics, as in the vLLM on Kubernetes article. When you grow to 5+ models, reassess.

Scenario B — Mid-sized company, 5-15 models, internal multi-tenant. vLLM Production Stack or OME are the reasonable options. Production Stack if vLLM is all you are going to use; OME if you want runtime flexibility and idiomatic CRDs.

Scenario C — Internal corporate platform or an external service for end customers. llm-d or Dynamo. llm-d if you value vendor neutrality; Dynamo if you live on NVIDIA infrastructure and want the path they recommend.

Scenario D — Mixed cluster of LLMs plus traditional models. KServe as the base, with an LLM operator alongside (OME is the most natural given its conceptual kinship).

Common traps

“I’ll start with pure KServe because it’s mature”. For mid-sized LLMs and up, pure KServe leaves a lot of optimisation on the table. The sensible route is KServe as the base if you live alongside other models, but with an LLM-specific operator next to it.

“I’ll wire everything by hand to understand it”. Reasonable in a PoC, suicidal in production. There are 8 derived resources per model. Multiply by 10 models. You are writing 80 YAMLs and maintaining them. Use an operator.

“I’ll pick the one I like best and pivot if I’m wrong”. Pivoting between operators is not free: although the InferenceService abstraction is converging, the details (how LoRA is modelled, how routing is configured, how metrics are exposed) vary. Migrating from OME to Dynamo is a project of weeks, not days.

“I’ll go with Dynamo because it’s NVIDIA’s and therefore better”. Only if your organisation is already aligned with its operational philosophy (its own scheduler, acceptable vendor lock-in). For many cases, vLLM Production Stack or llm-d give 95% of the value with less friction.

“Helm chart vs operator is a technical decision”. It is a cultural and operational decision. If your team ships via Argo CD with Helm values in Git, Production Stack fits without friction. If your team lives in direct kubectl apply -f and the idea of operators feels natural, OME or llm-d.

What we have not covered

Mooncake: the KV cache system shared between instances that Kimi/Moonshot runs in production across hundreds of millions of queries. It is a primitive (not a complete operator), but it integrates as a cache tier with several of the above.
Ray Serve LLM: Anyscale’s offering, on Kubernetes through KubeRay. More tied to the Ray ecosystem than to native K8s CRDs. Useful if Ray is already part of your infrastructure.
Fireworks AI, Modular MAX: commercial platforms with similar primitives, but hosted. They are not K8s operators; they are competitors at another layer.
Gateway API Inference Extension: the sigwg proposal to extend Gateway API with LLM primitives (model-aware routing, sticky sessions, fairness). In 2026 it is in alpha; the operators above are already starting to support it. When it matures, routing will stop being each operator’s problem and become part of the Kubernetes standard.
A generic inference observability stack: Prometheus + Grafana is standardising around the vllm:* metrics we covered in the vLLM article. There is an OpenTelemetry effort for LLMs (gen-ai semantic conventions) that is probably the next link in the chain.

Closing the series

This series of four articles has walked through LLM inference in production from the bottom up:

KV cache: the working memory that holds up LLM inference — why each token consumes VRAM, and how much.
vLLM on Kubernetes: the LLM inference piece that actually scales — how a model is served in production with a serious Deployment.
PagedAttention from the inside and the state of the art of the KV cache in 2026 — what happens inside the engine at block level, and what has arrived since.
This one — how many models are orchestrated in a cluster.

If you have got this far, you have the vocabulary and the map to sit in a meeting where five people throw acronyms around and place each one correctly. And, more importantly, to start making reasoned decisions about where to begin.

References

Operators and projects covered:

OME — Open Model Engine (GitHub) — LMSYS operator for LLM serving with SGLang/vLLM/TRT-LLM/Triton.
Introducing OME (LMSYS Blog, jul 2025) — announcement and architecture.
vLLM Production Stack (GitHub) — official vLLM Helm chart for K8s.
vLLM Production Stack docs — installation and configuration.
LMCache (GitHub) — KV cache with tiers.
NVIDIA Dynamo — successor to Triton.
NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges (InfoQ, dic 2025) — K8s integration.
llm-d (GitHub) — CNCF Sandbox project.
IBM, Red Hat, and Google donated llm-d to CNCF (The New Stack) — KubeCon EU 2026 announcement.
Red Hat bets big on Kubernetes inference with llm-d (SiliconANGLE, mar 2026) — coverage of the announcement.

Ancestors and primitives:

KServe (site) and KServe joins CNCF (The New Stack).
Kueue — gang scheduling.
LeaderWorkerSet — coordinated workloads such as multi-pod tensor parallel.
KEDA — autoscaling on external metrics.
Gateway API — the successor to Ingress.

Analysis and perspectives:

Building Efficient LLM Inference with the Cloud Native Quartet: KServe, vLLM, llm-d, and WG Serving (Jimmy Song) — an integrating view.
Complete Guide to llm-d CNCF Sandbox (DEV Community) — an operational walkthrough.
Previous articles on this blog: KV cache, vLLM on Kubernetes, PagedAttention deep dive.

vLLM on Kubernetes: the LLM inference piece that actually scales

Mon, 18 May 2026 13:00:00 +0200

TL;DR

vLLM is the inference engine that turns a general-purpose GPU into a productive LLM server. Its value is not in running a model, since any transformers.pipeline does that in three lines of Python, but in squeezing the GPU down to the last gigabyte and the last cycle: PagedAttention for the KV cache, continuous batching to mix requests, its own scheduler to share GPU time between sessions. Kubernetes is its natural habitat because vLLM behaves like a modern UNIX process, with a health endpoint, Prometheus metrics, orderly draining and declarable resources, and K8s already knows how to manage those. But there are traps: the standard HPA does not scale vLLM well, the model takes minutes to load, and naive rolling updates cut sessions halfway through decoding. This article takes the engine apart and then fits it, with real manifests, into a cluster that can actually serve it.

This article is the natural continuation of KV cache: the working memory that holds up LLM inference. There we explained why every token consumes VRAM. Here we look at what you do with that VRAM when you want to offer it as a service.

The analogy: a multiprocess kernel for your GPU

Imagine you have a single processor and need to serve a hundred concurrent processes without any of them blocking the rest. Nobody in their right mind would write a while-true loop dispatching processes one by one: they would install an operating system. The kernel takes care of scheduling, memory paging, isolation, priorities and cleanup on exit. The “process” becomes a convenient abstraction and the kernel does the dirty work.

vLLM is, for your GPU, what the kernel is for your CPU. As far as the GPU is concerned, a conversation with an LLM is a process that lives across many decoding steps, occupies a slice of VRAM (its KV cache) and demands compute time every time a token has to be generated. You have a hundred of those processes at once. You need to:

Share GPU time between them without pausing any of them entirely (it would be disastrous if one long conversation monopolised the GPU).
Manage memory with paging because, just as in RAM, contiguous allocation is inefficient.
Queue new requests when the GPU is saturated and serve them in a reasonable order.
Reclaim resources when a session ends.

PagedAttention is the virtual memory of the KV cache. Continuous batching is the time-slicing scheduler that shares the GPU token by token. The OpenAI-compatible server is the uniform syscall interface. Calling it a “kernel” for the GPU is marketing, but it is marketing that captures the idea well.

What vLLM does inside

Continuous batching: stop waiting for the slowest

The naive inference engine does static batching: it groups N requests, processes them until all of them are finished, returns and starts another round. The problem is obvious: if one request asks for 8 tokens and another asks for 800, the other seven wait for the slow one. GPU utilisation collapses.

Continuous batching (Yu et al., 2022, popularised by vLLM) changes the model. At every decode step, which produces one token for each active session, the engine composes the batch out of the active tokens of ALL sessions alive at that instant. When a session finishes its generation it releases its slot immediately and another request from the queue takes it. The batch never waits for the slowest session because nobody is blocked: everyone advances at one token per step.

The original paper measured 5–23× more throughput than the equivalent static batching. The exact number depends on how variable the response lengths are, but the order of magnitude holds up in practice.

The consequence for the operator is counterintuitive: a single vLLM replica performs like three naive replicas. There is no point adding pods without justifying it with real metrics.

PagedAttention: virtual memory for the KV cache

We already flagged it in the KV cache article: the naive engine reserves one contiguous block per session, sized for the worst case (max_context_len), and wastes 60–80% of the VRAM because real sessions come nowhere near their ceiling.

PagedAttention borrows the solution operating systems have been using for half a century: split the VRAM into small blocks (16 tokens in the default implementation) and keep a logical → physical page table per session. A session with 273 tokens of context occupies 18 blocks (not necessarily contiguous), and grows block by block as it generates. The paper measured <4% waste, an order of magnitude better than contiguous allocation, and that translates into 2–4× more aggregate throughput on the same hardware, because more sessions fit at once.

There is a cost: every attention operation has to be indirected through the page table. But vLLM’s CUDA kernels are written so that this indirection is cheap, and the net result is massively positive.

Prefill vs decode: two phases with opposite profiles

An LLM request has two phases with radically different GPU profiles:

Prefill: processes the entire prompt in one go. It is compute-bound: it uses the tensor cores intensively, the GPU sits above 90%, and it lasts between hundreds of ms and a few seconds depending on prompt size.
Decode: generates token by token. It is memory-bound: the compute is modest but the whole KV cache has to be read for every token, and it lasts from tens of ms per token up to minutes for long responses.

A naive server treats each request as a unit and serves the two phases in series. vLLM decouples them: it mixes requests in prefill with requests in decode in the same step (a technique called chunked prefill when it also splits long prefills into chunks). The result: the GPU is always busy doing something, the tensor cores with prefills, the HBM bandwidth with decodes, instead of oscillating between phases.

Operational implication: the “% GPU utilisation” metric from nvidia-smi is misleading. A GPU at 100% doing prefills may have its HBM bandwidth idle. A GPU at 40% doing decodes may have its HBM saturated. For LLM serving, the useful metric is effective HBM bandwidth, not the compute percentage.

Tensor parallel: when the model does not fit on one GPU

Llama 3 70B in BF16 is ~140 GB. There is no single GPU on the market that can hold it. The solution is tensor parallel: split each model layer by columns and run the partitions on N GPUs in parallel, synchronising with an all-reduce after each layer.

For N=5 GPUs and a 70B model, each GPU sees roughly 28 GB of weights. That sounds fine until you remember that the all-reduce on every layer means reading and writing large tensors between GPUs. If the GPUs share NVLink/NVSwitch (300–900 GB/s), the all-reduce is cheap. If they only share PCIe (~32 GB/s gen4 x16), the all-reduce eats half the time and throughput collapses.

The implication for K8s, which comes next: the scheduler has to guarantee that the N GPUs are physically close. That translates into NodeAffinity to the right product (NVIDIA-H100-80GB-HBM3), a single pod with nvidia.com/gpu: N (not N pods sharing) and, if multi-node is unavoidable, InfiniBand with NCCL as transport.

The OpenAI-compatible server

On top of all of the above, vLLM exposes an HTTP server with endpoints identical to OpenAI’s: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models. It supports Server-Sent Events streaming. It supports tool calling. It supports logprobs.

The value of this is enormous and underrated: any client using the OpenAI SDK works without changes. Your application points at https://vllm.your-cluster.local/v1 instead of https://api.openai.com/v1, and everything else, the LangChain, LlamaIndex, OpenAI Python and OpenAI JS SDKs, just works. It is the main reason vLLM has gained traction over technically comparable alternatives: it is the boring option that works.

Why Kubernetes is the natural habitat

vLLM is a well-behaved process: it starts, exposes metrics, serves a health endpoint, takes SIGTERM with dignity and declares the resources it needs. Kubernetes has spent ten years perfecting the management of processes like that. The only thing K8s took a while to absorb properly was the GPU, and that is now solved.

The GPU as a primitive resource

The plumbing goes like this:

The node has the NVIDIA driver installed (or the GPU Operator installs it).
A DaemonSet, nvidia-device-plugin, registers the physical GPUs as nvidia.com/gpu resources with kubelet.
The Kubernetes scheduler sees those resources the way it sees CPU and memory, puts them in its accounting and assigns them to Pods that ask for them.
The nvidia-container-toolkit makes sure containerd injects the right devices into the container at startup.

For the pod, asking for a GPU looks like this:

resources:
 requests:
 nvidia.com/gpu: 1
 limits:
 nvidia.com/gpu: 1

Without MIG, MPS or time-slicing configured, a GPU is not shared between pods: you ask for it whole or you do not ask for it. For vLLM, which wants the entire GPU to itself, that is exactly what you want.

The vLLM Pod lifecycle

Differences from a typical webapp Pod:

Long startup. Loading 16 GB of weights into VRAM over the network takes 30 seconds in the best case and 5 minutes in the worst. A readinessProbe with initialDelaySeconds: 30 and failureThreshold: 3 kills the pod before it starts. Solution: a startupProbe with a high threshold before the livenessProbe starts evaluating.
Warm-up matters. The first prefill compiles CUDA kernels specific to the input shape. The first 2–3 requests are noticeably slower. If latency matters from second one, it is worth firing a warm-up POST after ready.
Draining is not instantaneous. SIGTERM must not kill in-flight sessions. vLLM, configured with --disable-graceful-shutdown false (the default), finishes active requests before closing. That can take 30–180 seconds. terminationGracePeriodSeconds has to accommodate it.
Hostile rollouts. A naive rolling update (maxUnavailable: 1) can leave you with no replicas serving if the new one is slow to load. Set maxSurge: 1, maxUnavailable: 0 so the new pod is Ready before the old one is drained.

Anatomy of a serious deployment

First of all: GPU Operator

Without the GPU Operator (or an equivalent manual installation), a Pod with nvidia.com/gpu: 1 stays Pending forever. What the operator installs as DaemonSets on every GPU node:

nvidia-driver-daemonset — the kernel-mode driver (if you do not have it installed at host level).
nvidia-device-plugin-daemonset — registers the GPUs as a kubelet resource.
nvidia-container-toolkit-daemonset — the containerd integration.
nvidia-dcgm-exporter — Prometheus metrics for the GPU (utilisation, temperature, ECC errors, memory).
gpu-feature-discovery — node labels: nvidia.com/gpu.product, nvidia.com/gpu.memory, and so on, essential for NodeAffinity.

The recommended installation is the official Helm chart. The delicate part is aligning the driver with the host kernel version: if the nodes run kernel 6.x, the operator needs a compatible driver branch.

A complete, annotated vLLM Deployment

The following deploys Llama 3 8B with an FP8-quantised KV cache, up to 32K of context, on an RTX 4090. It is the reference manifest; the comments explain the non-obvious decisions.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 replicas: 1
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 0 # never run out of replicas during the rollout
 selector:
 matchLabels:
 app: vllm-llama3-8b
 template:
 metadata:
 labels:
 app: vllm-llama3-8b
 annotations:
 prometheus.io/scrape: "true"
 prometheus.io/port: "8000"
 prometheus.io/path: "/metrics"
 spec:
 # Only nodes with the GPU we expect
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-GeForce-RTX-4090
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 # Pre-download the weights if they are not on the shared PVC
 initContainers:
 - name: model-download
 image: ghcr.io/huggingface/huggingface-cli:latest
 command: ["sh", "-c"]
 args:
 - |
 if [ ! -f /models/llama-3-8b/config.json ]; then
 huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct \
 --local-dir /models/llama-3-8b --local-dir-use-symlinks False
 fi
 env:
 - name: HF_TOKEN
 valueFrom:
 secretKeyRef:
 name: huggingface
 key: token
 volumeMounts:
 - name: models
 mountPath: /models
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-8b
 - --served-model-name=llama-3-8b
 - --tensor-parallel-size=1
 - --max-model-len=32768
 - --kv-cache-dtype=fp8
 - --enable-chunked-prefill
 - --enable-prefix-caching
 - --gpu-memory-utilization=0.92
 - --port=8000
 ports:
 - name: http
 containerPort: 8000
 - name: metrics
 containerPort: 8000 # same port as http; /metrics
 resources:
 requests:
 cpu: "4"
 memory: 8Gi
 nvidia.com/gpu: 1
 limits:
 cpu: "8"
 memory: 16Gi
 nvidia.com/gpu: 1
 startupProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 10
 failureThreshold: 60 # 10 min of grace to load the model
 readinessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 5
 livenessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 20
 failureThreshold: 3
 volumeMounts:
 - name: models
 mountPath: /models
 readOnly: true # no process should write here at runtime
 - name: shm
 mountPath: /dev/shm  # vLLM uses shared memory for IPC between workers
 volumes:
 - name: models
 persistentVolumeClaim:
 claimName: model-cache
 - name: shm
 emptyDir:
 medium: Memory
 sizeLimit: 4Gi
 terminationGracePeriodSeconds: 120 # accommodates draining of active sessions
---
apiVersion: v1
kind: Service
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 selector:
 app: vllm-llama3-8b
 ports:
 - name: http
 port: 80
 targetPort: 8000

Five things that are not visible on a first read:

/dev/shm in memory, 4 GB. vLLM launches worker processes (one per GPU in tensor parallel, plus the driver) that communicate over shared memory. Docker’s default (64 MB) blows up as soon as the model is mid-sized. Without this, the pod starts but fails the moment it serves the first complex request.
--enable-prefix-caching. If the prompts in your workload share structure (a common system prompt, few-shot examples), vLLM reuses the KV cache of the shared part. A free 30–60% gain in TTFT.
--gpu-memory-utilization=0.92. vLLM reserves the given percentage of the VRAM for itself. The remaining 8% leaves room for activations, CUDA kernels and the overhead that does not get counted. Lowering it buys safety; raising it above 0.95 invites OOM.
A ReadOnlyMany PVC is ideal. The model does not change at runtime. Several pods can mount the same PVC without contention.
No livenessProbe shorter than the terminationGracePeriodSeconds. If a drain takes 90s and liveness kills at 60s, rollouts lose sessions.

Multi-pod tensor parallel: LeaderWorkerSet

When the model needs more GPUs than a single node has, the pattern is a group of coordinated pods, one per GPU, that behave as a single replica. For years this was modelled with a StatefulSet plus init scripts; since Kubernetes 1.32, the idiomatic primitive is LeaderWorkerSet (LWS):

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
 name: vllm-llama3-70b
 namespace: inference
spec:
 replicas: 1
 leaderWorkerTemplate:
 size: 5 # 1 leader + 4 workers = 5 pods, 5 GPUs
 restartPolicy: RecreateGroupOnPodRestart
 leaderTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-leader
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-70b
 - --tensor-parallel-size=5
 - --distributed-executor-backend=ray
 # ...
 workerTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-worker
 image: vllm/vllm-openai:v0.6.3
 # the workers join the leader's Ray cluster

LWS guarantees startup order (workers first, leader afterwards) and an atomic lifecycle (if a worker goes down, the whole group restarts, not a single pod). Without it, the coordination is manually fragile.

A simpler alternative, if all the tensor parallel GPUs fit on a single node (the case with HGX H100 boxes with 8 GPUs and an internal NVSwitch): one single Pod with nvidia.com/gpu: 5, --tensor-parallel-size=5, and vLLM handles everything internally. No Ray, no LWS, far simpler. It is the recommended path whenever it is available.

Autoscaling: the standard HPA is no use

An HPA on CPU% is useless for vLLM. The GPU does the work; the pod’s CPU sits at 5–10% even under maximum load. Nor is the GPU utilisation percentage from dcgm-exporter any use: a pod at 100% GPU% with gpu_cache_usage_perc=15% is serving one long session without being saturated, while a pod at 60% GPU% with gpu_cache_usage_perc=95% is on the edge of evicting sessions.

The right metrics are exported by vLLM itself at /metrics (Prometheus format):

Metric	What it says	When to scale
`vllm:num_requests_waiting`	Requests queued and not yet in the batch.	If it stays above 5–10.
`vllm:num_requests_running`	Active requests in the batch.	For capacity planning, not for scaling.
`vllm:gpu_cache_usage_perc`	% of the KV cache occupied.	If sustained above 80%, there is a risk of preemption.
`vllm:time_to_first_token_seconds`	Prefill latency (histogram).	If p95 exceeds your SLA.
`vllm:e2e_request_latency_seconds`	Total latency per request.	An output metric.

For the HPA to consume them there are two routes: Prometheus Adapter (exposes custom metrics to the K8s API) or KEDA (scales on Prometheus queries directly, far more convenient). With KEDA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama3-8b
 minReplicaCount: 1
 maxReplicaCount: 8
 pollingInterval: 10
 cooldownPeriod: 120 # 2 min before scale-down (long sessions)
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.monitoring:9090
 threshold: '5'
 query: |
 sum(vllm:num_requests_waiting{app="vllm-llama3-8b"})

The long cooldownPeriod matters: if you drop replicas while there are sessions decoding, you kill them. Better to leave 2 minutes of slack.

Observability: the four metrics that matter

Of everything /metrics exports, a minimal dashboard needs these four:

TTFT p50/p95 (time to first token) — what the user perceives when they hit send.
TPOT p50/p95 (time per output token) — the “speed” of the streaming.
Aggregate throughput (tokens generated per second across the cluster) — for capacity planning.
Queue depth (vllm:num_requests_waiting) — the leading indicator: if it grows, everything is about to degrade.

To that, add HBM utilisation and free memory per GPU (from dcgm-exporter) to spot bandwidth saturation and fragmentation problems. A decent Grafana dashboard with those 6 charts anticipates 90% of incidents.

Two concrete scenarios

We reuse the same hardware as the previous article for continuity. The same cache arithmetic, now with the engine on top.

Scenario A — 1×RTX 4090 (workstation or small K8s node)

Topology: 1 Pod, --tensor-parallel-size=1, 1 GPU, 1 node.
Model: up to 8B BF16 (Llama 3 8B, Qwen3 8B, Mistral 7B) or up to 14B in FP8/AWQ.
PVC: the node’s local SSD. The 4090 reads 1 TB/s from HBM; an NVMe SSD at 5 GB/s takes 5 seconds to feed 25 GB of weights into VRAM, negligible next to initialisation.
HPA: irrelevant inside the 4090 (always 1 vLLM replica per GPU), but useful across nodes: 3 replicas on 3 nodes with a 4090 each, and the K8s Service spreads round-robin.
Useful concurrency: 4–8 simultaneous sessions with 8K of context, 1–2 with 32K.
Natural use case: PoC, small teams, departmental environments, edge.

The manifest above is sized for this scenario. Changing only the model and the args, the same Deployment serves Qwen, Mistral or whatever is next.

Scenario B — 5×H100 SXM (cluster with NVLink/NVSwitch)

Topology: 1 Pod with nvidia.com/gpu: 5 on an HGX node, --tensor-parallel-size=5. If the platform does not allow grouping 5 GPUs into a single Pod, a LeaderWorkerSet with 5 pods coordinated by Ray.
Model: up to 70B BF16 (Llama 3 70B) or up to 200B+ in FP8 with cache quantisation.
PVC: NVMe attached directly to the node, or fast network storage (Ceph with a 25/100 GbE network, Lustre, GPFS). Loading 140 GB of weights over a slow network takes 5 minutes per start.
HPA: irrelevant inside the 5-GPU cluster (the 5 are an indivisible unit), but useful for adding whole HGX nodes when the load passes a given threshold. This combines with Cluster Autoscaler if the underlying infrastructure allows it.
Useful concurrency: 32–128 simultaneous sessions with mid-sized contexts, 4–16 with huge contexts.
Natural use case: internal corporate service, public exposure with an SLA, multi-tenant.

A and B, side by side

Aspect	A (1×4090)	B (5×H100 SXM)
Pod topology	1 pod, 1 GPU	1 pod with 5 GPUs (or an LWS of 5)
Maximum BF16 model	8 B	70 B
TTFT @ 8K context, idle	~250 ms	~80 ms
TPOT, idle	~30 ms/tok	~15 ms/tok
Throughput @ concurrency 16	~50 tok/s/session	~200 tok/s/session
Session drain	30–60 s	60–180 s
Useful autoscaling	Replicas on peer nodes	Whole nodes via Cluster Autoscaler
Reasonable multi-tenancy	Limited: 4–8 sessions	Comfortable: 32–128 sessions
Indicative cost (hardware)	~2 K €	~250 K € (≈ 125×)

The asymmetry is the same one as in the previous article: 125× more expensive, only ~4× more throughput per session and ~10× more concurrency. What the cluster buys is not proportional; it buys access to models an order of magnitude larger and latencies low enough for interactive use at scale. If your workload is batch or asynchronous agents where latency is not critical, several 4090s come surprisingly close.

vLLM against TensorRT-LLM and SGLang

Honestly, all three are good engines. The choice depends on practical criteria, not technical ones. A decision map, not a benchmark:

Criterion	vLLM	TensorRT-LLM	SGLang
Supported hardware	NVIDIA, AMD ROCm, Intel Gaudi	NVIDIA only	NVIDIA, AMD ROCm
Pure latency (TTFT)	Good	Best: kernels compiled for the exact hardware	Good
Aggregate throughput	Excellent	Excellent	Excellent (RadixAttention)
Deployment	Trivial: Docker image + args	Complex: build an engine per model and per GPU	Moderate
OpenAI-compatible API	Native, complete	Yes, through Triton Inference Server	Yes
Support for new models	Days after release	Weeks (recompile the engine)	Days
Quantization	AWQ, GPTQ, FP8 cache	INT4/INT8/FP8, very mature	AWQ, FP8
Multi-modal	Yes (Llava, Pixtral, Qwen-VL)	Yes	Excellent, a priority
Function calling / tool use	Good	Limited	First class
Community / release cadence	Very active, weekly	Active, NVIDIA-driven	Very active, academic
Licence	Apache 2.0	Apache 2.0	Apache 2.0

When to choose each one:

vLLM: the “boring choice” that works. The path of least friction to production. If your team has no dedicated inference serving specialist, this one. It supports varied hardware, up-to-date models, a stable API and a huge community.
TensorRT-LLM: when per-request latency is the only metric that matters and your model is stable (trained in-house, not swapped every fortnight). The price of the performance is that every model plus every GPU plus every TRT version requires an engine rebuild, and that blocks fast iteration.
SGLang: for agent-dominated workloads (heavy tool calling) or complex multi-modal. Its RadixAttention, structural caching of prompts with shared prefixes, shines in ReAct-style patterns where the same system prompt is repeated thousands of times.

For most teams starting out with on-prem LLM serving, vLLM is the right answer until you have production data pushing you elsewhere.

Common operational traps

A list of gotchas that come up again and again:

The model is downloaded on every rolling update

Symptom: every deploy takes 5+ minutes to become available. Cause: there is no shared PVC. Every new pod downloads the model from Hugging Face from scratch. Fix: a ReadOnlyMany PVC on fast storage, or a local mirror of the registry (a Pod with huggingface-cli serving a directory over HTTP). In CI/CD, hydrating the PVC before the rollout is 1 line of bash.

Short readiness timeouts that kill pods while they load

Symptom: new pods go into CrashLoopBackOff during the first model load. Cause: a readinessProbe with too low a timeout fires before vLLM finishes loading; the livenessProbe finishes the job. Fix: a startupProbe with failureThreshold: 60 or more (10 minutes of grace) before liveness starts evaluating.

An unquantised KV cache, then OOM

Symptom: the pod starts fine, serves for five minutes, then gets OOMKilled when session number five arrives with a long context. Cause: a BF16 KV cache (the default) consumes twice what FP8 does. Fix: --kv-cache-dtype=fp8. Negligible quality loss in the vast majority of cases, double the capacity.

Confusing replicas with concurrency

Symptom: the HPA scales to 8 replicas under light real load and the cloud bill goes up. Latency does not improve. Cause: someone configured targetAverageUtilization: 50% on CPU, thinking that is “load”. The reality: a single vLLM replica serves dozens of simultaneous sessions. Fix: an HPA on vllm:num_requests_waiting. If the queue is empty, one replica is enough even if the GPU is at 90%.

Tensor parallel on GPUs without NVLink

Symptom: throughput 3× worse than expected, GPUs at 30%, a lot of PCIe traffic. Cause: tensor_parallel=4 on 4 GPUs connected only by PCIe; the all-reduce saturates the bus on every layer. Fix: either the GPUs share NVLink/NVSwitch (SXM/HGX models), or you use pipeline parallel (worse latency but less all-reduce), or you reduce TP and accept that the whole model does not fit.

Sessions cut off during a rolling update

Symptom: users see truncated responses during the deploy. Cause: terminationGracePeriodSeconds: 30 (the default) is not enough to drain long generations. Fix: terminationGracePeriodSeconds: 120–180. Combined with maxUnavailable: 0, rollouts are invisible to active users.

What we have not covered (upcoming articles)

vLLM with hot LoRA adapters: serving a base model plus N tenant-specific adapters without reloading weights.
Disaggregated serving: separating prefill and decode into specialised pods, each optimised for its GPU profile.
Quantization deep-dive: AWQ vs GPTQ vs dynamic FP8 vs FP4, the real trade-offs, and when to use each.
Gateway API + AI Inference Extensions: the sigwg proposal to make LLMs first-class citizens in K8s (routing by model, sticky sessions per conversation, multi-tenant fairness).
Multi-modal serving: the same runtime, a different kind of request — images, audio, embeddings.

References

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — the original vLLM paper.
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — the paper that popularised continuous batching.
Official vLLM documentation — operational and well maintained.
NVIDIA GPU Operator — installation and troubleshooting of the GPU layer on Kubernetes.
LeaderWorkerSet — the primitive for coordinated workloads such as multi-pod tensor parallel.
KEDA — event-driven autoscaling, ideal for scaling on queue metrics.
TensorRT-LLM and SGLang — the two most serious comparables.
LMSYS Chatbot Arena — periodic benchmarks comparing the three engines.
Previous article on this blog: KV cache: the working memory that holds up LLM inference.

Llm-Inference on lo0 — Tech Blog

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

TL;DR

The analogy: the station master and the track gauge

A Deployment serves tokens, but offers no contract

KServe: from KFServing to a CNCF incubating project

Anatomy of the control plane

InferenceService: predictor, transformer, explainer

ServingRuntime and ClusterServingRuntime: the catalogue of trains

The storage initializer and storageUri

The deployment modes

The LLM branch: LLMInferenceService

The Open Inference Protocol: the contract itself

The routes and the payload

Who actually implements it

The generate extension and why it did not win

The tension, unvarnished

Four manifests you can actually apply

Autoscaling and the problem of zero

Operational traps

Decision map: when KServe, and when it is overkill

For an inference factory

See also

Sources

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

TL;DR

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

The four layers of a multi-tenant inference platform

Layer 1 — AI Gateway: the single front door

Why centralise

The three dominant options in 2026

When to choose each one

The minimum the gateway has to do

Example of a multi-tenant LiteLLM configuration

Layer 2 — Policy & Quota Plane: what each tenant can do

Technical quotas

Economic budgets

Model whitelist and blacklist

Priority classes

Admission control

The typical pattern in 2026

Layer 3 — Isolation Plane: isolating workloads physically

Three NVIDIA mechanisms for sharing a GPU

The choice for multi-tenant in 2026

MIG partitions on the H100

Approach A — A shared large model with quotas at the gateway

Approach B — Dedicating GPUs per model / tenant

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

The operational choice: start with A, move up to C if you need to

Isolation at the Kubernetes level

Layer 4 — Observability Plane: seeing what happens per tenant

The four mandatory properties

Showback vs chargeback

Tools

The minimum multi-tenant dashboard

Sizing on mid-scale GPU clusters: concrete decisions

Default decisions

An example tenant scheme

When to add hardware

Common operational traps

A gateway without auth: a backdoor into the cluster

MIG and NVLink are incompatible

Quotas pinned to the cluster’s ceiling

No multi-tenant observability from day 1

Showback that never reaches chargeback

Non-whitelisted models eating the budget

Badly calibrated priority classes

No failover from the gateway

An operational roadmap for getting started

What we have not covered (upcoming posts)

References

LLM inference operators on Kubernetes: OME, vLLM Production Stack, NVIDIA Dynamo and llm-d

TL;DR

The analogy: from init.d to systemd to operators

Why an operator, and not just a Deployment

KServe: the common ancestor

OME (Open Model Engine)

The CRD hierarchy

The four deployment modes

A `Deployment` serves tokens, but offers no contract

`InferenceService`: predictor, transformer, explainer

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

The storage initializer and `storageUri`

The LLM branch: `LLMInferenceService`

The `generate` extension and why it did not win

The analogy: from `init.d` to systemd to operators