Cncf on lo0 — Tech Blog

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

Sun, 26 Jul 2026 20:00:00 +0200

This blog has devoted a good part of its 145 articles to what happens inside the inference process and another good part to what happens around it. What is missing is a thread that cuts across all of that and answers a question that takes thirty seconds to ask in an audit and does not get answered in three weeks in production: when the endpoint returns a token, where did that model come from, who signed it, and who proves that the binary running on the GPU is the one that was approved?

This series walks that chain backwards from the only visible point, the endpoint: the control plane and the API contract (this article), the registry and the distribution of the bytes (2/4), the signing and provenance that let you trust them (3/4), and the identity and isolation of whoever runs them (4/4). All of it with projects under the governance of CNCF, LF AI & Data or the Linux Foundation: a chain of trust that depends on a proprietary product is no better than a support contract.

TL;DR

A vLLM Deployment serves tokens perfectly well. What it does not give you is a contract: no model versioning, no rollout by percentage, no scale-to-zero, and no common surface for the thirty non-LLM artifacts of an inference factory.
KServe provides the vocabulary: InferenceService, ServingRuntime/ClusterServingRuntime, storage initializer and storageUri. It has been CNCF incubating since 29 September 2025, after seven years as KFServing and a spell in LF AI & Data. The cadence has accelerated: v0.18.0 on 29 April 2026 and v0.19.0 on 14 June 2026.
The LLM branch is recent and different: the LLMInferenceService CRD (serving.kserve.io/v1alpha1) arrived in v0.16 in November 2025, on top of llm-d, with spec.prefill, spec.worker, spec.parallelism and spec.router.
The Open Inference Protocol (V2) is the contract: /v2/health/*, /v2/models/<m>, /v2/models/<m>/infer, in REST and gRPC, implemented by Triton, OpenVINO Model Server, Seldon MLServer and AMD Inference Server. It is tensor-centric, and the LLM world standardised de facto on the OpenAI API: KServe serves both, with the OpenAI ones under the openai/ prefix.
The honest position: the portability of an LLM today comes from the OpenAI API, not from V2. Accepting the dual track is correct; pretending there is a single standard is not.
KServe is overkill if you serve one model on one node. It starts to pay off from the fifth heterogeneous artifact onwards, and it competes on a different axis from the LLM operators: contract and lifecycle versus engine and topology.

The analogy: the station master and the track gauge

An inference factory is like a railway station with several lines.

The station master does not drive any train. His job is the service timetable: which formation leaves from which platform, how many units are coupled each hour, when an engine is pulled out for maintenance and, when new rolling stock arrives, how to divert ten per cent of the passengers to the test train before committing the whole service. That is the control plane. KServe does not generate a single token, but it decides what runs, with what configuration, with how many replicas and with what traffic split. The trains are the engines (vLLM, SGLang, TensorRT-LLM, Triton, TEI), and a competent station master does not marry a manufacturer.

And then there is the track gauge: as long as everybody builds to the same one, any locomotive rolls on any line and changing supplier is a commercial decision, not a civil engineering project. That gauge is the Open Inference Protocol, the contract that makes a client written against Triton work against OpenVINO Model Server without touching a line.

The analogy has a twist that needs no explanation in Spain: the agreed gauge is not the gauge the new rolling stock runs on. The conventional network was built to Iberian gauge and the high-speed one to standard gauge, and the practical solution was not to convert the network but to install gauge changers at the boundary points. The same thing has happened in inference: V2 is the agreed gauge and the OpenAI API is the gauge everything generative runs on. What sustains interoperability is the changer, not a single standard. We will come back to it.

A `Deployment` serves tokens, but offers no contract

In vLLM on Kubernetes we built a service with Deployment, Service, HPA and an initContainer that pulls the weights down. It works, and for one model on one node it is the right answer. What it does not give you, and cannot give you without reinventing it every time:

A stable API contract. The endpoint is “whatever the image you put there exposes”: moving from vLLM to TensorRT-LLM changes routes, error format and metric names, and the clients find out in production.

Versioning with semantics. The model is a path inside a volume; knowing which version a pod is serving requires kubectl exec. There is no declarative field that says “this service serves artifact X” and can be audited from outside.

Progressive model rollout. A RollingUpdate replaces pods; it does not split traffic between two versions by percentage. For that you need an L7 gateway in front, like the one in canary, blue-green and shadow, or a control plane that emits it.

Scale-to-zero. The HPA does not go below one replica: for a model queried twelve times a day, that is an 80 GB GPU tied up twenty-four hours.

And, above all, a common surface for what is not an LLM. A real factory does not serve three LLMs: it serves two or three LLMs, four embedding models, a couple of rerankers, ticket classifiers, a PII detector, an OCR, industrial vision and three scikit-learn artifacts that nobody remembers training. Fifteen or thirty artifacts, each with its own runtime and its own way of saying “I am ready”. The cost is not in serving the first one: it is in making sure the thirtieth does not cost thirty times what the first one did.

KServe: from KFServing to a CNCF incubating project

KServe was born in 2019 as KFServing, inside Kubeflow, in a collaboration between Google, IBM, Bloomberg, NVIDIA and Seldon. In February 2022 it was donated to the LF AI & Data Foundation; in September 2022 it split from Kubeflow and was renamed. The relevant jump: on 29 September 2025 the CNCF accepted it directly at incubating level, without going through sandbox, with a public announcement on 11 November. The figures it declared then, more than 300 contributors, 19 maintainers, more than 30 adopting organisations (Bloomberg, Red Hat, Cloudera, Nutanix, SAP, NVIDIA), come from an interested party, but the list is verifiable and the 625 contributing organisations on the CNCF dashboard support the thesis of governance not captured by a vendor.

On cadence you have to be precise, because the cliché of “KServe is slow” no longer describes reality:

Version	Date	What it brought
v0.15.0	31 March 2025	`LocalModelCache`, multi-node inference, OpenAI-compatible embeddings API, Gateway API in raw mode, KEDA integration
v0.16.0	November 2025	`LLMInferenceService`; rename of `RawDeployment` to `Standard` and of `Serverless` to `KNative`; progressive rollout in raw mode
v0.18.0	29 April 2026	WVA + KEDA/HPA autoscaling for LLMISvc, `/v1/responses` route (OpenAI Responses API), `restricted` Pod Security Standards by default on LLMISvc, namespace-scoped `ModelCache`
v0.19.0	14 June 2026	Routing by model name, static LoRA adapters, balancing across heterogeneous GPUs, dual REST/gRPC routing in Standard mode, automatic migration to `InferencePool` v1

Two major releases in under two months, with the bulk of the changelog in the LLM branch. The project is not slow: it has pivoted from classic predictive serving to generative serving. The consequences run in both directions.

The negative one is called ModelMesh. It was the high-density mode (hundreds of small models sharing processes, with dynamic load and unload) and it was, on paper, the natural answer to the problem of thirty heterogeneous artifacts. Its last release was v0.12.0 on 9 July 2023, and the kserve/modelmesh-serving repository was archived read-only on 14 April 2026. Red Hat published specific guidance for migrating to Standard mode. If somebody proposes ModelMesh in a design today, it is a sign that the documentation they read is three years old.

Anatomy of the control plane

`InferenceService`: predictor, transformer, explainer

The core CRD is InferenceService (serving.kserve.io/v1beta1), and its contribution, the one that became the common vocabulary of the field, is to break an inference service down into three components each with its own contract:

predictor: mandatory. It declares modelFormat, optionally runtime, the storageUri the weights come from and the protocolVersion it exposes.
transformer: optional. Pre- and post-processing. That it is a separate component, rather than code inside the server, is what allows it to be scaled separately (it is usually CPU-bound while the predictor is GPU-bound) and versioned separately.
explainer: optional, and the one that has aged worst: the V2 protocol does not support the explain endpoint, which only exists in V1. Anyone who needs explainability with a standard contract has a real gap here.

All three share ComponentExtensionSpec: minReplicas, maxReplicas, scaleTarget, scaleMetric, containerConcurrency, timeout and canaryTrafficPercent.

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

A ServingRuntime is a pod template that knows how to serve one or several formats. It declares supportedModelFormats (with name, version, autoSelect and priority), protocolVersions (v1, v2) and containers, with args parameterised by templates in the {{.Name}} style that are substituted with metadata from the InferenceService.

Selection is what makes this scale. If the InferenceService names a runtime, the controller looks for it first in the namespace and then at cluster level; if it does not name one, it picks among those with autoSelect: true for that modelFormat, breaking ties by priority. That is what lets a platform publish a curated catalogue (“here, huggingface is served with this vLLM and these flags”) without the product teams having to know what --gpu-memory-utilization is. In multi-tenant setups, the ClusterServingRuntimes belong to the platform and the InferenceServices to the teams: the cleanest separation-of-duties line KServe offers.

The storage initializer and `storageUri`

The storageUri field triggers the storage initializer: an init container that downloads the artifact to /mnt/models before the server starts. Schemes supported as of July 2026: s3://, gs://, https://, pvc://, hf://, oci:// and git repositories; credentials are attached via a ServiceAccount with annotated secrets. That oci:// connects with the second article of the series: the weights can live in the same registry as the images, with the same access control and the same signing capability.

The deployment modes

This is where nearly everything operational is decided. It is selected with the serving.kserve.io/deploymentMode annotation.

	KNative (formerly Serverless)	Standard (formerly RawDeployment)	ModelMesh
Resources emitted	Knative `Service` + `Revision`, `VirtualService` with Istio	`Deployment`, `Service`, `HPA`, Gateway API `HTTPRoute`	Its own high-density runtime
Dependencies	Knative Serving + Istio (or an alternative gateway)	Kubernetes 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1 and a controller	Its own components
Autoscaling	KPA (concurrency / RPS)	HPA or KEDA	—
Scale-to-zero	Yes, with an activator that holds the request	No for HTTP requests	—
`canaryTrafficPercent`	Yes, by revision weights in Istio	Silently ignored (issue #5335, open since 2 April 2026)	—
Sidecars in the pod	`queue-proxy` (+ `istio-proxy`)	None	—
Status as of July 2026	Current	Current and the default mode of the LLM branch	Archived on 14 April 2026

That table has an uncomfortable reading: the two most quoted KServe capabilities, scale-to-zero and canary by percentage, live in the mode that drags Knative and Istio along, while the mode the project has pivoted towards is the other one.

The LLM branch: `LLMInferenceService`

Since v0.16 KServe has a separate CRD for LLMs: LLMInferenceService, in serving.kserve.io/v1alpha1. It is not an extension of the classic InferenceService and does not share its API version, and confusing the two is the first mistake made by anyone arriving from old documentation.

It is built on llm-d, the project that Red Hat, Google, IBM, CoreWeave and NVIDIA donated to the CNCF. The division of roles the project itself declares is clean: KServe is the control plane (lifecycle, scaling and operational governance) and llm-d contributes distributed scheduling, with GPU utilisation, queue depth, cache residency and SLA. In the analogy: llm-d decides which line each train takes; KServe decides which trains exist and when they are withdrawn.

The fields that matter: spec.model (URI, invocation name, scheduling criticality and LoRA adapters, with static adapter reconciliation from v0.19, which connects with multi-LoRA serving); spec.template (single node, or the decode pool); spec.worker (multi-node, which triggers LeaderWorkerSet); spec.prefill (the disaggregation we analysed in disaggregated serving turned into a declarative field); spec.parallelism (tensor, data and expert parallelism); and spec.router (gateway, route, scheduler), where KServe leans on the Gateway API Inference Extension, v1.5.0 of 19 April 2026, already GA, with automatic migration to InferencePool v1 from v0.19.

The endpoint it exposes is OpenAI-compatible: /v1/chat/completions with streaming, and since v0.18 also /v1/responses. That detail underpins the next section: the LLM branch of KServe does not speak V2, it speaks OpenAI.

For multi-node there is the kserve-huggingfaceserver-multinode runtime, based on vLLM over Ray, with workerSpec.tensorParallelSize and workerSpec.pipelineParallelSize. Its restrictions are hard: it requires a ReadWriteMany PVC, it only works in Standard mode, it does not support autoscaling and it requires exactly one head pod. A reference 4×H100 SXM node with NVLink needs none of this, because a 70B fits with tensor parallel 4 on a single node; you need it from there upwards.

The Open Inference Protocol: the contract itself

The routes and the payload

The Open Inference Protocol (historically the “V2 Inference Protocol”) lives in kserve/open-inference-protocol, with REST and gRPC specifications and versioning under SemVer 2.0. Its HTTP surface is deliberately small:

Resource	Verb	Route
Server metadata	GET	`/v2`
Liveness	GET	`/v2/health/live`
Server readiness	GET	`/v2/health/ready`
Model metadata	GET	`/v2/models/<model>[/versions/<v>]`
Model readiness	GET	`/v2/models/<model>[/versions/<v>]/ready`
Inference	POST	`/v2/models/<model>[/versions/<v>]/infer`

The payload is tensorial: id, a list of inputs (each with name, shape, datatype and data) and optionally the desired outputs. The types are BOOL, signed and unsigned integers from 8 to 64 bits, 16-, 32- and 64-bit floating point, and BYTES. In gRPC the same contract is GRPCInferenceService, with six RPCs: ServerLive, ServerReady, ModelReady, ServerMetadata, ModelMetadata and ModelInfer. For high-volume embeddings, the serialisation difference is not cosmetic.

Who actually implements it

The specification repository declares KServe, NVIDIA Triton, Seldon MLServer and Core v2, OpenVINO Model Server, AMD Inference Server and TorchServe as adopters. Not all entries are worth the same as of July 2026:

Triton implements it in a documented way (“Triton exposes HTTP/REST and gRPC endpoints based on the standard inference protocols proposed by the KServe project”), plus its own extensions. It is the de facto reference implementation.
OpenVINO Model Server exposes the KServe API over gRPC and REST, the TensorFlow Serving one and OpenAI-compatible endpoints: the clearest example of a server that speaks both track gauges.
TorchServe forces you to be honest. Its official documentation carries the warning: “This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches.” An adopter on zero maintenance is not an adopter: it is a row nobody has updated.

On the health of the standard, the most eloquent data point is not in any press release: the specification repository does not have a single published release, has accumulated something like thirty commits and seventy-five stars, and its governance consists of monthly meetings and a Slack channel. There is also a “vendor neutral” fork at open-inference/open-inference-protocol with more commits than the original and no traction whatsoever. V2 is a stable contract, widely implemented and barely evolving. For a serialisation format that can be a virtue; for a standard aspiring to cover the generative world, it is a death sentence.

The `generate` extension and why it did not win

The standard did try to cover generation: the specification contains a generate_rest.yaml with two routes, /v2/models/<m>/versions/<v>/generate and .../generate_stream, and a minimalist schema: a request with text_input and parameters (temperature, top_p, max_tokens defaulting to 20, stop, details); a response with text_output, model_name, model_version and, if requested, finish_reason and logprobs; streaming over server-sent events. It is clean, it is enough for a simple completion and nobody uses it: the KServe generative inference documentation, as of July 2026, lists four endpoints, and none of them is generate:

Task	Endpoint in KServe
Text generation	`openai/v1/completions`
Chat	`openai/v1/chat/completions`
Embeddings	`openai/v1/embeddings`
Reranking	`openai/v1/rerank`

The openai/ prefix exists, according to the documentation itself, “to avoid confusion with the V1 API protocol”, and it is changed with KSERVE_OPENAI_ROUTE_PREFIX. It is plumbing that says a lot: the in-house standard was relegated to a defensive prefix while the outside one takes all the traffic.

The tension, unvarnished

V2 solved the problem of 2021, and it solved it well. A tensorial, typed contract with metadata and health checks, in REST and gRPC, is what you need to serve a classifier, a vision model, a ranker or an embedding, and it is still the right answer for that half of the catalogue: swapping an embedding server from MLServer to OpenVINO Model Server without touching clients is the track gauge doing its job.

The LLM world standardised somewhere else, not by committee, but because the entire client ecosystem (SDKs, agent frameworks, gateways, evaluation) was written against the OpenAI API, and any engine that wanted to be used had to speak it. What that means for an architecture:

There is not one contract: there are two, and both are legitimate. Designing as if there were one does violence to half the catalogue.
LLM portability is superficial. It covers chat and completion well; as soon as you use detailed logprobs, guided decoding, priority headers, prefix caching parameters or tool calling with specific schemas, you leave the common subset and get tied back to the engine. It is the warning from structured output: what is portable is the envelope, not always the contents.
The gauge changer is the gateway. What makes the set switchable is the translation and routing layer of choosing an OSS LLM inference gateway and the router as an L7 gateway, not a single standard.
Crossing over loses the versioning. V2 has /versions/<v> in the route; the OpenAI API only has the model field. The version has to be encoded in the name (asistente-70b-2026-07) or managed in the control plane. That is the difference between being able to audit which version answered and not being able to.

Four manifests you can actually apply

Platform runtime for the blog’s reference node, 4×H100 SXM 80 GB with NVLink, with tensor parallel 4:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
 name: vllm-h100-tp4
spec:
 annotations:
 prometheus.kserve.io/port: "8080"
 prometheus.kserve.io/path: "/metrics"
 supportedModelFormats:
 - name: huggingface
 version: "1"
 autoSelect: true
 priority: 10
 protocolVersions:
 - v2
 containers:
 - name: kserve-container
 # ALWAYS pin by digest in production; the tag is only for readability
 image: vllm/vllm-openai:v0.11.0
 command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
 args:
 - --port=8080
 - --model=/mnt/models
 - --served-model-name={{.Name}}
 - --tensor-parallel-size=4
 - --max-model-len=32768
 - --gpu-memory-utilization=0.92
 - --enable-prefix-caching
 - --kv-cache-dtype=fp8
 resources:
 requests: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 limits: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 # the probe that stops the pod dying while the model loads
 startupProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 15
 failureThreshold: 40 # up to 10 minutes of loading
 readinessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 10
 livenessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 30

The InferenceService that consumes it, with KEDA on vLLM metrics:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 serving.kserve.io/deploymentMode: "Standard"
 serving.kserve.io/autoscalerClass: "keda"
spec:
 predictor:
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 # the OCI registry from article 2/4 of this series
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-07"
 autoScaling:
 metrics:
 - type: External
 external:
 metric:
 backend: "prometheus"
 serverAddress: "http://prometheus.monitoring.svc.cluster.local:9090"
 query: 'sum(vllm:num_requests_waiting{model_name="asistente-70b"})'
 target:
 type: Value
 value: "8"

The canary, with its small print:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 # WATCH OUT: canaryTrafficPercent only takes effect in KNative mode.
 # In Standard mode it is silently ignored (issue kserve/kserve#5335, open).
 serving.kserve.io/deploymentMode: "KNative"
spec:
 predictor:
 canaryTrafficPercent: 10
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-08"

KServe keeps three references in the status (LatestReadyRevision, LatestRolledoutRevision and PreviousRolledoutRevision): promoting means setting the percentage to 100, reverting means pinning traffic on the previous one, and a revision that does not become ready automatically receives zero traffic.

And the two calls. Against the V2 contract, for a classifier served with MLServer or Triton:

# metadata: which model it is, what inputs it expects, what types
curl -s http://clasificador.inferencia.example/v2/models/clasificador-tickets | jq .

# readiness of the specific model, not of the server
curl -s -o /dev/null -w '%{http_code}\n' \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/ready

curl -s -X POST \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/infer \
 -H 'Content-Type: application/json' \
 -d '{ "id": "req-4711",
 "inputs": [ { "name": "input-0", "shape": [1, 4], "datatype": "FP32",
 "data": [5.1, 3.5, 1.4, 0.2] } ],
 "outputs": [ { "name": "output-0" } ] }'

Against the same control plane, but on the OpenAI-compatible surface of the LLM:

curl -s -N -X POST \
 http://asistente-70b.inferencia.example/openai/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{ "model": "asistente-70b",
 "messages": [ {"role": "user",
 "content": "Summarise last night'\''s incident report."} ],
 "max_tokens": 256,
 "stream": true }'

Two contracts and two payload shapes, with the same InferenceService as the unit of governance. That is the value proposition of a control plane: the contract is chosen by the model and the management is single.

Autoscaling and the problem of zero

Three autoscalers coexist, and choosing the wrong one is the most common cause of disappointment. KPA, in KNative mode, scales by concurrency or RPS and is the only one that goes down to zero in a usable way, because Knative provides an activator that holds the request while the pod starts; its metric does not distinguish a 30-token request from an 8,000-token one. HPA, in Standard, scales by CPU or memory: for LLMs it is useless, because the CPU of a saturated vLLM pod and that of an idle one look too much alike. KEDA, in Standard, is the right option, the one we develop in autoscaling with KEDA: it is enabled with serving.kserve.io/autoscalerClass: "keda" and autoScaling.metrics accepts external metrics against Prometheus (vllm:num_requests_running or vllm:num_requests_waiting) or, with the OpenTelemetry add-on, pushed metrics. Since v0.18 there is also Workload Variant Autoscaling, which uses HPA or KEDA as a backend.

That leaves the serious problem, which no autoscaler solves: zero. A 70B model in fp8 is on the order of 70 GB to move into HBM, and that path, from disk to HBM, is measured in minutes: the first request after idling takes that long, or it fails.

Worse: the installation documentation is explicit as of July 2026: “Scale from Zero is currently not supported in Standard mode for HTTP requests”. The mode the project has pivoted towards, and the only one that supports KEDA, does not know how to come up from zero on an HTTP request: KEDA can go down to zero (default cooldown of 300 seconds), but the way back requires an activator that Standard does not have. Anyone who wants real zero with HTTP activation still needs Knative.

The levers, in order of effectiveness: do not scale the hot model to zero (minReplicas: 1 for anything with an SLA; zero is for the long tail); attack the load time, the only solution that is not a patch, with the techniques from speeding up cold start; use LocalModelCache, available since v0.15 and extended to LLMInferenceService in v0.19, to preload artifacts into a node PVC; and set correct probes, which is what comes next.

Operational traps

The probe that kills the pod while the model loads. The most frequent and the most expensive to diagnose, because the symptom is a CrashLoopBackOff with no error in the server logs. Until early 2026 the KServe vLLM templates used initialDelaySeconds of 120 to 300 seconds on liveness, with the result described in issue #5062 (12 February 2026, now resolved): the fast models waited too long and the large ones died before finishing loading. The fix applied: a startupProbe of up to ten minutes and probes with no initial delay. A legacy runtime has to be reviewed by hand. In KNative mode there is a worse variant (issue #3795, closed as not planned): what dies is not the model container but the queue-proxy, which does not respond OK within the deadline and takes the whole pod down with it.

canaryTrafficPercent silently ignored. It deserves its own trap because there is no error and no condition in the status to give it away: in Standard mode the field does nothing. The team believes it is serving 10 % with the new version and it is serving 100 %. While issue #5335 remains open, in Standard the canary is done with HTTPRoute weights by hand, or in the gateway.

Canary and scarce GPUs do not get along. A 10 % canary of a 70B with tensor parallel 4 costs a whole 4-GPU node, not 10 % of the resources, because the minimum granularity is a complete replica. During the window there are two versions occupying twice the hardware, or capacity has to be taken from the stable one. That is the difference between a microservice canary and a model canary, and the reason a shadow over duplicated traffic often pays off better.

storageUri and credentials. Storage initializer failures are always the same ones: a ServiceAccount without the annotated secret, a KMS-encrypted bucket with a role that has no decryption permission, an S3-compatible but not exactly S3 endpoint, or intermittent 403s, with a specific fix in v0.18. It is diagnosed in the init container logs, not in the server’s. And one standing recommendation: no downloading over https:// from the internet in production; that is precisely the problem of article 2/4.

The model size against the emptyDir. By default the artifact lands in an ephemeral volume on the node’s disk. Seventy gigabytes per replica fill a system disk with surprising ease, and then the kubelet evicts pods for disk pressure, including ones that have nothing to do with it. You have to set sizeLimit or use a dedicated pvc://. For multi-node the restriction is harder, with a mandatory ReadWriteMany PVC, which rules out a good part of local block storage and pushes you towards shared file systems, with the consequences covered in storage for AI.

The dependency on Knative and Istio. The real cost of KNative mode: two more control planes to version and correlate with Kubernetes, and two sidecars, queue-proxy and istio-proxy, in every pod, with their CPU and memory consumption and their network hop per request. On a platform that already has a mesh it is a marginal cost; on a cluster dedicated to inference it has to be justified by the capabilities you are actually going to use.

Observability: the name of the metric. The ServingRuntime declares prometheus.kserve.io/port and /path annotations, but the metrics that come out are the engine’s, not KServe’s: vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:gpu_cache_usage_perc. The control plane does not normalise names across runtimes, so a dashboard written against vLLM is no use for Triton, and correlating with DCGM is still your own work. v0.19 raises the HPA/KEDA scaling status into the service conditions and emits events on readiness transitions: exactly what was needed to alert on the control plane and not only on the engine.

Decision map: when KServe, and when it is overkill

The useful comparison is not “KServe versus OME, vLLM Production Stack, Dynamo or llm-d”. Those four are covered in LLM inference operators and compete on a different axis:

Axis	What gets decided	Who competes
Contract and control plane	Which CRD describes a service, which API clients see, how the model is versioned, how the rollout is done, how LLM and non-LLM coexist	KServe, Seldon Core v2
Engine and topology	Which engine runs, how prefill/decode is disaggregated, how routing is done with cache awareness, how multi-node tensor parallel is done	vLLM Production Stack, OME, NVIDIA Dynamo, llm-d

The axes are orthogonal, and LLMInferenceService proves it: KServe did not reimplement distributed scheduling, it leaned on llm-d. Choosing KServe is not choosing instead of an LLM operator: it is choosing the contract layer that operator lives on.

Choose KServe when the catalogue is heterogeneous, LLMs plus embeddings, rerankers, classifiers and vision, which is its decisive advantage and the one none of the four operators has: a service like the one in TEI in production fits into the same vocabulary as the LLM. Also when you need a contract decoupled from the engine because you have clients you do not control; when you need declarative, auditable governance (which artifact, from which origin and with which runtime, in an object versionable in git and subject to admission policies), the piece that makes the rest of this series possible; and when you want a catalogue of platform runtimes separated from the teams’ services.

KServe is overkill when you serve one model, on one node, with up to three replicas: the Deployment from vLLM on Kubernetes is the right answer. When your workload is LLM only and the priority is performance per GPU, because there the value is in the engine. When you are going to use neither scale-to-zero nor canary by revision, because what remains is a Deployment with more YAML. And when you already have an LLM operator and only serve LLMs: putting KServe on top for completeness is adding CRDs without adding decisions.

For an inference factory

First: separate the runtime catalogue from the service catalogue, and do it on day one. The ClusterServingRuntimes belong to the platform; the InferenceServices, to the teams. That boundary turns “serving the thirtieth model” into a five-line change instead of a negotiation about vLLM flags, and it fits with GitOps with Flux: the runtime is reviewed as platform code, the service as product configuration.

Second: accept the two contracts and put the gauge changer where it belongs. V2 for the tensorial world, the OpenAI API for the generative one, and the translation in the gateway, not in every client. And since the OpenAI API does not version in the route, encode the version in the model name and make that name the same identifier that appears in the registry, in the signature and in the traces. That is what makes a request traceable back to the artifact.

Third: measure start-up before promising elasticity. The time from the pod being scheduled to the readiness probe going green, with the real model and from the real origin, decides whether scale-to-zero is a FinOps lever or an incident machine. That number, and not the price of the GPU-hour, is the one to take into the cost conversation.

And here this piece falls short on purpose. KServe gives you the contract, the lifecycle and the rollout, but its whole trust model hangs off a text string: storageUri. The storage initializer downloads whatever is at that destination and puts it in front of the GPU without asking anything else; nothing in the InferenceService guarantees that the artifact is the one that was approved, that it has not changed since the last audit, or that the origin is legitimate. The next article goes there: where the model bytes come from, and why an OCI registry with ORAS, Harbor, MLflow or the Kubeflow Model Registry is the difference between distributing models and scattering them.

Sources

CNCF, KServe becomes a CNCF incubating project (11 de noviembre de 2025) — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
CNCF, KServe project page (incubating, aceptado el 29 de septiembre de 2025) — https://www.cncf.io/projects/kserve/
CNCF TOC, Project Moving Levels Checklist: KServe joining CNCF at Incubation level — https://github.com/cncf/toc/issues/1905
KServe, Releases (v0.15.0, v0.16.0, v0.18.0, v0.19.0) — https://github.com/kserve/kserve/releases
KServe, Inference Protocol V2 (Open Inference Protocol) — https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol
KServe, open-inference-protocol (especificación, adoptantes, gobernanza) — https://github.com/kserve/open-inference-protocol
KServe, open-inference-protocol — generate_rest.yaml — https://github.com/kserve/open-inference-protocol/blob/main/specification/protocol/generate_rest.yaml
open-inference, Vendor neutral fork of kserve/open-inference-protocol — https://github.com/open-inference/open-inference-protocol
KServe, Generative Inference — Runtime Overview (endpoints con prefijo openai/) — https://kserve.github.io/website/docs/model-serving/generative-inference/overview
KServe, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
KServe, Cloud-Native AI Inference at Scale using KServe and llm-d (5 de marzo de 2026) — https://kserve.github.io/website/blog/cloud-native-ai-inference-kserve-llm-d
KServe, Serving Runtime (ServingRuntime y ClusterServingRuntime) — https://kserve.github.io/website/docs/concepts/resources/servingruntime
KServe, Kubernetes Deployment Installation Guide (K8s 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1, sin scale-from-zero en Standard) — https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment
KServe, Canary Rollout Strategy — https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary
KServe, Autoscaling with KEDA — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/keda-autoscaler
KServe, Multi-node/Multi-GPU Inference (PVC RWX, sin autoescalado) — https://kserve.github.io/website/docs/model-serving/generative-inference/multi-node
KServe, Storage Options for Model Artifacts — https://kserve.github.io/website/docs/model-serving/storage/overview
KServe, issue #5335 — Support canaryTrafficPercent for RawDeployment mode via Gateway API HTTPRoute weights (abierta, 2 de abril de 2026) — https://github.com/kserve/kserve/issues/5335
KServe, issue #5062 — Add startupProbe to vLLM main containers (12 de febrero de 2026) — https://github.com/kserve/kserve/issues/5062
KServe, modelmesh-serving (última release v0.12.0 de julio de 2023; repositorio archivado el 14 de abril de 2026) — https://github.com/kserve/modelmesh-serving
Red Hat, Converting ModelMesh and Serverless InferenceServices to RawDeployment (Standard) Mode — https://access.redhat.com/articles/7134025
NVIDIA, Triton Inference Server — Inference Protocols and APIs — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inference_protocols.html
OpenVINO, What is OpenVINO Model Server (API KServe, TFS y endpoints compatibles con OpenAI) — https://docs.openvino.ai/2026/model-server/ovms_what_is_openvino_model_server.html
PyTorch, TorchServe — Notice: Limited Maintenance — https://docs.pytorch.org/serve/
Kubernetes SIG Network, Gateway API Inference Extension (v1.5.0, 19 de abril de 2026) — https://github.com/kubernetes-sigs/gateway-api-inference-extension

OpenCost in depth: how GPU cost is allocated in Kubernetes

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. European references (OVH is among the supported providers). The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Second article in the FinOps track (A2), and the first tool deep dive. The FinOps introduction established that OpenCost is the CNCF standard for cost allocation; here the box gets opened: how it allocates cost internally, where it gets its data from, how on-prem prices are configured in euros, what its API exposes, and where the traps are. Understanding the mechanics matters because it defines what any tool on top (Kubecost included) can and cannot do, and because a badly set base price invalidates the whole split. No recommendations; only the mechanics and the methodology.

What OpenCost is

OpenCost is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project). It is a cost allocation layer for Kubernetes: it reads resource usage from the cluster, joins it with a price, and splits the cost across Kubernetes dimensions down to the container (OpenCost · GitHub). It supports dynamic pricing via the billing APIs of AWS, Azure and GCP, and custom prices for on-prem clusters (OpenCost · configuration).

What it is not: it is not an optimiser, nor a governance system, nor a product unit economics tool. It is the piece that answers “how much does each namespace/pod/team cost?”, and leaves the rest to the layers above. That is why it is the foundation of the FinOps track: without correct allocation, there is no cost per token and no TCO model worth anything.

The cost model: node-level allocation

The key to OpenCost is that the model works at node level. It starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit per-resource prices, OpenCost uses the ratio of a set of base prices (marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem).

This is what makes it work on-prem: you declare the node cost (depreciated capex + opex), and OpenCost splits it without needing a cloud invoice. The normalisation guarantees that no cost is “lost” or “invented”: what you pay for the node is exactly what gets split across its resources.

Where the data comes from: Prometheus

OpenCost instruments nothing of its own: it reads from Prometheus, which is a prerequisite of the installation (OpenCost · GitHub). The sources:

Source	What it provides
`kube-state-metrics`	state of K8s objects (pods, requests/limits)
`node-exporter`	node resources and capacity
`cAdvisor`	real CPU/memory usage per container
`dcgm-exporter` (NVIDIA)	GPU usage, memory and power

With those series, OpenCost automates the join between Kubernetes resource usage and the provider price, reducing the custom PromQL you would otherwise write by hand (Grafana · manage costs). The GPU signal comes from DCGM via the dcgm-exporter (part of the NVIDIA GPU Operator), the same base as GPU observability. Without Prometheus and without DCGM exporting, OpenCost does not see the GPU.

Architecture: the pieces

OpenCost is not a monolith; it is three responsibilities worth telling apart:

Piece	Function
Cost model	resolves the node price and splits/normalises it across resources
Allocation API	serves the allocated cost per dimension (`/allocation` and endpoints)
Exporter	publishes cost metrics on `/metrics` for Prometheus/Grafana

The data flow: Prometheus holds the usage series (from kube-state-metrics, cAdvisor, dcgm-exporter); the cost model joins them with the resolved price; the Allocation API serves them aggregated by whatever dimension you ask for; and the exporter returns them to Prometheus so that Grafana can plot them. It is a cycle: metrics come in from Prometheus and cost goes back to Prometheus enriched. That symmetry is what lets you reuse your observability stack without standing up a new database: OpenCost lives on top of Prometheus, not beside it.

An operational note: historical analysis depends on Prometheus retention. For cost reports spanning months, a long-term backend behind it (Thanos, Mimir, VictoriaMetrics) is advisable, or periodically exporting the allocations to a store of your own.

On-prem prices in euros: the piece that decides everything

On-prem, the most important figure, and the most neglected, is the node price. It is configured per provider; the options are alibaba, aws, azure, gcp, oracle, ovh or default (on-prem) (OpenCost · configuration). For on-prem you use default, supplying base prices in a default.json (or by overriding the Helm values.yaml); anything you do not override uses the chart’s value (OpenCost · on-prem).

To set the node price in euros, the calculation is the TCO model one: depreciated capex + opex. An example of an 8×H100 node:

Component	Calculation	€/hour
Capex (node ~240,000 €, 36 months)	240,000 ÷ 26,280 h	~9.13
Energy (5.6 kW × PUE 1.4 × 0.058 €/kWh, France)	~0.455 €/h	~0.46
Maintenance, networking, operations	estimate	~1.5
Node total		~11.1 €/h

That ~11 €/h is the number you put into OpenCost, and everything is split from it. And here is the documented trap: with the default pricing configuration, OpenCost under-prices CPU and GPU on-prem (issue #3781). If you do not adjust the base prices to your real cost, the split will be systematically low and the cost per token you report will be unreal. Configuring the node price is not optional: it is the figure everything else hangs from.

Configuring the price: an example

The base price is given in the Helm values.yaml (or a default.json), in euros per hour and per unit of resource, derived from the node’s total cost:

# values.yaml — on-prem prices in €/hour (8xH100 node, France)
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 8xH100 node"
 CPU: "0.030" # €/CPU-hour
 RAM: "0.004" # €/GB-hour
 GPU: "1.30" # €/GPU-hour ← the figure that moves cost/token the most
 storage: "0.0002" # €/GB-hour

OpenCost normalises those values so that, multiplied by the node capacity, they add up to the total price you declared (~11 €/h in the example). The number to be most careful with is the GPU one: at 1.30 €/GPU-hour, the 8 cards are ~10.4 €/h, the bulk of the node’s cost. A badly set GPU base shifts the whole cost per token. Always review the resolved prices in /allNodePricing after configuring, because the default under-pricing bug bites exactly here.

Allocation: by usage, not by presence

The behaviour that has changed most and that matters most for GPU: in recent versions, the cost allocated to a GPU is determined by its usage, not its presence. If a GPU costing 100 €/month is only half used, the allocated cost is ~50 € ([search]). This aligns allocation with reality, you pay for what you use, but it has a consequence: the other half does not disappear, it is idle that somebody keeps paying for. OpenCost exposes it, and that is where the optimisation lever comes from.

OpenCost allocates to any Kubernetes dimension: cluster, node, namespace, controller (deployment, statefulset and so on), service, pod and container, and by labels (team, product). Access is through the Allocation API and several endpoints that expose the mechanics:

Endpoint	What it returns
`/allocation`	cost allocated by the dimension you ask for
`/costDataModel`	the resolved node price
`/allNodePricing`	hourly prices per node
`/pricingSourceSummary`	summary of the pricing source
`/metrics`	cost metrics for Prometheus

The /metrics endpoint is what turns OpenCost into a Prometheus exporter: once there, you can write PromQL to calculate the cost and efficiency of any Kubernetes concept and build panels in Grafana (OpenCost · exporter).

Allocation API parameters

/allocation is controlled by a handful of parameters worth knowing, because they define which number you get:

Parameter	What it controls	Example
`window`	time range	`7d`, `today`, specific dates
`aggregate`	aggregation dimension	`namespace`, `label:team`, `pod`
`accumulate`	sum the range or break it per interval	`true` (total) / `false` (series)
`idle`	include idle cost or not	`true` / `false` / `separate`
`filter`	filter by namespace, label and so on	`namespace:"llm-prod"`

The idle parameter is the most revealing: with idle=separate, OpenCost returns the idle cost as a separate row, which lets you see at a glance how much is being paid for unused capacity. A query with aggregate=label:team and idle=separate over window=30d is, literally, the monthly chargeback report with the waste highlighted.

GPU in depth: DCGM, usage, MIG and idle

The GPU is the expensive resource, so its allocation deserves detail:

Usage signal: dcgm-exporter gives DCGM_FI_DEV_GPU_UTIL (utilisation), DCGM_FI_DEV_FB_USED (memory), and power. OpenCost uses utilisation to allocate by usage.
MIG (Multi-Instance GPU): an A100 is split into up to 7 isolated instances, which allows cutting idle from 50 % to almost 0 % by sharing one card across small workloads ([search]; see sharing the GPU: time-slicing, MPS and MIG). The cost allocation of each MIG instance follows the same usage logic.
Idle: the reliable detection pattern combines three pieces, DCGM metrics in Prometheus, an alert that fires when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, and routing of that alert to the team that owns the namespace ([search]).

An attribution nuance worth knowing: MIG creates isolated instances that Kubernetes sees as distinct resources, so OpenCost allocates them cleanly, each one to its pod. Time-slicing, by contrast, shares the same physical GPU across several pods with no compute isolation: there, per-pod utilisation is harder to separate, and cost attribution becomes approximate (the GPU is split across the pods sharing it, according to their measured usage). The practical rule: if you need exact per-team chargeback on a shared GPU, MIG gives cleaner attribution than time-slicing, at the price of rigid fixed partitions. It is a trade-off between cost accuracy and flexibility that is better decided before, not after, building the split.

Efficiency: the number that triggers action

With cost allocated by usage, OpenCost lets you calculate efficiency: how much of the allocated cost corresponds to useful work against the total. A GPU at 30 % utilisation has an efficiency of 30 %: the remaining 70 % is idle cost that somebody pays without receiving work. That figure, per team and per namespace, is what turns a dashboard into a chargeback conversation: “your namespace has 5 GPUs allocated at 25 % efficiency; either you raise utilisation or you release three cards”. It connects with FinOps and multi-tenancy: allocation locates the waste; scheduling and co-residency recover it.

Worked example: the allocation report

On the ~11 €/h node (8×H100), three teams share the cluster for a month. What a /allocation?aggregate=label:team&idle=separate&window=30d query returns:

Team	GPU-hours	Avg. util.	Allocated cost (€)	Efficiency
A · chat-prod	2,880 (4 GPU)	78 %	~3,744	high
B · batch	1,440 (2 GPU)	55 %	~1,872	medium
C · experimentation	1,440 (2 GPU)	22 %	~1,872	low
idle (separate)	—	—	~1,100	—

What the report reveals: the cost allocated to A, B and C comes out of their GPU-hours and the node price; but the idle row (~1,100 €/month) is capacity nobody used and everyone pays for in the pro rata. And the efficiency column points at C: 2 GPUs at 22 % is ~1,560 € a month of which only ~340 € is useful work. That is the actionable figure: C does not have a model problem, it has a utilisation problem. Without OpenCost, those ~1,100 € of idle and C’s inefficiency stay diluted in an aggregate invoice nobody questions. With it, they are a named row and a number in euros.

Useful PromQL queries

As a Prometheus exporter, OpenCost lets you build the panels by hand. Some reference queries (exact metric names vary by version; check in your /metrics):

# Cost per node and hour (resolved price)
node_total_hourly_cost

# GPU cost allocated per namespace (€/hour)
sum by (namespace) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
)

# Average GPU utilisation per namespace (for efficiency)
avg by (namespace) (DCGM_FI_DEV_GPU_UTIL)

# Idle GPUs: utilisation < 10 % sustained
DCGM_FI_DEV_GPU_UTIL < 10

The last one, wrapped in an alert with for: 15m and routed to the namespace owner, is the standard idle detection pattern. With these series in Grafana you have cost per team, efficiency and waste on one dashboard, the “Inform report” of the FinOps phase in real time.

Dynamic cloud prices vs custom on-prem prices

The difference is worth understanding, because it changes how reliable the figure is:

Cloud (aws/azure/gcp/oracle/ovh): OpenCost obtains the price dynamically from the provider’s billing API, so the allocated cost reflects the real rate (including discounts, spot and so on) without you declaring anything.
On-prem (default/custom): you declare the price, derived from your TCO model. The advantage is total control; the risk is that a badly calculated price skews everything. That is why the TCO model (depreciated capex + opex) is not an academic exercise: it is literally OpenCost’s input.

The ovh provider among those supported is relevant for a European platform: if part of the workload goes to a European sovereign cloud, OpenCost can allocate its cost with dynamic pricing, and mix it with on-prem in the same allocation view.

From pod cost to cost per token

OpenCost gets as far as “this vLLM pod cost X €/hour”. The cost per token, the metric that compares on-prem against cloud, needs one more piece: the gateway (LiteLLM) that counts tokens per request and per team, as covered in the FinOps introduction. The full chain is: OpenCost gives the pod cost by usage → the gateway gives the tokens per team → dividing them gives the cost per token per team. OpenCost is the hardware half of that equation; without it, you would know the tokens but not their real cost. That is why A2 (allocation) is the foundation the whole cost per token of the proposal hangs from.

Deployment: Helm + Prometheus

The typical deployment is via Helm, with Prometheus as a prerequisite (for scraping and series storage). Conceptual steps:

Have Prometheus with kube-state-metrics, node-exporter, cAdvisor and dcgm-exporter.
Install OpenCost with Helm, pointing it at your Prometheus.
Configure the node price in euros (default.json / values.yaml), the step most often neglected.
Expose /metrics and build panels in Grafana with PromQL over the cost metrics.

As a Prometheus exporter, OpenCost lets you write PromQL for the cost and efficiency of any concept and create bespoke dashboards (OpenCost · exporter). It reuses infrastructure a GPU cluster already has for observability (DCGM, Prometheus, Grafana): OpenCost only adds the pricing and allocation layer.

OpenCost vs Kubecost (just the essentials)

OpenCost is the free base; Kubecost (IBM) is the commercial product built on it. Briefly, what the commercial one adds:

Capability	OpenCost	Kubecost
Cost allocation (CPU/GPU/mem/PV)	✓	✓
GPU via DCGM	✓	✓ (3.0)
Long historical retention	depends on your Prometheus	included
Automatic rightsizing	—	✓ (Turbonomic)
Governance, alerts, enterprise RBAC	basic	✓
Commercial support	community	✓ (IBM)

For most on-prem clusters with a platform team, OpenCost covers allocation; Kubecost adds value when you want automated optimisation and enterprise features without operating it yourself. The detail of the comparison, and when paying is worth it, is article A3.

Implementation checklist

For OpenCost to give a defensible per-team cost, order matters:

Step	Action	Verification
1	Prometheus with kube-state-metrics, node-exporter, cAdvisor	series present
2	NVIDIA GPU Operator + dcgm-exporter	`DCGM_FI_DEV_GPU_UTIL` in Prometheus
3	Install OpenCost (Helm) pointing at Prometheus	`/allocation` responds
4	Configure the node price in € (capex+opex)	`/allNodePricing` shows the real one
5	Label pods/namespaces by team and product	`aggregate=label:team` splits correctly
6	Grafana panels for cost, efficiency and idle	the waste is visible
7	Idle alert (`util<10` `for: 15m`) to the owner	the alert reaches the team
8	(optional) long-retention backend	months of history available

Step 4 is the one most often skipped and the one that most invalidates the result. If you only do one thing right, make it that one: without the correct node price in euros, steps 5 to 7 split a wrong cost with great precision.

Limits and traps (data-driven)

Default under-pricing. With the default pricing config, OpenCost underestimates on-prem CPU and GPU (#3781). Adjust the base prices to your real cost or the whole split will come out low.
It depends on Prometheus and DCGM. Without the right series (especially dcgm-exporter), there is no GPU allocation. Input data quality rules.
Allocation by usage ≠ total cost. A GPU at 50 % charges half; the other half is idle that still costs. Do not confuse “allocated cost” with “cost paid”.
Prometheus retention. Historical cost analysis depends on how much your Prometheus retains; for long series, plan a long-term backend.
Allocation, not per-token measurement. OpenCost reaches the pod; cost per token needs the gateway (LiteLLM) above it, as covered in the FinOps introduction.

With OpenCost’s mechanics clear, the next article (A3) compares what Kubecost and the commercial alternatives add, to decide which FinOps stack to adopt. But the foundation, correct allocation with the node price properly set in euros, is this one.

Closing

OpenCost looks like “installing a cost tool” and is, in reality, declaring what your hardware is worth and letting it split itself. That simplicity is its strength, it lives on top of the Prometheus and DCGM you already have, with no new database, and also its trap: the split is only as good as the node price you declare, and the default value under-prices on-prem GPU. Properly configured in euros, OpenCost turns the cluster’s aggregate invoice into a split per team, per product and per GPU, separates used cost from idle, and exposes the efficiency that triggers chargeback conversations. It is the hardware half of the cost per token equation, the other half, the tokens, is supplied by the gateway, and without this half there is no TCO model or on-prem vs cloud comparison that holds up. For a sovereign architecture proposal, OpenCost is the piece that makes cost stop being an intuition and become a table in euros, reproducible and auditable, with the idle GPU named and identified. The Inform phase of GPU FinOps starts here, and everything else hangs on this number being right.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · configuración (proveedores: aws/azure/gcp/oracle/ovh/default) — https://opencost.io/docs/configuration/
OpenCost · configuración on-prem (precios personalizados) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exportador de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
OpenCost · issue #3781 (infra-precio CPU/GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Grafana · manage costs (OpenCost + Kubernetes) — https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/manage-costs/

Cncf on lo0 — Tech Blog

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

TL;DR

The analogy: the station master and the track gauge

A Deployment serves tokens, but offers no contract

KServe: from KFServing to a CNCF incubating project

Anatomy of the control plane

InferenceService: predictor, transformer, explainer

ServingRuntime and ClusterServingRuntime: the catalogue of trains

The storage initializer and storageUri

The deployment modes

The LLM branch: LLMInferenceService

The Open Inference Protocol: the contract itself

The routes and the payload

Who actually implements it

The generate extension and why it did not win

The tension, unvarnished

Four manifests you can actually apply

Autoscaling and the problem of zero

Operational traps

Decision map: when KServe, and when it is overkill

For an inference factory

See also

Sources

OpenCost in depth: how GPU cost is allocated in Kubernetes

What this article covers

What OpenCost is

The cost model: node-level allocation

Where the data comes from: Prometheus

Architecture: the pieces

On-prem prices in euros: the piece that decides everything

Configuring the price: an example

Allocation: by usage, not by presence

Allocation API parameters

GPU in depth: DCGM, usage, MIG and idle

Efficiency: the number that triggers action

Worked example: the allocation report

Useful PromQL queries

Dynamic cloud prices vs custom on-prem prices

From pod cost to cost per token

Deployment: Helm + Prometheus

OpenCost vs Kubecost (just the essentials)

Implementation checklist

Limits and traps (data-driven)

Closing

See also

Sources

A `Deployment` serves tokens, but offers no contract

`InferenceService`: predictor, transformer, explainer

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

The storage initializer and `storageUri`

The LLM branch: `LLMInferenceService`

The `generate` extension and why it did not win