Gpu on lo0 — Tech Blog

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

Mon, 31 Aug 2026 10:00:00 +0200

Closing the platform and self-service run, after Backstage and Kubeflow. The question here is purely economic with a technical trap inside: switching off the GPU when nobody is using it looks like free money, until the first user of the day waits two minutes.

TL;DR

A GPU costs the same serving as it does idle. An H100 on dedicated cloud runs at 2 to 4 euros an hour whether it is used or not, so switching the service off when there is no traffic looks like money given away. Knative is the piece that does it: its autoscaler scales down to zero replicas and brings them back when a request arrives, measuring concurrency instead of CPU, which is what the Kubernetes autoscaler cannot do.

The trap is in the cold start. Knative brings up a pod in seconds, but that is only the beginning: an LLM has to load its weights from disk into HBM and compile its graphs, and there you go from tens of seconds to minutes depending on model size and storage. One reference study measures vLLM startup at around 20 seconds even for a 3 billion parameter model, and weight loading grows linearly with size. The first user of every traffic trough pays that whole wait.

Hence the rule. Switching off the GPU pays off with sporadic traffic, many lightly used models or development environments, and it is a trap with a strict latency SLO, sustained traffic or huge models whose startup breaks any interactive budget. Between all and nothing there are three middle grounds that almost always win: keep one replica warm, share the GPU with model-swap, or speed up weight loading so that startup fits inside the autoscaler’s cycle.

The analogy: the restaurant that switches off the kitchen

A restaurant with the kitchen running at four in the afternoon, without a single customer, burns gas for nothing. The temptation is to switch it off and light it again when somebody walks in. The problem is the time it takes to heat up. If the kitchen is an induction hob, it heats in seconds and switching it off between services is obvious. If it is a wood-fired oven that takes two hours to come up to temperature, switching it off means the first customer of the night eats cold or leaves.

Serving a small model is an induction hob. Serving a 70 billion parameter model is the wood-fired oven: the weights that have to go up into HBM are the wood you have to burn until you reach temperature, and wishing does not speed it up. The decision to switch off the kitchen, that is, to scale to zero, does not depend on whether you want to save gas, but on how long your kitchen takes to heat and on how long your first customer is willing to wait. This whole article is that calculation.

How Knative scales

Knative graduated in the CNCF in October 2025 and has three components; the one that concerns us is Serving, the serverless application component with autoscaling. Its object model has four pieces: the high-level Service, the Configuration that describes the desired state, the immutable Revision which is what actually scales, and the Route that splits traffic between revisions.

What makes scaling to zero possible are two data plane components. The queue-proxy is a sidecar in every pod that measures concurrency and applies the simultaneous request limit. The activator is the key piece: when the service is at zero replicas, requests are routed to it, and it parks them in a queue, tells the autoscaler that capacity is needed, and forwards the parked request once a pod is ready. It is the waiter who takes your order and tells you the kitchen is warming up, instead of shutting the door on you.

KPA versus HPA

Knative’s own autoscaler, the KPA, measures concurrency and requests per second, not CPU or memory. That is the difference that matters: the Kubernetes autoscaler, the HPA, scales on CPU and does not know how to go down to zero, its operational minimum is one. To switch off completely you need the KPA.

The KPA works with two windows. The stable one, 60 seconds by default, averages normal traffic. The panic one, much shorter, reacts to sharp spikes: when demand exceeds twice the capacity of the current replicas, it enters panic mode and scales in one go. The default concurrency target per replica is 100, but for LLM inference the useful value is usually 1 to 4, because each request occupies the GPU heavily; the official KServe example uses a target of 1.

Two timers govern the shutdown. The scale-to-zero-grace-period, 30 seconds by default, is how long the system waits to have the restart machinery ready before withdrawing the last replica. The scale-to-zero-pod-retention-period, zero by default, is the minimum time the last pod survives after the decision to switch off. Raising this second value is one of the levers for not switching off as soon as there is a micro-trough in traffic.

Why an LLM cold start is not starting a pod

Here is the core of the article, and there is a nuance that most discussions skip. Knative bringing up a pod in seconds does not mean the model is ready in seconds. Starting an inference service has several phases, and the two costs that dominate are not the ones you would expect.

A recent study measuring vLLM cold start breaks it down step by step. For a 3 billion parameter model, total startup is around 20 seconds, and the counterintuitive finding is that most of it is CPU-bound, not GPU-bound: process initialisation, library imports, engine construction. Inside that there are two costs that grow, and they are the ones the acceleration techniques attack.

The first is weight loading, which scales linearly with model size. In that same study it goes from half a second for a small model to almost five for a 16 billion one. Projected onto real sizes, the on-disk weight rules: an 8 billion model takes around 16 GB in FP16 and 8 in FP8; a 70 billion one, 140 and 70; a 405 billion one, 810 and 405. Loading 140 GB from disk into HBM does not happen in a blink, and if storage is slow or sits on the network, the minute arrives on its own.

The second is graph compilation. Graph transformation and capture with torch.compile costs 11 to 21 seconds if there is no compilation cache, and drops to 3 to 6 with one. It is half a battle that has nothing to do with the weights, and it is won by keeping the compilation cache between startups.

The operational conclusion is that the enemy is not only HBM: it is weight loading plus compilation. For small models the weights are only a fraction of startup, and switching off the GPU is almost painless. For large models the weights dominate, and the cold start becomes the problem. Loading a 70 billion model with the naive loader of the usual library can run to several minutes, and that is exactly the time your first user waits.

The techniques that shorten the wait

The good news is that the cold start can be attacked on both fronts. On the weights side, streaming loaders such as NVIDIA’s transfer the weights to the GPU concurrently instead of sequentially: the published data takes an 8 billion model from around 48 seconds with the standard loader to around 14 with streaming from SSD, and to under 8 with fast storage. The most graphic case is a 122 billion model, 233 GB on disk, which goes from three and a half minutes to around 37 seconds. That number has a direct consequence: 37 seconds fit inside an autoscaler polling cycle of 30 to 60 seconds; three and a half minutes do not. It is the difference between scaling to zero being viable with large models or not being viable at all.

On the compilation side, the technique is cheaper still: keep the torch.compile cache between startups, and with it the 11 to 21 seconds become 3 to 6. Tools such as Tensorizer and OCI images with the weights packaged inside attack the same problem from another angle, as we saw in the article on speeding up cold start.

Knative and KServe: the two modes

KServe leans on Knative for its serverless mode, and this is what connects everything above to a real platform. KServe has two deployment modes.

The Serverless mode uses Knative Serving and the KPA, supports scaling to zero and measures concurrency and requests per second. It is enabled by setting minReplicas: 0 in the InferenceService. It is the natural path for serving many intermittent models.

The Standard or raw mode uses the Kubernetes HPA, does not scale to zero, and measures CPU and memory. You pick it when you need something Knative restricts, such as mounting several volumes, and you do not need to switch off completely.

There is a third path of particular interest to anyone who already has KEDA deployed: in Standard mode, KServe recommends KEDA for generative inference, and KEDA can scale to zero on Prometheus metrics such as queue length or tokens per second. It is a route to scaling to zero on your own metrics, outside Knative, and it fits better with autoscaling guided by the model’s real saturation signal rather than by generic concurrency.

For anyone serving many lightweight predictive models there is also ModelMesh, the high-density pattern that packs many models into the same pods and rotates them in memory, without leaning on Knative. It is the equivalent of model-swap: instead of switching off and on per model, it shares one GPU between many. It is aimed at classic short-response inference, not at generative LLMs.

The economics

The decision comes down to comparing what you save by switching off with what you risk at startup. The saving is easy to estimate. If a GPU costs $ C $ euros per hour and your service is idle $ H $ hours a day, switching it off saves on the order of

$$\text{daily saving} \approx C \cdot H$$

With an H100 at 3 euros an hour and 16 idle hours a day, that is 48 euros a day per GPU, on the order of 1,400 a month. Across a fleet of several lightly used models the figure multiplies and stops being negligible.

The cost is on the other side, and it is not measured in euros: it is measured in SLO. Scaling to zero only pays off if the cold start fits inside what your worst-case latency tolerates. With a small model and fast storage, tens of seconds at most, and for an internal assistant used in bursts that may be acceptable. With a 70 billion model and the naive loader, minutes, and no interactive SLO survives that.

Between switching off completely and never switching off there are three middle grounds, and almost always one of them is the right answer:

Warm replica, with minReplicas: 1. You remove the first user’s cold start in exchange for paying a continuous GPU-hour. The break-even point is direct: keeping it warm pays off when the cost of that GPU running hot is lower than the reputational or SLO cost of the first user waiting. For a customer-facing service, almost always.

Model-swap or sleep on a shared GPU, as we saw in serving several models on one GPU. The process stays alive and the model is unloaded and reloaded in seconds instead of starting from scratch. It is the middle ground for several models that do not justify a GPU each but do not tolerate a full startup either.

Speed up loading so that the cold start fits inside the autoscaler’s cycle, with weight streaming and a compilation cache. It is what makes real scaling to zero viable with large models, and what turns “three and a half minutes” into “37 seconds”.

Operational traps and honest scepticism

The pod starts in seconds, the model does not. Do not confuse Knative’s startup time with the time until the model answers. The difference is the whole article.

The default concurrency target is 100, and for an LLM that is a disaster. Each request occupies the GPU; a target of 1 to 4 is the reasonable one. Leaving the default 100 means accepting a hundred simultaneous requests on a replica that handles four.

The cold start is not just weight loading. Graph compilation costs what it costs, and it is attacked separately, with a cache. Optimising only storage leaves half the problem untouched.

Cilium and Gateway API are not a supported path yet. Knative supports Gateway API in beta and tests with Istio, Contour and Envoy Gateway; integration with Cilium does not appear as tested and has documented friction. If your ingress is Cilium, verify it before counting on it.

KEDA may be a better signal than the KPA. Scaling on generic concurrency is worse than scaling on the model’s real saturation metric. If you already have KEDA, Standard mode with scaling to zero on your own metrics deserves a test against the default Serverless.

For an inference factory

The decision to switch off the GPU is not ideological, it is arithmetic, and it depends on two numbers: how long your model takes to start and how long your worst user waits. Three rules close the matter.

First, switch off what is intermittent and cheap to start: development environments, small models used sporadically, the long tail of models almost nobody touches. There, scaling to zero is direct money with no appreciable SLO cost.

Second, do not switch off what is critical and expensive to start: the customer-facing model with a strict SLO, and the huge models whose cold start is measured in minutes. There a warm replica costs less than a user waiting.

Third, before deciding, measure your real startup and fix it. Many services are ruled out of scaling to zero over a three-minute startup that, with weight streaming and a compilation cache, drops to thirty seconds and changes the answer. The most profitable lever is not choosing well between switching off or not: it is making switching off cheap.

With this we close the platform and self-service run. The portal shows what is there, the ML toolbox gives what is missing without duplicating what you already have, and scaling to zero decides how much of all that is switched on when nobody is looking. All three answer the same question from different angles: how to put an expensive platform at the disposal of its users without going broke keeping it switched on.

Sources

CNCF, Cloud Native Computing Foundation announces Knative’s graduation — https://www.cncf.io/announcements/2025/10/08/cloud-native-computing-foundation-announces-knatives-graduation/
Knative Docs, Serving architecture — https://knative.dev/docs/serving/architecture/
Knative Docs, Request flow — https://knative.dev/docs/serving/request-flow/
Knative Docs, Configuring KPA-specific autoscaling — https://knative.dev/docs/serving/autoscaling/kpa-specific/
Knative Docs, Configuring concurrency — https://knative.dev/docs/serving/autoscaling/concurrency/
Knative Docs, Configuring scale-to-zero — https://knative.dev/docs/serving/autoscaling/scale-to-zero/
Knative Blog, Managing Gateway API ingress with the Knative Operator — https://knative.dev/blog/articles/gateway-api-ingress-with-knative-operator/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Autoscaling with Kubernetes HPA (Serverless vs Standard) — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/hpa-autoscaler
KServe Docs, Autoscaler for generative inference (KEDA) — https://kserve.github.io/website/docs/model-serving/generative-inference/autoscaling
KServe GitHub, docs/samples/autoscaling (InferenceService minReplicas 0) — https://github.com/kserve/kserve/blob/master/docs/samples/autoscaling/README.md
Wang et al. (MLSys 2026), Breaking the ice: analyzing cold start latency in vLLM — https://arxiv.org/abs/2606.07362
NVIDIA Developer Blog, Reducing cold start latency for LLM inference with NVIDIA Run:ai Model Streamer — https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer
Microsoft Azure SDK Blog, Eliminate LLM cold starts: load models up to 6x faster — https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/
Hugging Face Blog, Llama 3.1 (VRAM and precision tables) — https://huggingface.co/blog/llama31
CloudZero, H100 GPU cost 2026 — https://www.cloudzero.com/blog/h100-gpu-cost/

Model chain of trust (4/4): who serves the model and which machine you trust

Sun, 26 Jul 2026 17:00:00 +0200

The three previous articles closed questions about the artefact: what contract the served model exposes, where its bytes come from and why you trust them. What remains is the one almost nobody asks until an auditor asks it: nothing proves that the process currently returning tokens on port 8000 is the inference engine and not something that slipped into the cluster and inherited its token. And a second one, worse: whether the machine it runs on is trustworthy, and against whom.

These are different problems with different projects. Workload identity is solved by SPIFFE/SPIRE, a CNCF graduated project since 2022. Environment isolation is covered by Kata Containers and Confidential Containers, the latter promoted to CNCF incubating on 22 July 2026, four days before this article was published. The first is cheap and almost always worth it. The second is expensive, more than the vendor’s press release suggests, and is only justified against a specific threat model.

TL;DR

User identity ≠ workload identity. OIDC says which person is behind a request; it does not say which process serves it. ServiceAccount tokens, static keys and gateway API keys are bearer credentials: whoever holds them, is them.
SPIFFE defines a URI identifier (spiffe://domain/path) and a verifiable document (SVID, X.509 or JWT) delivered by a Workload API that does not require the workload to hold any prior secret. SPIRE implements it with node and workload attestation; defaults: default_x509_svid_ttl 1 h, default_jwt_svid_ttl 5 min, ca_ttl 24 h.
Identity is only worth something if something enforces it: Istio/Envoy via SDS, Cilium with mutual authentication (with serious reservations) or an ext_authz with OPA in front of the gateway.
Three rungs of isolation: container with a shared kernel → Kata Containers (its own kernel and VM; v3.32.0 from June 2026) → Confidential Containers (hardware TEE, hypervisor outside the trust base, remote attestation with Trustee per RFC 9334).
The overhead of confidential mode runs from ~0 % to ~28 % depending on what you measure: GPU-only in CC with large batches sits around 4-8 %, while the full stack (a CVM with Intel TDX plus an H100 in CC), measured independently in 2026, gives +21.8 % to +27.8 % on TTFT and −17.7 % to −21.1 % on throughput. Honest heuristic: reserve 15-25 % extra capacity.
MIG and vGPU are forbidden in CC mode according to NVIDIA’s reference architecture, and every GPU on a node goes in the same mode.
Closing criterion: the order is contract → digest → signature verified at admission → workload identity → TEE.

The analogy: the biosafety laboratory

A laboratory handling pathogens solves two problems much like ours, separately.

The first is who gets in. The company badge is not enough: at the airlock there is a specific credential, short-lived and automatically renewed (if lost, it expires on its own within an hour), issued only after independently checking that the applicant is where they say they are and is who they say they are. That credential is the SVID; the prior check is attestation.

The second is which room you work in, and here there are containment levels. On the open bench the air is shared and anything aerosolised affects everyone: the normal container, with a kernel shared by all the neighbours. One rung up, the biological safety cabinet, with a physical barrier and its own airflow: Kata Containers. At the very top, the maximum containment laboratory, where maintenance staff change filters without ever seeing the sample: the TEE, where the operator maintains the machine but cannot read its memory. And before opening the airlock somebody verifies that the integrity of the suit and of the room is as expected: remote attestation with conditional secret release.

The uncomfortable lesson: the maximum containment level is a cost decision before it is one more degree of security. Nobody builds a BSL-4 to grow yeast. The useful question is who it isolates from, not how much.

Part A — Workload identity

Six services talking to each other and none of them knows who the other is

In a serious inference cluster, like the seven layers one, the gateway talks to the serving engine, the engine queries the vector database, the trace collector receives spans from everyone and the agents with MCP invoke tools that in turn call the gateway. Many east-west connections and, in most deployments, none of them authenticated.

User identity says which person is behind the request and is resolved with OIDC, as when putting Keycloak in front of MCP. Workload identity says which process issues it: there is no human on the other side, no browser, no consent, and the process is born and dies in seconds. Its three usual substitutes fail for different reasons:

Mechanism	Where it fails
Projected `ServiceAccount` token	SA granularity, not workload granularity: two pods with the same SA are indistinguishable. It does not cross the cluster boundary and does not federate. It is a bearer token on the filesystem
Static key	It does not rotate, it gets shared over Slack, it does not distinguish issuer from bearer; once leaked, it is valid until somebody notices
Gateway API key	It authenticates the client, not the process. LiteLLM’s virtual keys are for quota and budget, not for proving identity

There is no cryptographic link between credential and process: copy the credential, copy the identity. In a multi-tenant cluster like the H100 cluster one, real isolation then depends on network topology and not on identity.

SPIFFE: the standard

SPIFFE and its implementation SPIRE graduated together in the CNCF on 20 September 2022. As of July 2026 they are settled infrastructure, not a bet.

The SPIFFE ID is a URI of the form spiffe://<trust-domain>/<workload-identifier>, for example spiffe://inferencia.example/ns/inferencia/sa/vllm: a name, not a credential. The trust domain is the root of trust (organisation, environment or site), and the guidance recommends separating workloads from different sites or security environments into different domains.

The SVID is the credential that proves that ID. The X509-SVID carries the SPIFFE ID in the URI-type SAN, not in the CN, which the specification discourages as a source of identity, and it is the preferred format. The JWT-SVID exists for when there are L7 proxies terminating TLS in the middle, with the replay risk the documentation warns about:

Certificate:
Issuer: C=ES, O=SPIFFE
Validity
Not Before: Jul 26 08:00:00 2026 GMT
Not After : Jul 26 09:00:00 2026 GMT
Subject: C=ES, O=SPIRE, CN=vllm.inferencia
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
URI:spiffe://inferencia.example/ns/inferencia/sa/vllm

One hour of validity, and what authenticates is the SAN URI: the CN is decorative.

The Workload API is the elegant piece: a local UNIX socket from which the workload obtains its SVID, its private key and the trust bundle, without presenting any secret. The documentation is explicit: “the Workload API does not require that a calling workload have any knowledge of its own identity, or possess any authentication token”. That solves the cold start of all identity cryptography: the secret you would need in order to obtain the first secret.

SPIRE: how that document is issued

SPIRE has a server (the authority that signs) and one agent per node (which exposes the Workload API), and it chains two checks.

Node attestation verifies that the agent runs where it says it does. In Kubernetes the reference attestor is k8s_psat, which validates a projected token with an audience against the API server; on bare metal there are TPM-based attestors. A note of honesty: SPIRE v1.15.1, from 28 May 2026, is a security patch over an incorrect PKCS7 validation in the azure_imds attestor that allowed attested documents to be forged and a virtual machine to be impersonated.

Workload attestation answers the opening question. The k8s attestor receives the PID of the process that opens the socket, infers the pod from its cgroup membership and queries the kubelet for the metadata. From there come the selectors: namespace, service account, pod name and UID, labels, owner, node and, above all, container and image by tag or by digest. From SPIRE v1.15.0 (19 May 2026) Sigstore support stopped being experimental, enabling selectors by signature verification status, certificate subject and issuer, and transparency log. That closes the series’ circle: you can require that the engine’s SVID go only to the container whose image carries a valid cosign signature from the expected issuer, which is what was verified in article 3/4. Provenance goes from being a one-off admission check to a precondition of identity at runtime.

spire-server entry create \
 -parentID spiffe://inferencia.example/spire/agent/k8s_psat/prod/nodo-gpu-a \
 -spiffeID spiffe://inferencia.example/ns/inferencia/sa/vllm \
 -selector k8s:ns:inferencia \
 -selector k8s:sa:vllm \
 -selector k8s:container-name:vllm \
 -selector k8s:container-image:vllm/vllm-openai@sha256:aa11bb22cc33 \
 -x509SVIDTTL 3600 \
 -federatesWith spiffe://sede-b.example

The selectors are conjunctive: a pod in another namespace, with another SA or with another image does not obtain that SVID even if it shares the node.

The default TTLs have three consequences. Rotation is continuous, not a one-off event: code that loads the certificate once at startup will break exactly one hour later. Server downtime has a clock: twenty minutes are invisible, two hours bring down the cluster’s authenticated communications, and the scaling guide admits that “a single SPIRE Server instance represents a single point of failure”, with the datastore as the bottleneck and indicative figures running from two replicas of 1 CPU and 1 GB for 10 agents to eight of 16 CPU and 16 GB for 5,000 agents and 10,000 workloads. And tag-based selectors are fragile: the runtime may report one tag or another depending on the node and the moment, so the correct selector is the digest, as in article 2/4.

Federation between trust domains

Two sites with independent clusters, or an external partner exposing a reranker, should not share an authority. SPIFFE federation makes each domain publish a bundle endpoint with its public trust material and the other poll it periodically: the https_web profile authenticates it with web PKI and https_spiffe with an X509-SVID from the domain itself, enabling automatic rotation and revocation of the root. The relationship is one-directional, bundles from different domains must never be merged and refresh is by polling with a default suggestion of five minutes. In practice, site A’s gateway accepts requests from site B’s engine without sharing a CA or an identity database, and cutting the relationship means deleting a bundle, not revoking certificates.

From identity to effective authorisation

An SVID blocks nothing by itself; somebody has to compare the presented ID against a policy.

Istio + SPIRE is the best-worn integration: Istio detects a UNIX socket implementing Envoy’s SDS API and the proxy obtains its identities from there instead of from istiod, mounted with the SPIFFE CSI Driver (recommended over hostMount). Two conditions break deployments: SPIRE’s trust domain and Istio’s must match exactly, and SPIRE only issues to previously registered workloads, including Istio’s own components. With that, policy is written in terms of identity and not IP:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: vllm-solo-desde-gateway
 namespace: inferencia
spec:
 selector:
 matchLabels:
 app: vllm
 action: ALLOW
 rules:
 - from:
 - source:
 principals: ["inferencia.example/ns/inferencia/sa/gateway"]
 to:
 - operation:
 methods: ["POST"]
 paths: ["/v1/chat/completions", "/v1/completions"]

It is a layer on top of the NetworkPolicy from layered hardening: there we closed communication pairs by topology; here we also require who calls and on what path.

Cilium offers mutual authentication backed by SPIRE, tempting if you already use eBPF for the data network. It deserves criticism: it is still marked beta in the stable documentation, and The New Stack published a reasoned critique arguing that the design does not preserve mTLS properties over the life of the connection (it uses the handshake only to authenticate and discards the session keys) and that the identity model rests on eventually consistent per-node IP-identity caches, which can allow traffic that policy should deny.

The inference gateway and LiteLLM do not speak SPIFFE natively as of this article, and saying so is better than suggesting an integration that does not exist. The pattern that does work is the one the project itself documents: an Envoy in front that terminates mTLS with an X509-SVID or validates a JWT-SVID and delegates to OPA via ext_authz. The gateway carries on with its own job (quota, budget, routing), as we saw when choosing an OSS gateway and in the L7 router; workload identity is resolved one hop earlier.

AI agents and MCP: the missing piece

An agent invoking tools over MCP is a shared secret with legs: it is handed a static credential, the MCP server cannot know whether the caller is the legitimate agent or any process holding the same string, and the blast radius is the entire set of exposed tools. The threat model for agent isolation warned that a granted permission is a usable permission; with bearer credentials it is also transferable.

SPIFFE provides what is missing: the agent presents a short-lived SVID obtained through attestation and the MCP server authorises against the SPIFFE ID. In 2026 this stopped being theory: there is work at the IETF on dynamic OAuth client registration based on trusted SPIFFE issuers, and in June 2026 Google published a SPIFFE-based Agent Identity in its IAM. None of it is fully standardised.

Two honest caveats. SPIFFE answers “who”, not “why”: to know whether the action is the one the user asked for you need traceability (MCP with OTel) and runtime detection with Tetragon. And both identities must be composed: the agent proves its own with an SVID and propagates the delegating user’s in the token, authorising on the pair. Collapsing them into one is how you end up with agents able to do, in the system’s name, things no user could.

Part B — Isolation of the execution environment

### Kata Containers: the virtual machine sandbox

Kata replaces runc with a runtime that boots a lightweight virtual machine per pod, with its own kernel and a kata-agent inside; for Kubernetes it is transparent via RuntimeClass. As of July 2026 the stable branch is 3.32.x (3.32.0 is from June 2026, with Rust 1.94, Go 1.25.11, QEMU 11.0.1, guest kernel 6.18.35 and containerd 2.3), and in April 2026 the 4.0.0 preview was published, making the Rust runtime (runtime-rs) the default and leaving the Go one deprecated until 5.0.0, because of memory footprint.

For GPUs the route is VFIO passthrough. The NVIDIA GPU Operator automates it with sandboxWorkloads.enabled=true and sandboxWorkloads.mode=kata, installing the VFIO Manager, the Sandbox Device Plugin, the Confidential Computing Manager and the Kata Manager. It requires virtualisation and ACS in the BIOS, IOMMU, kata-deploy 3.29.0 or later, containerd (not CRI-O), the KubeletPodResourcesGet feature gate (on by default from Kubernetes 1.34) and removing the NVIDIA drivers from the host: the GPU is handed over whole to the VM and the guest manages it.

apiVersion: v1
kind: Pod
metadata:
 name: vllm-kata
 namespace: inferencia
spec:
 runtimeClassName: kata-qemu-nvidia-gpu
 containers:
 - name: vllm
 image: vllm/vllm-openai@sha256:aa11bb22cc33
 resources:
 limits:
 nvidia.com/pgpu: "1"

Two measurable costs. Memory: the VM reserves its own and you have to declare it as overhead in the RuntimeClass so the scheduler does not overcommit the node. And startup: according to the SESAME'24 study on serverless confidential containers, cold start goes from around 6 s with runc to around 7 s with Kata, and warm start from 1 s to 2 s. Acceptable for an engine that takes minutes to load weights; unacceptable for ephemeral functions.

Confidential Containers: the TEE

Confidential Containers (CoCo) takes the hypervisor and the platform operator out of the trust base. The CNCF promoted it to incubating on 22 July 2026, with more than 150 active contributors, 26 repositories and over 1,200 merged pull requests, and with Microsoft Azure, Intel, AMD, IBM, NVIDIA, Alibaba and Red Hat behind it. A serious project, but incubating means it is not graduated and that its integration surface changes between versions.

The components are the CoCo pods (unmodified containers run in a TEE via Kata) and Trustee: KBS (Key Broker Service, which releases secrets conditionally), AS (Attestation Service, which validates the hardware evidence) and RVPS (Reference Value Provider Service, which holds the reference values). Inside the guest, the Attestation Agent collects the evidence and the Confidential Data Hub consumes the secrets.

The flow is a clean instance of the RATS architecture from RFC 9334: the agent inside the TEE is the attester, the Attestation Service the verifier and the Confidential Data Hub the relying party. The evidence is a hardware-signed report with the boot measurements; the AS validates it against the RVPS values; and only if it matches does the KBS release the key that decrypts the image layers or the weights. Without valid attestation there is no key, and without a key there is no model: that is what stops the operator extracting it.

A mapping warning: the confidential-containers/operator repository was archived in February 2026 and deployment moved to the Helm charts and the trustee-operator. As of July 2026 the latest chart release is v0.21.0, aligned with kata-deploy 3.31.0, and Trustee v0.20.0, with TLS 1.3 and post-quantum cryptography, external plugins for the KBS and multi-GPU support via Intel Trust Authority and NVIDIA NVSwitch.

apiVersion: confidentialcontainers.org/v1alpha1
kind: TrusteeConfig
metadata:
 name: trusteeconfig
 namespace: operators
spec:
 profileType: Restrictive
 httpsSpec:
 tlsSecretName: trustee-tls-cert

The Permissive profile exists for development and should not leave it: it is the mode in which attestation does not block.

The hardware: CPU and GPU

On the CPU side the two relevant TEEs are AMD SEV-SNP and Intel TDX; NVIDIA’s reference architecture sets EPYC Milan/Genoa and Xeon Emerald/Granite Rapids as the baseline. Host support is no longer the hard part.

On the GPU side, the technical description of the design of the first confidential GPUs explains what CC mode on an H100 buys you. Memory is split into a Compute Protected Region that hardware firewalls stop both the CPU over PCIe and other GPUs over NVLink from accessing. Everything crossing the CPU-GPU boundary passes through bounce buffers encrypted with AES-GCM-256, and command buffers and CUDA kernels are encrypted and signed before crossing the bus. There is a chain of trust from GPU boot with a signed attestation report, and only NVIDIA-signed firmware runs in CC mode, validated against NRAS or locally in isolated environments. And the performance counters are disabled in hardware as a side-channel mitigation: that is the telemetry much of observability with DCGM depends on.

What it does not protect: the package’s HBM memory is not encrypted (it is considered resistant to interposers, which is a claim about physical difficulty and not a cryptographic guarantee); nothing against denial of service, because the operator who cannot read you can still switch you off; nothing against timing or access-pattern side channels; and nothing against a bug in your own code inside the enclave or prompt injection, which remains the job of guardrails.

On Blackwell, NVIDIA announces (2 July 2026) NVLink encryption that extends confidential computing to up to 8 GPUs, non-existent on Hopper and a condition for serving a large model with tensor parallelism. The reference architecture lists H100, H200, RTX Pro 6000 Blackwell Server Edition and B200 in single-GPU passthrough, and H100/H200 in PPCIe mode and B200 for multi-GPU.

The overhead numbers, with source and with judgement

Here it pays to distrust everybody, papers included. The published spread runs from 0 % to 28 % because the studies do not measure the same thing.

Source (date)	What it measures	Result	Nature
NVIDIA, Blackwell blog (Jul. 2026)	B200 with CC enabled	−1.0 % to −7.5 % throughput; per-token latency under 8 %; “up to 98 % of native performance”	Vendor claim
Corvex, HGX B200 with TDX (2026)	“Verified” deployment, encrypted NVSwitch	“Near-native performance”, with no figures of its own	Commercial claim
arXiv 2409.03992 (2024)	H100 GPU only in CC, Llama-3.1 8B/70B	−0.36 % to 6.85 % throughput; TTFT up to +19 %; overhead → 0 as the model grows	Independent, partial
arXiv 2509.18886 (Sep. 2025)	CPU TEE (TDX/SGX) and GPU TEE (H100), Llama2 7B/13B/70B	CPU TEE: under 10 % throughput and 20 % latency. GPU TEE: 4-8 %	Independent
arXiv 2607.19353 (May 2026)	Full stack: H100 in CC inside a CVM with TDX, Mistral-7B and Qwen3-30B-A3B under load	TTFT +21.8 % and +27.8 %; throughput −17.7 % and −21.1 %; closed-loop, 11.5-20.2 %	Independent, complete

The reconciliation: the penalty is in the data path, not in the GPU’s compute. NVIDIA itself quantifies the bottleneck: the effective bandwidth of the CPU-GPU interconnect in CC mode is limited by the CPU’s encryption throughput, “around 4 GB/s”. From there, three rules. The higher the compute/I-O ratio, the lower the overhead: a 70B with large batches and long sequences amortises the toll almost entirely, and a 7B with small batches and short prompts pays it in full. Adding the CPU TEE brings its own cost, and that is where most of the difference between 4-8 % and 20 % lies. And model loading and cold start are the worst cases, with tens of gigabytes crossing an encrypted bus at 4 GB/s, which makes everything discussed in from disk to HBM and in speeding up cold start more important.

The recommendation from the authors of the 2026 study, reserving between 15 % and 25 % extra capacity, is the figure I would take to a capacity planning exercise, and not the vendor’s “98 % of native performance”. Both can be true at once; only one is prudent for sizing.

When each rung is worth it

Scenario	Kata	CoCo (TEE)	Reason
Tenants that do not trust each other	Yes	Depends	Kata removes the shared kernel, a real escape vector
Unaudited third-party code or models	Yes	Optional	A shared kernel is too much surface for someone else’s code
Model intellectual property against the infrastructure operator	Not enough	Yes	The canonical case: third-party hosting, cloud, a partner providing the iron
Regulated third-party data on infrastructure you do not control	Not enough	Yes	See defence and healthcare
Sovereign on-premise, one tenant, trusted operator	Useful	Expensive over-engineering	It protects you from an adversary you do not have
Low latency with small batches	Yes	Bad idea	The worst point on the bus-encryption cost/benefit curve

The nuance that saves the most money: the two axes are independent. Kata without a TEE is cheap and delivers most of the isolation between tenants; CoCo only adds something if your threat model includes the operator. If you build a sovereign factory with your own iron and your own staff and you answer “no” to that question, the TEE is a cost with no return.

Operational traps

MIG and vGPU against confidential mode. NVIDIA’s GA 1.0.0 reference architecture is blunt: MIG and vGPU are forbidden in CC mode, and mixed-mode nodes are not supported; every GPU on a host goes in CC or none does. The forums show the typical contradiction of a moving area: in February 2026 a moderator states that MIG+CC is not supported and in April a user cites the MIG page suggesting that Hopper and Blackwell already allow it. As of this article the reference architecture overrides the product page: if your multi-tenancy rested on partitioning the GPU with MIG, enabling CC sends you back to “one whole GPU per tenant”.

Startup grows non-linearly. SESAME'24 breaks down where: SEV memory provisioning adds around one second per 2 GB of guest (SEV pins every page in advance, and an inference VM allocates a lot), the OVMF firmware about three seconds and layer decryption another three to five. Scaling from 0 to 16 instances goes from 16 s with runc to 190 s: irrelevant for a stable Deployment, a design change for aggressive autoscaling with KEDA.

Version coupling is brutal. Kata, kata-deploy, GPU Operator, containerd, QEMU with its own patches, guest kernel, NVIDIA driver and Trustee are a matrix that has to be treated as a unit: the reference architecture fixes Kubernetes 1.32 or later, Kata 3.29, GPU Operator 26.3.1 or later, containerd 2.2.2 or later and QEMU 10.1. Add it to the weight the stack’s operators already carry.

Attestation fails after a firmware update, and that is normal. The RVPS reference values describe a specific boot state. When you patch the SEV-SNP firmware, the microcode or the GPU firmware, the TCB version changes, the evidence stops matching and the KBS does not release keys, so the pods do not start. Since these updates usually respond to a security bulletin, the typical scenario is “we applied the critical patch on a Friday and on Saturday inference would not start”. The correct procedure reverses the order: update the reference values first with an overlap window, and patch the iron afterwards.

The attestation service is a single point of failure by design. If the KBS does not answer, no new confidential pod gets its key; those already started survive, but any rescheduling or scaling does not. The same dependency as the SPIRE server and the same treatment: real high availability, its own alert and rehearsed degradation.

Closing the series: the complete chain

Article 1/4 set the contract: what the endpoint exposes and what control plane governs it reproducibly. In 2/4 the subject was the provenance of the bytes: which registry they come from, by immutable digest and not by a moving tag. 3/4 added the cryptographic proof of what they are, with signatures, build attestations and an AIBOM verified at admission. This 4/4 closes with who runs them and where: an SVID that proves the serving process’s identity and an isolation rung chosen for the adversary you actually have. If one of the four fails, the other three are worth less than they look: an impeccably signed model served by an unidentified process on a machine a third party can dump is still a problem.

With a limited budget, the order follows cost per unit of risk removed. First, digests everywhere: almost free, and it removes a whole family of substitution attacks. Second, signature verified at admission, low cost and the first thing an auditor asks to see. Third, a declarative contract and control plane, which make everything else auditable. Fourth, workload identity with SPIFFE/SPIRE: expensive to operate, but it is the only thing that turns east-west traffic into something authorisable, and it becomes mandatory as soon as agents with MCP arrive. And last, the TEE, only if the infrastructure operator is in your threat model; Kata without a TEE can jump ahead to fourth place if there are tenants that do not trust each other.

Mapping the series to ENS, ISO/IEC 42001 and the EU AI Act

The detail is in technical controls ENS × ISO 42001 × EU AI Act; this table is the summary by link.

Link	ENS measure (RD 311/2022)	ISO/IEC 42001 (Annex A)	EU AI Act
1/4 Contract and control plane	`op.exp.2`; `op.mon.1`	A.6 life cycle	Art. 12 (record-keeping); Art. 13 (transparency)
2/4 Registry and distribution by digest	`op.ext.3`	A.10 third parties	Art. 11 and Annex IV (technical documentation)
3/4 Signature, attestations and AIBOM	`op.exp.6`; `op.ext.3`	A.6.2 and A.7 data and traceability	Art. 15(5), data and model poisoning
4/4-A Workload identity	`op.acc.1/2/5`; `op.exp.11`; `mp.com.2-3`	A.9 use of the AI system	Art. 15(5), unauthorised third parties
4/4-B Isolation and TEE	`mp.info.3`; `op.exp.2`; `mp.com.4`	A.6 life cycle controls	Art. 15(4) robustness; Art. 15(5)

Art. 15(5) carries the most weight: it requires high-risk systems to be “resilient against attempts by unauthorised third parties to alter their use, outputs or performance”, and it enumerates data and model poisoning and confidentiality attacks. Links 3 and 4 are its technical implementation. For the management framework, see ISO/IEC 42001 as an AIMS and the AI Act mapping.

For an inference factory

Start workload identity with the gateway↔engine pair and with the agents, not with the whole cluster. Registering all forty workloads on day one is the most reliable way to abandon the project: register two entries, set up the Istio policy that requires the gateway’s principal and live with it for a couple of weeks to learn what breaks when the certificate rotates. Then the agents with MCP, where static credentials are the entire threat model.

Separate the Kata decision from the TEE decision, and take them in that order. Kata is a RuntimeClass change with a second of startup and some memory. CoCo is a redesign: you lose MIG and the GPU’s performance counters, you gain ten seconds or more of startup, you pay 15-25 % of capacity and you add two dependencies whose failure prevents workloads from starting. Evaluate the second with a written threat model, not with a slide.

If you go to CoCo, rehearse patch day before you need it. The failure that will take your inference down will not be an attack, it will be a firmware update that misaligns the RVPS reference values. Write the runbook (update the RVPS with overlap, patch, retire the old value), rehearse it on one node and monitor the attestation success rate as a first-class metric, just as for SPIRE: alert on attestation failures before they turn into 401s at the gateway.

That closes the series: from the contract to the artefact, from the artefact to its proof, and from the proof to the process and the machine. What lies below is no longer a chain of trust but daily operation: seeing what each process does with Tetragon and measuring whether what is served is still good with evals. Trust is established once; vigilance is continuous.

Sources

CNCF, SPIFFE and SPIRE Projects Graduate from CNCF Incubator (20 sep. 2022) — https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/
SPIFFE, SPIFFE Concepts (SPIFFE ID, trust domain, SVID, Workload API) — https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/
SPIFFE, SPIFFE Federation specification — https://spiffe.io/docs/latest/spiffe-specs/spiffe_federation/
SPIFFE, Scaling SPIRE (dimensionamiento y punto único de fallo) — https://spiffe.io/docs/latest/planning/scaling_spire/
spiffe/spire, Releases (v1.15.0 de 19 may. 2026; v1.15.1 de 28 may. 2026, parche de azure_imds) — https://github.com/spiffe/spire/releases
spiffe/spire, Kubernetes Workload Attestor plugin (selectores, kubelet, cgroups, Sigstore) — https://github.com/spiffe/spire/blob/main/doc/plugin_agent_workloadattestor_k8s.md
spiffe/spire, SPIRE Server configuration reference (default_x509_svid_ttl, default_jwt_svid_ttl, ca_ttl) — https://github.com/spiffe/spire/blob/main/doc/spire_server.md
Istio, SPIRE integration (Envoy SDS, SPIFFE CSI Driver, trust domain) — https://istio.io/latest/docs/ops/integrations/spire/
SPIFFE, OPA Authorization with Envoy and JWT-SVIDs — https://spiffe.io/docs/latest/microservices/envoy-jwt-opa/readme/
The New Stack, How Cilium’s Mutual Authentication Can Compromise Security — https://thenewstack.io/how-ciliums-mutual-authentication-can-compromise-security/
Riptides, Bringing SPIFFE to OAuth for MCP — https://riptides.io/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads/
CNCF, Confidential Containers becomes a CNCF incubating project (22 jul. 2026) — https://www.cncf.io/blog/2026/07/22/confidential-containers-becomes-a-cncf-incubating-project/
Confidential Containers, Attestation with Trustee (KBS, AS, RVPS, CDH) — https://confidentialcontainers.org/docs/attestation/
Confidential Containers, Deploy Trustee in Kubernetes (11 feb. 2026) — https://confidentialcontainers.org/blog/2026/02/11/deploy-trustee-in-kubernetes/
confidential-containers/trustee, Releases (v0.20.0: TLS 1.3, PQC, multi-GPU ITA, NVSwitch) — https://github.com/confidential-containers/trustee/releases
IETF, RFC 9334 — Remote ATtestation procedureS (RATS) Architecture — https://www.rfc-editor.org/info/rfc9334/
Kata Containers, Kata Containers 4.0.0 Preview (28 abr. 2026, runtime-rs por defecto) — https://katacontainers.io/blog/release-4-0-0-preview/
kata-containers, Release 3.32.0 — https://github.com/kata-containers/kata-containers/releases/tag/3.32.0
NVIDIA, Deploy with Kata Containers — GPU Operator — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html
NVIDIA, Deploying Proprietary Models Securely with Confidential Computing on Self-Hosted Kubernetes (GA 1.0.0; MIG y vGPU prohibidos en CC) — https://docs.nvidia.com/enterprise-reference-architectures/deploying-proprietary-models-confidential-compute-self-hosted-kubernetes/latest/reference-implementations.html
CACM, Creating the First Confidential GPUs (CPR, bounce buffers, contadores deshabilitados, HBM no cifrada) — https://cacm.acm.org/practice/creating-the-first-confidential-gpus/
NVIDIA, Confidential Computing on H100 GPUs for Secure and Trustworthy AI (límite de ~4 GB/s, NRAS) — https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/
NVIDIA, Hardware-Rooted AI Security That Won’t Slow You Down (2 jul. 2026; Blackwell, NVLink cifrado hasta 8 GPU) — https://developer.nvidia.com/blog/hardware-rooted-ai-security-that-wont-slow-you-down
Corvex, Confidential Computing Meets NVIDIA HGX B200 (claim comercial) — https://www.corvex.ai/blog/confidential-computing-meets-nvidia-hgxtm-b200-secure-ai-without-the-performance-trade-off
arXiv 2409.03992, Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study — https://arxiv.org/html/2409.03992v1
arXiv 2509.18886, Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs (23 sep. 2025) — https://arxiv.org/abs/2509.18886
arXiv 2607.19353, Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX (may. 2026; heurística del 15-25 %) — https://arxiv.org/html/2607.19353
Segarra et al., Serverless Confidential Containers: Challenges and Opportunities (SESAME'24) — https://carlossegarra.com/assets/papers/sesame24-serverlesscoco.pdf
EU Artificial Intelligence Act, Article 15: Accuracy, Robustness and Cybersecurity — https://artificialintelligenceact.eu/article/15/
EU Artificial Intelligence Act, Article 12: Record-Keeping — https://artificialintelligenceact.eu/article/12/

Volcano and Kueue: gang scheduling, queues and GPU quotas for distributed workloads on Kubernetes

Tue, 16 Jun 2026 13:00:00 +0200

TL;DR

Volcano (volcano-sh, CNCF incubating) is a complete batch scheduler that replaces or complements the kube-scheduler: it places pods with gang semantics (all-or-nothing via PodGroup/minMember), manages queues with priority, DRF fair-share and preemption between queues, and understands network and NUMA topology.

Kueue (kubernetes-sigs/kueue) is a queue and quota manager at the Job level: it does NOT place pods (it delegates to the kube-scheduler or to Volcano), but it decides when a workload can be admitted according to available quota (ClusterQueue/LocalQueue/Cohort), with fair sharing, borrowing between teams and preemption by priority. It natively integrates Job, JobSet, RayJob, all the Kubeflow operators and more.

The winning combination in production for multi-tenant GPU workloads is: Kueue for quota and queues + Volcano (or the sig-scheduler coscheduling plugin) for the job’s gang.

The analogy

Imagine a club with limited capacity and a dance floor inside.

Kueue is the doorman and the booking manager: it checks whether the team’s quota (its reserved capacity) allows the group in, applies the fair waiting list, borrows capacity from other teams if any are idle, and takes the space back when the owner needs it. But the doorman does not decide where each person sits inside the venue.

Volcano is the head waiter: once the group has permission to come in, he decides which tables they sit at, makes sure the whole group sits down at once or nobody comes in (gang), picks the tables according to topology (who needs to talk to whom) and throws out lower-priority groups to make room if necessary.

Without a doorman (Kueue), the head waiter does not know how many groups he can take at once or whether a team is exceeding its capacity. Without a head waiter (Volcano), the doorman lets the group in but its members scatter across the available tables on their own, and the party of 8 that needs to sit together never manages it.

The problem neither solves by default: the kube-scheduler

Kubernetes’ kube-scheduler is a pod scheduler, not a job scheduler. It assigns pods one by one to the most suitable node according to available resources and affinity constraints. For a distributed training workload that needs, say, 8 pods with 4 GPUs each (32 GPUs in total across 8 nodes of 4×H100), the standard scheduler does the following:

It looks for a node with 4 available GPUs. It finds one. It schedules pod 1.
It looks for another node with 4 GPUs. It finds one. It schedules pod 2.
It carries on until it reaches pod 6 and it turns out there are no longer any nodes with 4 free GPUs: the cluster has exactly 32 GPUs and other workloads are using some of them.
Pods 1–5 are Running. Pods 6–8 are Pending.
Pods 1–5 can do nothing without the others: a distributed PyTorch job needs all the workers to start before the torchrun process can begin. It waits with the resources occupied. Deadlock.

This is not a bug, it is the design: the kube-scheduler has no concept of “schedule this group of pods only if you can schedule all of them”. As a result:

The resources of pods 1–5 are locked up without producing work.
Other jobs that could run with the partial resources also wait.
If several jobs are in this situation, the cluster can end up with fragmented resources, no job running and everyone in circular deadlock.

On top of that, the kube-scheduler has no notion of:

Queues per team or project with relative priority.
Quotas of resources per team with the ability to borrow from idle teams.
Fair-share: if team A has spent weeks using 80 % of the cluster, it should wait longer than team B, which has spent weeks idle.
Inter-queue preemption: evicting a lower-priority job from another team to make room for this team’s urgent job.

Solving any of these problems requires adding a layer on top of the scheduler. Volcano and Kueue are the two dominant OSS solutions in 2026, with complementary architectural approaches.

Volcano: the batch scheduler

What it is and what it replaces

Volcano (volcano-sh) is a Kubernetes-native batch scheduler accepted by the CNCF as its first and only official container batch scheduling project (volcano.sh/en/docs). At version v1.15.x as of June 2026, with CNCF incubating status.

Volcano is not an addon to the kube-scheduler: it is an alternative scheduler (or a complementary one, depending on the configuration) that places pods. It is installed as a deployment, and jobs that want to benefit from its capabilities must use the volcano scheduler class in their pod spec (schedulerName: volcano) or use the VolcanoJob CRD.

Volcano’s central value proposition is that it treats groups of pods as atomic scheduling units, not individual pods. That is what makes it possible to solve the deadlock described above.

Gang scheduling via PodGroup

Volcano’s central mechanism is the PodGroup: a CRD that groups a job’s pods and defines how many must be schedulable before Volcano starts any of them.

# PodGroup for a distributed PyTorch training job: 8 workers, minimum 8
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: pytorch-train-pg
 namespace: ml-training
spec:
 minMember: 8 # all-or-nothing: if there is no room for 8, none starts
 minResources:
 nvidia.com/gpu: "32" # 8 pods × 4 GPUs = 32 GPUs minimum in the cluster
 queue: team-datos  # which Volcano queue this job is assigned to
 priorityClassName: high-priority

The minMember parameter implements all-or-nothing semantics: Volcano only assigns nodes to the group’s pods when it can assign at least minMember pods simultaneously. If the cluster does not have capacity for 8 GPU pods at this moment, no pod in the group moves out of Pending. Nothing is blocked, nothing is fragmented.

minMember can be lower than the job’s total pod count: this allows elastic gang scheduling, where the job can start with fewer workers and scale up, useful for jobs that tolerate a reduced worker count.

For a job’s pods to be associated with the PodGroup, they carry the annotation:

# Pod spec of the PyTorch worker
metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: pytorch-train-pg
spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: pytorch/pytorch:2.5-cuda12.4
 resources:
 limits:
 nvidia.com/gpu: "4"

Queue: queues with quotas and priority

Volcano introduces the Queue CRD to manage multiple tenants with independent quotas:

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4 # relative weight for fair-share between queues (proportion plugin)
 capability: # absolute ceiling of resources this queue can use
 nvidia.com/gpu: "16"
 guarantee: # guaranteed resource, never lent to other queues
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true # if true, others can reclaim the resources it lends when they need them

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "8"
 guarantee:
 resource:
 nvidia.com/gpu: "4"
 reclaimable: true

The weight field feeds the proportion plugin: queues compete for the cluster’s available resources in proportion to their weight. A cluster with 32 GPUs and two queues of weight 4 and 2 splits the GPUs in a 4:2 ratio (≈21 and 11 GPUs respectively) when both are saturated.

Scheduler plugins: DRF, binpack, topology-aware

Volcano implements its scheduling logic as a pipeline of actions and plugins:

Actions (what the scheduler does in each cycle):

enqueue: moves jobs from the waiting queue to schedulable when quota is available.
allocate: assigns nodes to schedulable pods.
preempt: evicts lower-priority pods to make room for higher-priority ones within the same queue.
reclaim: evicts pods from other queues that are using more than their guarantee to return resources to the owner.
backfill: fills idle resources with best-effort jobs that do not interfere with the rest.

Plugins relevant to GPU workloads:

Plugin	What it does
`gang`	Implements the PodGroup’s all-or-nothing semantics
`proportion`	Fair-share by queue weight (proportional quota)
`capacity`	Quotas with `guarantee`/`capability` and reclaim; a more expressive alternative to proportion
`drf`	Dominant Resource Fairness: multi-dimensional fair-share (CPU, memory, GPU)
`binpack`	Packs pods onto the fullest nodes; reduces GPU fragmentation
`priority`	Orders jobs by priority within the same queue
`nodeorder`	Node scoring according to multiple criteria (affinity, resources, spread)
`task-topology`	Affinity between pods of the same job (inter-GPU communication)
`numa-aware`	NUMA affinity: aligns pods with the node’s NUMA socket to reduce memory latency

Network and NUMA topology (v1.11+)

Volcano v1.11 (February 2025) introduced Network Topology Aware Scheduling as a first-class feature (CNCF blog, March 2025). Distributed training jobs in a datacenter with a hierarchical network structure (spine/leaf, blocks of nodes with NVSwitch) can declare topology constraints:

# VolcanoJob with a network topology constraint
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: llm-pretrain
 namespace: ml-training
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 plugins:
 ssh: []
 env: []
 svc: []
 networkTopology:
 mode: hard # hard: the job MUST satisfy the constraint
 highestTierAllowed: block  # pods cannot span beyond a network "block"
 tasks:
 - replicas: 8
 name: worker
 template:
 spec:
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The highestTierAllowed: block semantics instruct Volcano to place the 8 pods within the same network block (for example, all the nodes under the same access switch), minimising the inter-block traffic that degrades a distributed all-reduce.

NUMA awareness works in a similar way: with the numa-aware plugin, pods request a NUMA policy (single-numa-node, restricted, best-effort) and Volcano selects nodes where the requested CPU, memory and GPU resources are in the same NUMA domain, avoiding the remote memory access overhead (NUMA-crossing) that can degrade training throughput by 15-30 % on multi-socket nodes.

GPU virtualization in v1.11+

Volcano v1.11 also introduces support for dynamic MIG and vCUDA: instead of declaring nvidia.com/gpu: 1 for a whole GPU, workloads can declare nvidia.com/gpu-memory: 20Gi and Volcano (with the corresponding device plugin) dynamically provisions the MIG instance or the vCUDA partition. This is [project marketing with no independent benchmarks published as of June 2026], but the feature’s architecture is documented in the code.

What it replaces or adds to the default scheduler

Capability	kube-scheduler	Volcano
Placing pods on nodes	Yes	Yes (it replaces it for workloads marked with `schedulerName: volcano`)
Gang scheduling (all-or-nothing)	No	Yes (PodGroup + minMember)
Queues with priority	No	Yes (Queue CRD)
Inter-queue fair-share	No	Yes (DRF, proportion, capacity plugins)
Inter-queue preemption	No	Yes (reclaim action)
Topology-aware (NUMA, network)	Partial (node affinity)	Yes (dedicated plugins)
Elastic gang (minMember < total)	No	Yes
Best-effort backfill	No	Yes
Framework integrations	Partial	MPI, PyTorch, Ray, TensorFlow, Spark, Flink, Horovod

Kueue: the queue and quota manager

What it is and what it does NOT do

Kueue (kubernetes-sigs/kueue) is a Kubernetes-native system that manages quotas and how jobs consume them (kueue.sigs.k8s.io). Kueue decides when a job must wait, when it should be admitted (pods can be created) and when it should be evicted (active pods must be deleted).

Kueue’s central design principle is explicit in its documentation: avoid duplicating mature functionality of Kubernetes components. Autoscaling is the cluster-autoscaler’s responsibility. Pod-to-node scheduling is the kube-scheduler’s responsibility. Job lifecycle management is the kube-controller-manager’s responsibility. Kueue replaces none of them: it sits on top as a layer of admission control and quota management at the Job level.

This is the fundamental distinction: Kueue does not place pods on nodes. When Kueue admits a workload, it simply allows the corresponding job controller to create the pods, and those pods are scheduled by the kube-scheduler (or by Volcano, if it is configured as the scheduler).

The four core objects

ResourceFlavor: maps abstract resources to concrete groups of physical nodes.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 node-pool: gpu-training
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"

ClusterQueue: defines the resource quota per flavor for a tenant. A cluster-scoped object.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform  # cohort it belongs to (can lend/borrow)
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16 # GPUs guaranteed for this team
 borrowingLimit: 8 # can take up to 8 additional GPUs from the cohort
 lendingLimit: 8 # can lend up to 8 of its 16 nominal GPUs
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority  # reclaims lent quota by evicting lower-priority jobs
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority

LocalQueue: the namespace-scoped entry point for a team’s workloads. Jobs point to their LocalQueue; Kueue maps them to the corresponding ClusterQueue.

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos

Cohort: groups ClusterQueues that can lend quota to each other. It is not a standalone CRD; it is declared as a field in the ClusterQueue (spec.cohort: name). Kueue aggregates the available quota of all the ClusterQueues in the cohort and lets any of them borrow what the others are not using, respecting the borrowingLimit and lendingLimit.

Kueue implements Fair Sharing as the ordering policy for the queue of pending workloads (kueue.sigs.k8s.io/docs/concepts/fair_sharing): when several workloads compete for quota in the cohort, those belonging to ClusterQueues with higher accumulated historical usage have lower admission priority. This implements equitable sharing without permanently blocking any team.

Preemption in Kueue operates along two dimensions:

reclaimWithinCohort: the ClusterQueue that lends quota reclaims it by evicting workloads that are using it on loan, according to a priority policy.
withinClusterQueue: within the same ClusterQueue, lower-priority workloads are evicted to make way for higher-priority workloads from the same team.

Gang semantics in Kueue: all-or-nothing with ready Pods

Kueue provides gang admission at the Job level: it admits the complete workload only when all the necessary quota is available. If a RayJob needs 8 GPUs (1 head + 7 workers), Kueue does not admit the workload until 8 GPUs are available in the ClusterQueue (or borrowed from the cohort). waitForPodsReady with a timeout adds a second guarantee: if the created pods do not become Ready within the configured time, Kueue re-queues the workload and releases the quota (kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready).

This is gang semantics at the admission level, not at the pod placement level. It guarantees that the quota is available before creating the pods, but it does not guarantee that the kube-scheduler can place them all on concrete nodes at the same time. For that second guarantee you need Volcano or the coscheduling plugin.

Topology-Aware Scheduling (TAS)

Kueue v0.10+ introduces Topology-Aware Scheduling (kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling): it allows node topologies (blocks, subblocks, hosts) to be defined and workloads to request co-location levels. Kueue only admits the workload when it can satisfy the topological constraint, and it adds node selectors and taints at admission time so that the scheduler places the pods in the right topology.

TAS is configured with the Topology CRD:

apiVersion: kueue.x-k8s.io/v1beta1
kind: Topology
metadata:
 name: datacenter-topology
spec:
 levels:
 - nodeLabel: "topology.kubernetes.io/block"
 - nodeLabel: "topology.kubernetes.io/rack"
 - nodeLabel: "kubernetes.io/hostname"

And the ResourceFlavor references the topology:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 topologyName: datacenter-topology

Framework integrations

Kueue has built-in integration (no additional code) for the following workload types, activated with an annotation on the job:

metadata:
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # points to the team's LocalQueue

The supported workload types include: batch/Job, JobSet, RayJob, RayCluster, PytorchJob, TFJob, MPIJob, JAXJob, PaddleJob, XGBoostJob, TrainJob, AppWrapper, LeaderWorkerSet, Deployment, StatefulSet, and plain Pod/PodGroup.

For LLM workloads, the directly relevant cases:

RayJob (distributed training with Ray Train): Kueue admits the RayJob when there is quota for the whole Ray cluster (head + workers). Documented at docs.ray.io.
PyTorchJob (Kubeflow Training Operator): gang admission of the complete job.
JobSet: for coordinated multi-replica jobs (LWS, multi-step pipelines).
Deployment/StatefulSet: for continuous inference, allowing inference GPU quota to be managed the same way as training quota.

The key distinction: Volcano places, Kueue admits

This table sums up the fundamental architectural difference:

Dimension	Volcano	Kueue
Main role	Scheduler (places pods on nodes)	Admission controller + quota manager (decides when to create pods)
Gang scheduling	Yes, at the placement level (PodGroup/minMember)	Yes, at the admission level (all-or-nothing on quota)
Multi-tenant quotas	Yes (Queue with capability/guarantee)	Yes (ClusterQueue with nominalQuota/borrowingLimit)
Cohorts / borrowing	Partial (reclaimable between queues)	Yes (Cohort with explicit lendingLimit/borrowingLimit)
Fair sharing	Yes (DRF plugin)	Yes (Fair Sharing based on historical usage)
Preemption	Yes (preempt + reclaim actions)	Yes (reclaimWithinCohort, withinClusterQueue)
Topology/NUMA	Yes (dedicated plugins, network topology, NUMA-aware)	Yes (TAS, topology levels in ResourceFlavor)
Framework integrations	Volcano Job (MPI, PyTorch, Ray, TF, Spark, Flink)	Native: Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet
Who places the pods	Volcano	kube-scheduler (or Volcano if configured)
Installation footprint	Medium-high (its own scheduler, CRDs, webhook, metrics)	Light (controller, CRDs, webhook; does not replace the scheduler)
Maturity / status	CNCF incubating; v1.15 (June 2026); production at Huawei, Baidu, DiDi	kubernetes-sigs; API v1beta2; production at Google GKE, Red Hat OpenShift 4.20, Runway ML
Adoption curve	Steeper (requires changing schedulerName or using the VolcanoJob CRD)	Gentler (adds labels to existing jobs; does not change the scheduler)

How they coexist: the production pattern

In production, Kueue and Volcano are complementary, not mutually exclusive. The most common pattern in 2026 for multi-tenant GPU clusters is:

Kueue manages the global quota: which team can use how many GPUs, how much it can borrow, when a job goes into a queue versus being admitted.
Volcano does the gang scheduling at the pod level: once Kueue admits the job (the quota is available), Volcano places the pods, making sure they are all placed simultaneously on compatible nodes.

The integration is configured by specifying schedulerName: volcano in the pod specs of the workloads managed by Kueue. Kueue sees the Job/RayJob/PyTorchJob and manages its quota; when it admits it, the pods are created and Volcano places them with gang semantics. Volcano’s PodGroups are created automatically by the Volcano Job controller or by the Kubeflow Training Operator itself when it detects that the scheduler is Volcano.

# PyTorchJob managed by Kueue (quota) + Volcano (gang placement)
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: llm-finetune-70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the quota
 annotations:
 scheduling.volcano.sh/queue-name: team-datos  # Volcano uses its own Queue
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano  # Volcano does the placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The third way: the sig-scheduler coscheduling plugin

If you do not want to deploy a complete alternative scheduler but you need gang scheduling, there is the coscheduling plugin from kubernetes-sigs/scheduler-plugins. This plugin extends the kube-scheduler with a PodGroup mechanism similar to Volcano’s, implemented as a scheduling framework plugin (permit plugin). The advantage is that it does not replace the scheduler; the disadvantage is that it has less functionality than Volcano (no DRF, no Queue/fair-share, no network topology). It is the right option for simple clusters that only need gang and do not want Volcano’s operational complexity. Kueue can also work alongside this plugin.

Complete YAML examples

Volcano: Queue + PodGroup + VolcanoJob

# 1. Queue for the data team (16 nominal GPUs, ceiling at 24)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4
 capability:
 nvidia.com/gpu: "24"
 cpu: "192"
 memory: "768Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "16"
 reclaimable: true
---
# 2. Queue for the AI team (8 nominal GPUs, ceiling at 16)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "16"
 cpu: "64"
 memory: "256Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true
---
# 3. PodGroup: 70B fine-tuning job, 8 workers × 4 GPUs = 32 GPUs
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: finetune-70b-pg
 namespace: ns-datos
spec:
 minMember: 8
 minResources:
 nvidia.com/gpu: "32"
 queue: team-datos
 priorityClassName: training-high
---
# 4. VolcanoJob (wrapper that Volcano understands natively)
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: finetune-70b
 namespace: ns-datos
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 priorityClassName: training-high
 plugins:
 env: []
 svc: []
 policies:
 - event: PodEvicted
 action: RestartJob
 tasks:
 - replicas: 8
 name: worker
 policies:
 - event: TaskCompleted
 action: CompleteJob
 template:
 metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: finetune-70b-pg
 spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 command: ["torchrun", "--nproc_per_node=4", "--nnodes=8",
 "--node_rank=$(RANK)", "--master_addr=$(MASTER_ADDR)",
 "--master_port=23456", "train.py"]
 env:
 - name: NCCL_DEBUG
 value: "INFO"
 resources:
 requests:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 restartPolicy: Never

Kueue: ResourceFlavor + ClusterQueue + LocalQueue + annotated Job

# 1. ResourceFlavor: nodes with H100 SXM
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"
---
# 2. ClusterQueue for the data team: 16 nominal GPUs, can borrow 8
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16
 borrowingLimit: 8
 lendingLimit: 8
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority
---
# 3. ClusterQueue for the AI team: 8 nominal GPUs
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-ia
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 8
 borrowingLimit: 8
 lendingLimit: 4
 - name: "cpu"
 nominalQuota: "64"
 - name: "memory"
 nominalQuota: "256Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# 4. LocalQueue in the data team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos
---
# 5. LocalQueue in the AI team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-ia
 namespace: ns-ia
spec:
 clusterQueue: cq-team-ia
---
# 6. Batch inference RayJob managed by Kueue
apiVersion: ray.io/v1
kind: RayJob
metadata:
 name: batch-eval-llama70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the admission
spec:
 entrypoint: "python batch_eval.py --model /models/llama-70b"
 rayClusterSpec:
 headGroupSpec:
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-head
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 workerGroupSpecs:
 - replicas: 3
 minReplicas: 3 # gang: Kueue does not admit if there is no quota for 3 workers
 maxReplicas: 3
 groupName: gpu-worker
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-worker
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

Kueue + Volcano together: PyTorchJob with Kueue quota and Volcano gang placement

# PyTorchJob: Kueue controls the quota, Volcano does the gang placement
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: distributed-finetune
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos # Kueue: quota and admission
 annotations:
 # Volcano automatically creates the PodGroup when schedulerName=volcano
 scheduling.volcano.sh/queue-name: team-datos
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano # Volcano: gang placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

For LLM workloads: training, fine-tuning and batch inference

Distributed multi-GPU training and fine-tuning

Gang scheduling is essential for any distributed training job that uses NCCL all-reduce (PyTorch DDP, FSDP, DeepSpeed ZeRO). If a single worker in the group does not start, the torchrun coordinator waits indefinitely; with the standard kube-scheduler this scenario happens every time the cluster is under contention.

In a generic cluster of 4 nodes of 4×H100 SXM (16 GPUs in total), a fine-tuning job for a 70B model typically requires 8 GPUs in tensor-parallel 8 (TP=8) or 16 GPUs in TP=4 plus data-parallel 4. With Volcano, the PodGroup with minMember: 8 guarantees that either the 8 pods are placed at once or none blocks resources. With Kueue on top, the quota guarantees that the team does not exceed its 16 nominal GPUs and that other teams with available quota are not blocked by a waiting job.

The cross-link with inference capacity planning is direct: the model’s VRAM budget (weights plus KV-cache) determines the minimum TP and therefore the PodGroup’s minMember.

Batch inference and evaluations (evals)

Batch inference jobs, generating responses for an evaluation dataset, processing embeddings in bulk, offline re-ranking, are naturally parallel workloads that do not necessarily require strict gang scheduling (each request is independent), but they do benefit from quota and fair-share.

For these workloads, Kueue alone is enough: a batch/Job with multiple independent pods is managed with the ClusterQueue’s quota with no need for Volcano. If several teams are submitting evaluation jobs simultaneously, Kueue orders the admission by fair-share and priority, and with cohort borrowing the jobs of teams with free quota do not have to wait for the quota of busy teams.

The chargeback of these workloads connects directly with what is described in GPU chargeback and showback: the ClusterQueue’s nominalQuota is the expression of the GPU budget in Kubernetes, and OpenCost can attribute the cost per namespace or label for the monthly report.

Multi-tenant GPU quota and chargeback

The alignment between Kueue and the chargeback system is direct:

FinOps concept	Kueue mechanism
Guaranteed GPU budget	`nominalQuota` per ClusterQueue
Maximum spending ceiling	`nominalQuota + borrowingLimit`
Lending idle capacity	Cohort + `lendingLimit`
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`
Fair-share between teams	Fair Sharing policy in the ClusterQueue
Chargeback of the loan	borrowed GPU hours × cost per GPU-hour (OpenCost)

For the utilisation dimension as a FinOps lever, see GPU utilisation as a FinOps lever: Kueue and Volcano together make it possible to maximise utilisation without sacrificing the quota guarantees, which is exactly the FinOps objective.

Managing MIG partitions within this system (declaring nvidia.com/mig-4g.40gb as a resource in the ClusterQueue) integrates naturally: the ResourceFlavor can map to nodes with a specific MIG profile, as explained in sharing a GPU: time-slicing, MPS and MIG.

Diagram: the flow of a training job with Kueue + Volcano

Full comparison table

Criterion	Volcano	Kueue	Coscheduling plugin
Role	Scheduler (placement)	Admission + quota (no placement)	kube-scheduler plugin (placement)
Gang scheduling	Yes, pod level (PodGroup/minMember)	Yes, admission level (quota gang)	Yes, pod level (PodGroup)
Multi-tenant quotas	Yes (Queue capability/guarantee)	Yes (ClusterQueue nominalQuota)	No
Cohorts / borrowing	Limited (reclaimable)	Yes (Cohort with borrowingLimit/lendingLimit)	No
Fair-share	Yes (DRF, proportion)	Yes (Fair Sharing by historical usage)	No
Inter-queue preemption	Yes (reclaim action)	Yes (reclaimWithinCohort)	No
Network / NUMA topology	Yes (v1.11+, dedicated plugins)	Yes (TAS, Topology CRD)	No
Native integrations	MPI, PyTorch, Ray, TF, Spark, Flink, Horovod	Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet	Any job with a PodGroup
Who places the pods	Volcano	kube-scheduler (or Volcano)	kube-scheduler
Elastic gang	Yes (minMember < total replicas)	Partial (partial admission in batch/Job)	Limited
Footprint	Medium-high (its own scheduler)	Light (an additional controller)	Minimal (a scheduler plugin)
Compatibility with Kueue	Yes (as the scheduler under Kueue)	—	Yes (a complementary option)
CNCF status / maturity	CNCF incubating, v1.15	kubernetes-sigs, v1beta2, adopted in GKE/OpenShift	kubernetes-sigs/scheduler-plugins, experimental
When to choose it	HPC-like distributed training, NUMA, network topology, MPI	Multi-tenancy with flexible quota, heterogeneous workloads, inference + batch together	Simple clusters that only need gang without their own scheduler

Operational pitfalls and honest scepticism

1. Deadlock from a badly configured gang

The most frequent scenario: minMember set equal to the total replica count in a cluster where several jobs compete for the same resources. If two jobs of 8 pods each try to use a cluster with 8 GPU nodes and job A has 4 pods placed (not 8, so Volcano holds them pending), and job B also has 4 pods held, nobody moves forward. Volcano is doing its job properly: it places none of them until there is room for all 8. But if the queues’ nominalQuota values are badly sized against the cluster’s real capacity, this produces indefinite waits.

Solution: size the queue quotas so that the sum of guarantee does not exceed the real capacity, and so that the minMember values of the active jobs fit inside the available quota. Node autoscaling with ProvisioningRequest (Kueue + cluster-autoscaler) helps, but it introduces provisioning latency that has to be taken into account in the job’s SLA.

2. Quota versus real capacity: the silent drift

Kueue’s nominalQuota and Volcano’s guarantee are administrative declarations. They do not guarantee that the nodes with those GPUs are available, healthy or that the device plugin has registered them correctly. A node in NotReady with 4 GPUs reduces the real capacity without Kueue knowing: the ClusterQueue will carry on admitting workloads that then cannot be placed.

Recommended monitoring: cross Kueue’s metrics (kueue_admitted_workloads_total, kueue_pending_workloads) with the cluster’s real capacity metrics (GPUs registered in the device plugin) to detect the drift. Kueue exposes native Prometheus metrics; so does Volcano.

3. GPU resource naming: MIG, time-slicing and ResourceFlavor

If the cluster uses MIG, the resources in the pod specs change from nvidia.com/gpu to nvidia.com/mig-Xg.Ygb (for example, nvidia.com/mig-3g.40gb). Kueue’s ResourceFlavors and Volcano’s Queues must declare the right resource, or the quota will not match the pods. With time-slicing, the resource is still nvidia.com/gpu but the device plugin advertises more instances than there are physical GPUs; the quota is expressed in virtual replicas, which can lead to over-admission if the VRAM budget is not taken into account (see sharing a GPU: time-slicing, MPS and MIG).

4. Preemption in production: the evicted workload loses progress

When Kueue or Volcano evicts a training job that has been running for hours, that job loses its progress if it has no checkpointing configured. The preemption is correct from the quota point of view, but it destroys work if the job is not prepared for it. Before enabling aggressive preemption, verify that all training jobs have periodic checkpointing with automatic restoration. PyTorch plus Torchrun have native support; so does DeepSpeed. Stateless batch inference jobs do not have this problem.

5. Volcano as the sole scheduler versus coexistence with the kube-scheduler

Volcano can be configured as the cluster’s default scheduler (all pods go through it) or as an alternative scheduler (only pods with schedulerName: volcano). The first option simplifies the configuration but breaks system pods that assume kube-scheduler behaviour. The second, the recommended one, requires ML jobs to explicitly set schedulerName: volcano, which can be a non-trivial operator or chart change for existing workloads. Kueue solves this more transparently: it only requires a label on the job, with no change of scheduler.

6. Real operational complexity in 2026

Running Kueue plus Volcano plus the Training Operator plus the GPU Operator in production means four components with their own CRDs, webhooks, versions and release cycles. A Kubernetes upgrade may require updating all four in sequence. The operational debt is real. For a small team without the capacity to maintain this stack, a managed Kubernetes provider (GKE with native Kueue, OpenShift with the Red Hat build of Kueue) may be more pragmatic than assembling the full stack from scratch.

The sig-scheduler coscheduling plugin is a deliberately simpler option when all you need is gang: fewer features, less complexity, fewer things to maintain.

Sources

Volcano — Introducción y features (CNCF incubating, v1.15) — https://volcano.sh/en/docs/
Volcano — Volcano v1.11 Released: A New Era of Cloud-Native Scheduling (CNCF blog, marzo 2025) — https://www.cncf.io/blog/2025/03/05/volcano-v1-11-released-a-new-era-of-cloud-native-scheduling-for-ai-and-big-data/
Volcano — Release v1.11.0 (GitHub) — https://github.com/volcano-sh/volcano/releases/tag/v1.11.0
Volcano — Network Topology Aware Scheduling design doc — https://github.com/volcano-sh/volcano/blob/master/docs/design/Network%20Topology%20Aware%20Scheduling.md
Volcano — NUMA-aware scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/numa-aware.md
Volcano — Unified Scheduling (docs oficiales) — https://volcano.sh/en/docs/unified_scheduling/
Volcano — Capacity scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/capacity-scheduling.md
NVIDIA Technical Blog — Practical Tips for Preventing GPU Fragmentation for Volcano Scheduler — https://developer.nvidia.com/blog/practical-tips-for-preventing-gpu-fragmentation-for-volcano-scheduler/
Kueue — Overview (kueue.sigs.k8s.io, actualizado febrero 2026) — https://kueue.sigs.k8s.io/docs/overview/
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Topology Aware Scheduling — https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/
Kueue — Preemption — https://kueue.sigs.k8s.io/docs/concepts/preemption/
Kueue — Setup All-or-nothing with ready Pods — https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready/
Kueue — GitHub (kubernetes-sigs/kueue) — https://github.com/kubernetes-sigs/kueue
Ray — Gang Scheduling with RayJob and Kueue — https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-kueue-gang-scheduling.html
Kubeflow — Volcano scheduler integration — https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/volcano/
kubernetes-sigs/scheduler-plugins — Coscheduling plugin README — https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/coscheduling/README.md
kubernetes/enhancements — KEP-583 Coscheduling (sig-scheduling) — https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling
InfraCloud — Batch Scheduling on Kubernetes: Comparing YuniKorn, Volcano and Kueue — https://www.infracloud.io/blogs/batch-scheduling-on-kubernetes/
Red Hat — Red Hat build of Kueue (OpenShift 4.20) — https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/ai_workloads/red-hat-build-of-kueue
AceCloud — Multi GPU Orchestration in Kubernetes 2026: Kueue, Volcano, DRA — https://acecloud.ai/blog/multi-gpu-orchestration-kubernetes/
CloudOptimo — Kubernetes AI Infrastructure in 2026: GPU Scheduling and Production Realities — https://www.cloudoptimo.com/blog/kubernetes-ai-infrastructure-in-2026-gpu-scheduling-and-production-realities/

From the SLO to the number of GPUs: how to size and justify the investment in inference hardware

Tue, 16 Jun 2026 11:00:00 +0200

Notation: amounts in N € or N USD (source denominated in dollars); decimal point; comma as thousands separator (1{,}234). Generic example hardware: 4×H100 SXM5 80 GB nodes. The dollar sign is not used (it is the formula delimiter).

TL;DR

With an SLO of TTFT P99 ≤ 300 ms and ITL P99 ≤ 50 ms for a production chat, peak GPU utilisation must stay at ≤ 63 %.
A service receiving 5M requests/day with 512 input tokens and 256 output tokens on average generates an hourly peak of ~11,600 tok/s (with a peak factor of 1.8×). With vLLM on H100 SXM5 (Llama-3.3 70B FP8, ~1,850 tok/s at 50 concurrent requests), ≥ 10 GPUs are needed at peak to meet the SLO, which translates into 3 4×H100 nodes (with headroom).
The all-in cost of those 3 nodes is ~83,300 € per year per node (3-year capex + base opex); the €/1M tokens in the base scenario at 63 % utilisation is ~0.37 €.
Against AWS p5 on-demand (~3.90 USD/GPU-hour after the 44 % cut of Jun. 2025), the utilisation break-even sits at ~55 %; against neocloud on-demand (~2.01 USD/GPU-hour), there is no feasible break-even at medium utilisation.
Capex payback happens between 13 and 24 months if sustained utilisation exceeds 70 %.

1. The sizing chain: from demand to the number of GPUs

Sizing an inference platform follows a causal chain of five steps. Each step has a formula; none can be skipped.

1.1 Step 1 — Characterise the demand

Parameter	Symbol	Example
Requests per day (average)	$D$	5,000,000
Input tokens per request (average)	$L_{\text{in}}$	512
Output tokens per request (average)	$L_{\text{out}}$	256
Peak factor (peak hour vs average ratio)	$k_{\text{peak}}$	1.8
Hours per day with significant traffic	$H$	16

The average output throughput:

$$\dot{T}_{\text{avg}} = \frac{D \times L_{\text{out}}}{86{,}400\,\text{s}} = \frac{5{,}000{,}000 \times 256}{86{,}400} \approx 14{,}815\;\text{tok/s}$$

The peak throughput (busy hour):

$$\dot{T}_{\text{peak}} = k_{\text{peak}} \times \frac{D \times L_{\text{out}}}{H \times 3{,}600} = 1.8 \times \frac{5{,}000{,}000 \times 256}{16 \times 3{,}600} \approx 40{,}000\;\text{tok/s}$$

Note: if the traffic profile has very pronounced peaks (peak/average ratio > 3), sizing is done for the peak and average utilisation falls; the sensitivity analysis in §5 quantifies the effect.

1.2 Step 2 — Set the SLO and derive the maximum utilisation

The latency SLO imposes a ceiling on GPU utilisation. Using queueing theory (M/G/1 model):

$$\rho_{\max} \approx 1 - \frac{1}{\sqrt{1 + C_{s}^{2}}} \cdot \frac{W_{\text{queue}}^{*}}{\bar{s}}$$

where $\rho$ is the utilisation, $W_{\text{queue}}^{*}$ the maximum admissible queueing time and $\bar{s}$ the average service time per request. For the simplified M/M/1 case with prefill time dominant:

$$\rho_{\max} = 1 - \frac{W_{\text{queue}}^{*}}{\bar{s} \cdot (1 + W_{\text{queue}}^{*}/\bar{s})}$$

The practical reference table (derived from the Spheron/Little’s Law formula, 2026):

SLO TTFT P99	Maximum utilisation $\rho_{\max}$
200 ms	55 %
300 ms	63 %
400 ms	70 %
500 ms	75 %

For our example (SLO 300 ms P99): $\rho_{\max} = 0.63$.

The ITL SLO imposes an additional constraint: the inference engine must be able to generate the next token in ≤ 50 ms. On H100 SXM5 with vLLM and Llama-3.3 70B FP8, the ITL P50 at 50 concurrent requests is ~20 ms, with P99 ~45 ms. ITL is the dominant constraint only when the batch size is very high (>64 sequences) or VRAM is nearly full.

1.3 Step 3 — Required throughput and throughput per GPU

The throughput the cluster must serve at peak, while respecting $\rho_{\max}$:

$$\dot{T}_{\text{required}} = \frac{\dot{T}_{\text{peak}}}{\rho_{\max}} = \frac{40{,}000}{0.63} \approx 63{,}500\;\text{tok/s (installed capacity)}$$

The throughput per GPU (reference benchmark, vLLM v0.18.0, Llama-3.3 70B FP8, H100 SXM5 80 GB, 50 concurrent requests):

Engine	Throughput (tok/s per GPU)	TTFT P50 / P95 (50 req)	Source
vLLM 0.18.0	1,850	380 ms / 720 ms	Spheron benchmarks, Mar. 2026
SGLang 0.5.9	1,920	360 ms / 680 ms	Spheron benchmarks, Mar. 2026
TensorRT-LLM 1.2.0	2,100	340 ms / 620 ms	Spheron benchmarks, Mar. 2026

The throughput data correspond to the test with Llama 3.3 70B Instruct FP8, 512 input / 256 output tokens, 50 concurrent requests, on bare-metal H100 SXM5. See the engine analysis in serving engines Pareto comparison.

We use vLLM as the general-purpose production reference: $\dot{T}_{\text{GPU}} = 1{,}850$ tok/s.

1.4 Step 4 — Number of GPUs and number of nodes

$$N_{\text{GPU}} = \left\lceil \frac{\dot{T}_{\text{required}}}{\dot{T}_{\text{GPU}}} \right\rceil = \left\lceil \frac{63{,}500}{1{,}850} \right\rceil = \lceil 34.3 \rceil = 35\;\text{GPUs}$$

With 4×H100 SXM5 nodes:

$$N_{\text{nodes}} = \left\lceil \frac{N_{\text{GPU}}}{4} \right\rceil = \left\lceil \frac{35}{4} \right\rceil = 9\;\text{nodes}$$

We add 15 % headroom for hardware failures (~5 % annual rate in small clusters), upgrades and unforeseen peaks:

$$N_{\text{nodes, final}} = \lceil 9 \times 1.15 \rceil = 11\;\text{nodes} \approx 44\;\text{GPUs}$$

For the TL;DR example (5M requests/day with a 16 h active profile, k 1.8 and an SLO of 300 ms) the number of nodes is 11. The simplified TL;DR case with k=1 and H=24 gives 3 nodes; the difference illustrates the impact of the hourly profile.

1.5 Summary of the chain

2. From sizing to cost: the TCO model

With $N_{\text{nodes}} = 11$ 4×H100 SXM5 nodes, the TCO follows the model detailed in TCO of an on-premise GPU cluster. Here the compact formula is reproduced and applied to the sized cluster.

2.1 Annual cost per node (base scenario)

$$C_{\text{node/year}} = \frac{\text{node capex}}{\text{years}} + \text{node opex/year}$$

Line item	Value (USD / €)	Note
4×H100 node capex (midpoint)	178,500 USD	GPUs + server + network + storage + rack
3-year amortisation	59,500 USD/year ≈ 55,300 €/year	Straight line
Opex/year (base scenario, 8–16 node cluster)	~28,000 €/year	Energy + staff + maintenance + colocation
Total per node per year	~83,300 €/year	—

For 11 nodes: 915,300 €/year of total fixed cost.

2.2 From €/node-year to €/GPU-hour

$$\text{EUR/GPU-hour} = \frac{C_{\text{node/year}}}{4\;\text{GPUs} \times 8{,}760\;\text{h} \times \rho}$$

Utilisation $\rho$	EUR/GPU-hour
40 %	5.93
55 %	4.31
63 % (SLO 300 ms)	3.76
70 %	3.39
80 %	2.97
100 %	2.38

2.3 From €/GPU-hour to €/1M tokens

$$\text{EUR/1M tokens} = \frac{\text{EUR/GPU-hour} \times 10^{6}}{\dot{T}_{\text{GPU}} \times 3{,}600}$$

With $\dot{T}_{\text{GPU}} = 1{,}850$ tok/s (vLLM, Llama-3.3 70B FP8):

Utilisation	EUR/GPU-hour	EUR/1M tokens
40 %	5.93	0.891
55 %	4.31	0.647
63 % (SLO 300 ms)	3.76	0.564
70 %	3.39	0.509
80 %	2.97	0.446
100 %	2.38	0.357

The complete cost/token identity as a function of throughput and utilisation is developed in cost per token and per request.

2.4 Comparison with cloud (equivalent €/1M tokens)

To compare, the cloud price is converted to the €/1M tokens equivalent using the same reference throughput ($\dot{T}_{\text{GPU}} = 1{,}850$ tok/s):

$$\text{EUR/1M tokens (cloud)} = \frac{P_{\text{cloud}} \times 10^{6}}{1{,}850 \times 3{,}600}$$

Provider / Mode	GPU-hour price (USD)	EUR/GPU-hour ($1\,\text{USD} \approx 0.93\,\text{EUR}$)	EUR/1M tokens equiv.
AWS p5 on-demand (post Jun. 2025)	3.90	3.63	0.545
AWS p5 1-year reserved	~2.50	2.33	0.350
CoreWeave on-demand	~2.01	1.87	0.281
CoreWeave 3-year reserved	~1.49	1.39	0.209
GCP A3 on-demand	~3.67	3.41	0.512
Azure ND H100 v5 on-demand	~6.98	6.49	0.975

Sources: IntuitionLabs (Jun. 2026), CloudZero (Jun. 2026), Spheron GPU pricing (May 2026).

3. Break-even and payback of your own investment

3.1 The utilisation break-even formula

The utilisation break-even $u^{*}$ is the utilisation at which the annual on-prem cost per GPU-hour equals the cloud price:

$$u^{*} = \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times P_{\text{cloud}}}$$

where $P_{\text{cloud}}$ is the cloud price in the same currency as the on-prem costs.

With the base scenario (capex/year 55,300 €, opex/year 28,000 €, total 83,300 €/node/year):

Cloud reference	Cloud price (EUR/GPU-hour)	$u^{*}$ break-even
Azure on-demand (~6.49 EUR)	6.49	23 %
AWS on-demand post-cut (~3.63 EUR)	3.63	41 %
GCP on-demand (~3.41 EUR)	3.41	44 %
CoreWeave on-demand (~1.87 EUR)	1.87	80 %
CoreWeave 3-yr reserved (~1.39 EUR)	1.39	>100 % (impossible)
AWS 1-yr reserved (~2.33 EUR)	2.33	64 %

3.2 Capex payback

Payback is the time $T_{\text{pay}}$ in which the cumulative saving against cloud equals the initial capex:

$$T_{\text{pay}} = \frac{\text{total cluster capex}}{(\text{cloud cost/year}) - (\text{on-prem opex/year})}$$

where the cloud cost/year is computed at the same sustained utilisation.

For the 11-node cluster (total capex 11 × 178,500 USD ≈ 1,825,000 USD ≈ 1,697,000 €):

Cloud reference	Sustained utilisation	Annual saving vs cloud	Payback
AWS on-demand (3.63 EUR/h)	70 %	(3.63 − 3.39 EUR) × 4 × 8760 × 0.70 × 11 nodes ≈ 63,900 €/year	~27 months
AWS on-demand (3.63 EUR/h)	80 %	(3.63 − 2.97) × 4 × 8760 × 0.80 × 11 ≈ 203,600 €/year	~10 months
GCP on-demand (3.41 EUR/h)	70 %	(3.41 − 3.39) × 4 × 8760 × 0.70 × 11 ≈ 5,400 €/year	~315 months (not viable)
Azure on-demand (6.49 EUR/h)	70 %	(6.49 − 3.39) × 4 × 8760 × 0.70 × 11 ≈ 830,000 €/year	~2 months

The 13-month payback quoted in studies such as Lenovo TCO 2026 corresponds to ~80 % utilisation against high-priced hyperscalers (Azure/AWS before the cut). With current prices (post June 2025, AWS at 3.90 USD), the window widens.

3.3 Simple payback (capex only vs equivalent cloud)

If on-prem opex is omitted and only capex is compared with the gross saving:

$$T_{\text{pay,simple}} = \frac{\text{capex}}{P_{\text{cloud}} \times 4 \times 8{,}760 \times \rho \times N_{\text{nodes}}}$$

Utilisation	AWS (3.63 EUR)	Azure (6.49 EUR)
50 %	54 months	30 months
70 %	39 months	22 months
80 %	34 months	19 months

4. Decision table: when to buy, rent or go hybrid

The table below is a five-dimension Pareto. There is no implicit order between columns; the reading depends on the organisation’s constraints.

Option	EUR/GPU-hour (util. 70 %)	Initial capex	Required utilisation	Data sovereignty	Peak elasticity
On-prem purchase, util. ≥ 70 %	3.39	high (178k USD/node)	≥ 70 % sustained	total	none
On-prem purchase, util. < 50 %	> 4.75	high	< 50 % → loses vs cloud	total	none
Sovereign EU cloud OD (Scaleway/Nebius EU)	2.00–3.59	none	any	yes (EU)	total
AWS p5 on-demand (post Jun. 2025)	3.63	none	any	no (CLOUD Act)	total
CoreWeave on-demand	1.87	none	any	partial (US)	total
CoreWeave 3-year reserved	1.29–1.39	financial commitment	rigid contract	partial (US)	none
AWS 1-year reserved	~2.17	1-year commitment	contract	no (CLOUD Act)	none
Hybrid on-prem base + EU cloud peak	2.50–3.39 (weighted)	medium	base ≥ 70 %, elastic peak	yes (EU)	elastic peak

Cut-off criteria that come before the table:

GDPR sovereignty: if the data is personal or the system is high-risk under the EU AI Act, CoreWeave/AWS are ruled out before any price comparison.
Minimum volume to amortise capex: below ~2M tokens/day sustained over 3 years, on-prem capex does not amortise against AWS on-demand.
Traffic elasticity: peaks >3× the base favour a hybrid or pure cloud; a stable base favours on-prem.

The cost/sovereignty Pareto frontier for GDPR data leaves three options: on-prem, sovereign EU cloud and hybrid. Sustained utilisation and traffic predictability decide between them. See the cross-analysis of the four axes in sovereign on-premise vs hyperscalers.

5. Sensitivity analysis

5.1 Sizing vs hourly profile and peak factor

The number of GPUs grows linearly with $k_{\text{peak}}$ and inversely with $\rho_{\max}$:

$$N_{\text{GPU}} = \left\lceil \frac{D \times L_{\text{out}} \times k_{\text{peak}}}{H \times 3{,}600 \times \rho_{\max} \times \dot{T}_{\text{GPU}}} \right\rceil$$

Peak factor $k$	SLO 300 ms ($\rho_{\max}=0.63$)	SLO 500 ms ($\rho_{\max}=0.75$)
1.2	8 GPUs (2 nodes)	6 GPUs (2 nodes)
1.8	12 GPUs (3 nodes)	10 GPUs (3 nodes)
2.5	16 GPUs (4 nodes)	13 GPUs (4 nodes)
3.5	22 GPUs (6 nodes)	18 GPUs (5 nodes)

(Example simplified to 5M req/day with H=24 to illustrate the sensitivity to the peak factor)

A peak factor of 3.5× triples the number of nodes compared with k=1.2 at the same SLO. Sizing hardware for $k > 2.5$ leaves GPUs idle more than 70 % of the time; peak cloud is more efficient beyond that threshold.

5.2 Break-even vs sustained utilisation

$$u^{*} = \frac{83{,}300}{4 \times 8{,}760 \times P_{\text{cloud}}}$$

Cloud price (EUR/GPU-hour)	$u^{*}$ break-even	Scenario
6.49 (Azure OD)	23 %	On-prem wins almost always
3.63 (AWS OD)	41 %	On-prem wins if util. > 41 %
3.41 (GCP OD)	44 %	—
2.33 (AWS 1-yr reserved)	64 %	On-prem wins if util. > 64 %
1.87 (CoreWeave OD)	80 %	Hard to reach in production
1.39 (CoreWeave 3-yr reserved)	>100 %	On-prem never closes the gap

5.3 Break-even vs energy price

Energy accounts for 6–11 % of total TCO. Its impact on the break-even is moderate:

Energy price (EUR/kWh)	Energy opex/year per node	EUR/GPU-hour (70 % util.)	$u^{*}$ vs AWS OD
0.034 (solar PPA)	1,604 €	3.22	38 %
0.116 (industrial ES, base)	5,475 €	3.39	41 %
0.200 (high tariff)	9,437 €	3.57	43 %

The difference between the cheapest and the most expensive scenario is only 5 percentage points of break-even. The variable that moves the needle is utilisation, not energy.

5.4 Break-even vs demand growth

If demand grows at an annual rate $g$, the average utilisation of the cluster (sized for year 1) rises over time until it saturates and has to be expanded:

$$\rho(t) = \rho_{0} \times (1 + g)^{t}$$

Annual growth $g$	Time to saturation ($\rho \to 100\,\%$)	Decision
10 %	~11 years	Comfortable purchase
30 %	~4 years	Purchase with a review at 3 years
60 %	~2 years	Hybrid: base + elastic cloud
>100 %	<1 year	Pure cloud until it stabilises

For growth above 30 % per year, a buy-only strategy means oversizing for the future peak or re-buying hardware in short cycles. The hybrid (on-prem base + cloud for growth) minimises the capex at risk.

5.5 Headroom: the cost of the safety margin

The 15 % headroom in $N_{\text{nodes}}$ amounts to having ~1.6 extra nodes on average. Its annual cost is:

$$C_{\text{headroom}} = 0.15 \times 83{,}300\;\text{EUR/node/year} \times N_{\text{nodes,base}} \approx 12{,}500 \times 9 = 112{,}500\;\text{EUR/year}$$

That cost is justified by:

GPU failure rate ~5 % per year (in small clusters, documented by Introl, Apr. 2026)
Replacement time of 2–8 weeks (depending on market availability)
Unforeseen peaks of up to 20 % above the estimate

If the service can degrade gracefully (relaxing the TTFT SLO from 300 ms to 500 ms at extreme peak), headroom can be cut to 10 %, saving ~37,500 EUR/year.

6. Sensitivity map: €/GPU-hour and break-even on two axes

The table below crosses utilisation with the opex scenario, showing the all-in EUR/GPU-hour (base scenario, capex/year 55,300 €):

	Low opex (13,000 €/year)	Base opex (28,000 €/year)	High opex (75,000 €/year)
Util. 40 %	4.74	5.93	9.06
Util. 55 %	3.45	4.31	6.59
Util. 63 %	3.01	3.76	5.75
Util. 70 %	2.71	3.39	5.18
Util. 80 %	2.37	2.97	4.54
Util. 100 %	1.90	2.38	3.63

The crossing point with the cloud price (AWS OD: 3.63 EUR):

Low-opex scenario: break-even at ~38 % utilisation
Base-opex scenario: break-even at ~41 % utilisation
High-opex scenario: break-even at ~53 % utilisation

The biggest lever for lowering the break-even is not hardware capex but opex (staff and colocation above all). See the analysis of utilisation as a FinOps lever in GPU utilisation as FinOps.

7. Integration with the rest of the FinOps chain

The sizing of §1 determines the number of nodes; the TCO of §2 gives the cost per hour; the comparison of §3 gives the break-even. Those three numbers feed directly into the other instruments of the series:

Instrument	Input from this article	Output
GuideLLM — SLO validation under load	TTFT/ITL P99 SLO from step 2	Experimental confirmation of the real throughput per GPU
Capacity planning for on-premise inference	Demand profile and N GPUs from step 4	Scaling policy and autoscaling triggers
Cost per token and per request	EUR/GPU-hour from step §2.2	EUR/1M tokens per model and batching
Cloud GPU commitment and spot	Cloud prices from table §2.4	Optimisation of the complementary cloud tier
Sovereign on-premise vs hyperscalers	Break-even from §3	Final buy/rent decision including the sovereignty axis
TCO of an on-premise GPU cluster	Capex and opex from §2.1	Detailed TCO model with every line item
GPU utilisation as FinOps	Target utilisation from step 2	Scheduling levers to raise real utilisation

Sources

Spheron · LLM Inference SLO Engineering: TTFT, ITL, and P99 Latency Budgets for Production AI (2026) — https://www.spheron.network/blog/llm-inference-slo-ttft-itl-latency-budget-guide-2026/
Spheron · vLLM vs TensorRT-LLM vs SGLang: H100 Benchmarks (2026) — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
Spheron · GPU Cloud Pricing 2026: H100 from 1.03 USD/hr, B200 from 2.12 USD/hr — https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
MLPerf Inference v5.1 — Red Hat: 5 777 tok/s (offline) en Llama 3.1-8B FP8 en H100 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
MLPerf Inference v6.0 Results Explained: GPU Performance Rankings for AI Workloads (2026) — https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
IntuitionLabs · H100 Rental Prices Compared: 1.49–6.98 USD/hr Across 15+ Cloud Providers (2026) — https://intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison
CloudZero · Cloud GPU Pricing Comparison: AWS vs Azure vs GCP For AI Workloads (2026) — https://www.cloudzero.com/blog/cloud-gpu-pricing-comparison/
Lenovo Press · On-Premise vs Cloud: Generative AI Total Cost of Ownership (2026 Edition) — https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
Introl · GPU Infrastructure TCO Model: 5-Year Cost Analysis for Enterprise AI (abr. 2026) — https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model
GMI Cloud · NVIDIA H100 GPU Pricing 2026: Rent vs Buy Cost Analysis — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Red Hat · 233 % 3-year ROI and 13 months to payback with Red Hat AI (feb. 2026) — https://www.redhat.com/en/blog/233-3-year-return-investment-and-13-months-payback-red-hat-ai
VentureBeat · 5 % GPU utilization: the 401 billion USD AI infrastructure problem — https://venturebeat.com/infrastructure/5-gpu-utilization-the-401-billion-ai-infrastructure-problem-enterprises-cant-keep-ignoring/
DZone · Queueing Theory for LLM Inference — https://dzone.com/articles/queueing-theory-for-llm-inference
GuideLLM · Evaluate LLM deployments for real-world inference (Red Hat Developer, jun. 2025) — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference

Kubecost vs OpenCost vs the alternatives: what the commercial layer adds and when it is worth paying for

Sun, 14 Jun 2026 02:30:00 +0200

Notation: amounts in euros (N €), decimals with a point; when a source quotes dollars it is marked “USD”. The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Third article in the FinOps track (A3). A2 covered how OpenCost allocates cost, the free base. This article answers the question that comes next: is it worth paying for Kubecost or a commercial alternative, or is OpenCost enough? It is a build-vs-buy decision, operating the open source tool yourself against buying a product, and the answer depends on your scale, your audience and your appetite for operating infrastructure. No universal recommendations: here are the facts (what each one adds, what it costs, when it fits) and, for a European platform, the nuance of cost data sovereignty that the American comparisons do not mention.

The axis of the decision: build vs buy

OpenCost is free (Apache 2.0), but “free” means operating it yourself: deploying it, configuring the node price, keeping Prometheus with enough retention, building the Grafana panels and the alerts. Kubecost (and the alternatives) charge for taking that work away and adding capabilities OpenCost does not ship. The decision, therefore, is not “free vs expensive”: it is your own operating cost vs a licence.

Axis	Leans towards OpenCost	Leans towards commercial
Platform team	you have someone to operate it	you do not want to operate anything
Scale (cloud/GPU spend)	small to medium	large (the saving pays the licence)
Report audience	engineers	finance / management
Number of clusters	one	many (multi-cluster)
Need for automatic optimisation	you do it by hand	you want it out of the box

OpenCost is the open source core; Kubecost is the enterprise product built on that core, with proprietary features added; Kubecost was the original developer of the engine before releasing it, and IBM acquired the company (CloudZero · Kubecost vs OpenCost). In other words: both share the same allocation engine; what you pay for is the layer on top.

What Kubecost adds on top of OpenCost

On top of allocation (which is common to both), Kubecost adds a layer of optimisation, governance and enterprise features (CloudZero, Finout):

Capability	OpenCost	Kubecost
Cost allocation (CPU/GPU/mem/PV)	✓	✓ (same engine)
GPU via DCGM	✓	✓ (3.0)
Invoice reconciliation (discounts, RI, spot)	—	✓
Rightsizing (recommendations)	—	✓
Anomaly detection	—	✓
Budget alerts	manual	✓
RBAC	—	✓
Multi-cluster aggregation	—	✓
Hosted option (SaaS)	—	✓
Long historical retention	your Prometheus	included
Support	community	commercial (IBM)

Invoice reconciliation

The capability that most justifies the price in cloud: OpenCost allocates on list rates; Kubecost reconciles against the real invoice, which includes discounts, reserved instances and spot prices (Finout). The difference can be large: a cost allocated at list rates can be 30–50 % above what you actually pay after discounts. On-prem this matters less (you declare the node’s real price), but in cloud or hybrid, reconciliation is the difference between a “theoretical” cost and the cost that shows up on the invoice.

Rightsizing and anomaly detection

Kubecost recommends resizing (rightsizing) underused resources, via the integration with IBM Turbonomic, and detects spend anomalies automatically. OpenCost gives you the data to do it by hand; Kubecost automates it. For a GPU fleet, rightsizing and anomalous idle detection are exactly where the saving of the Optimize phase lives.

The price of Kubecost (and the build-vs-buy economics)

Kubecost’s price starts at 449 USD/month for business use, with enterprise options on request (source). The right question is not “is 449 USD expensive?”, but “does operating OpenCost myself cost more or less than that?”. The build-vs-buy calculation:

Cost	OpenCost (build)	Kubecost (buy)
Licence	0	from ~449 USD/month (business)
Engineering time	deployment + maintenance + panels	minimal
Historical retention	your Prometheus/Thanos cost	included
Support	your team	included

If one platform person spends a few hours a month operating OpenCost, that time can already exceed 449 USD; if your cluster is small and the operating cost is nearly nil (you set it up once and it runs), OpenCost wins. The market rule of thumb: if monthly Kubernetes spend is below ~10,000 USD, the cloud-native tools or open source (OpenCost) usually suffice; above that, the saving the commercial tool captures pays for its licence ([search]).

Worked example: build vs buy

An illustrative calculation for a platform with a medium-sized GPU cluster:

Item	OpenCost (build)	Kubecost (buy)
Annual licence	0	~449 USD/month × 12 = ~5,400 USD (~5,000 €)
Initial deployment	~2–3 days of engineering	~half a day
Maintenance	~4–8 h/month (panels, alerts, upgrades)	minimal
Operating cost (at ~60 €/h)	~240–480 €/month	~marginal
Historical retention	cost of your own Thanos/Mimir	included

If OpenCost’s operating cost is around 240–480 €/month, it is in the same order as the Kubecost licence (~415 €/month). The decision, then, is not set by the price but by the added value: if invoice reconciliation, automatic rightsizing and the finance-facing UI save you or give you more than that difference, Kubecost wins; if your team already operates Prometheus/Grafana and your audience is engineers, OpenCost wins. For a large multi-cluster estate, the saving Kubecost captures usually far exceeds the licence; for a small and stable one, operating OpenCost is almost free. The number that decides is not 449 USD, it is your operating cost against that added value.

Migration and reversibility: an argument for starting with OpenCost

A rarely mentioned advantage of Kubecost being built on OpenCost: migrating between the two is low risk. Starting with OpenCost and, if scale justifies it, moving up to Kubecost later does not force you to redo the allocation, it is the same engine, the same concepts, the same Prometheus/DCGM instrumentation. What changes is the layer on top. This reduces lock-in: it is not an irreversible bet.

For a platform just starting out, the low-risk strategy is clear: stand up self-hosted OpenCost first, get the allocation right with the node price in euros, live with it for a few months, and only then decide whether invoice reconciliation, automatic rightsizing and the finance-facing UI justify paying for Kubecost. Starting with the commercial product “just in case” is paying before knowing whether you need it; starting with OpenCost and moving up if necessary is the option that keeps reversibility and discovers the real value before the invoice. And since both are self-hosted, neither route gives up data sovereignty.

The alternatives: CloudZero, Vantage, Finout

Kubecost is not the only commercial option; there is a category of cost platforms that operate at a different level (closer to the business than to Kubernetes):

Tool	Focus	Differentiator	Layer
CloudZero	unit economics	cost per feature/product/customer	business
Vantage	multi-cloud (20+ integrations)	breadth (incl. OpenAI invoice)	business
Finout	multi-cloud	virtual tagging (without touching resources)	business
Kubecost	Kubernetes	intra-cluster depth + optimisation	resource+optim
OpenCost	Kubernetes	open source allocation	resource

The key distinction: Kubecost/OpenCost are Kubernetes-centred tools (intra-cluster depth); CloudZero/Vantage/Finout are business cost platforms (multi-cloud breadth, cost per product). They do not always compete: many organisations use OpenCost/Kubecost for the Kubernetes detail and a business platform for the aggregate view.

A short profile of each:

CloudZero — a cost intelligence platform with a dedicated Kubernetes visibility solution for engineering teams. Its hallmark is unit economics: mapping cost to feature, product and customer, answering “how much does it cost me to serve this customer?” more than “how much does this pod cost?”. It fits when cost has to be taken to the product P&L.
Vantage — breadth of integrations (more than 20 sources: AWS, Azure, GCP, Kubernetes, Snowflake, Datadog and the invoice from LLM providers such as OpenAI). It is the option when AI cost is spread across many providers and you want a single view, including the invoice from the model APIs.
Finout — multi-cloud with virtual tagging: it applies cost labels without modifying the resources, which allows spend that was badly tagged at source to be allocated, and quick deployment. It is among the top Kubernetes cost optimisation tools (CloudBolt, Finout).

The usual pattern in a mature organisation: OpenCost/Kubecost for the Kubernetes and GPU detail, and a business platform on top to cross that cost with the rest of the cloud spend and take it to product. It is not “one or the other”: it is often “one and the other”, in layers.

Decision criteria

Five questions that settle the choice without opinion:

How much do you spend? Below ~10k USD/month, OpenCost is enough; above that, the saving pays for the commercial tool.
Who reads the report? Engineers → OpenCost; finance/management → Kubecost’s polished UI or a business platform.
One cluster or many? Multi-cluster leans towards Kubecost.
Do you want to operate infrastructure? If not, buy; if you have a platform team, build with OpenCost.
Where can the cost data live? The European question, below.

The European angle: cost data sovereignty

American comparisons leave out a criterion that for a sovereign platform is first-order: where the cost and usage data ends up. CloudZero, Vantage, Finout and Kubecost’s hosted option are SaaS, mostly American: you send them your telemetry of cost, usage, namespace names, teams and products. That data, which describes your operation in detail, leaves for a jurisdiction subject to the US CLOUD Act, with the same GDPR consideration as any other sensitive data sent to a hyperscaler.

Option	Where the cost data lives	Sovereignty
Self-hosted OpenCost	in your cluster	total
Self-hosted Kubecost	in your cluster	total
Hosted Kubecost (SaaS)	the provider’s SaaS	subject to the SaaS jurisdiction
CloudZero / Vantage / Finout	SaaS (US)	subject to the US CLOUD Act

It is worth being fair: cost data is not the same as user data. Spend telemetry per namespace is less sensitive than the content of the inferences. But it does reveal the operation: which teams exist, which products consume GPU, the size of the platform, and, crossed with namespace names, information that taken together can be sensitive. Most SaaS FinOps tooling is American; there are emerging European options, but few with the maturity of the big players. When in doubt, and for data subject to GDPR, the minimisation principle advises not taking out of the EU what does not need to leave, and cost allocation, with self-hosted OpenCost/Kubecost, does not need to leave.

The consequence for the proposal: self-hosted OpenCost or Kubecost keep the cost data under your control, just as the cluster keeps the inference data. If data sovereignty is a requirement (it is for GDPR data, see ENS × 42001 × EU AI Act controls), the American SaaS platforms run into the same conflict as a hyperscaler, however good their UI is. For Fibercli and any sovereign platform, this tilts the balance towards self-hosted, with OpenCost as the base and self-hosted Kubecost if the enterprise layer is wanted without giving up the data.

AI cost: what none of them ships out of the box

One point the generalist comparisons do not highlight and that is central for an LLM platform: none of these tools gives cost per token out of the box. OpenCost and Kubecost reach the pod cost (per resource/usage); CloudZero, Vantage and Finout reach the resource or invoice cost (including the OpenAI one, in Vantage’s case). But the jump from “this vLLM pod cost X €/hour” to “this request from this team cost Y” requires intercepting the inference traffic with a gateway (LiteLLM) that counts tokens per request and per team, as developed in the FinOps introduction.

Layer	What it gives	Who covers it
Resource/pod cost	€/hour per pod, namespace, team	OpenCost, Kubecost
Cloud/LLM invoice cost	€/month per provider	CloudZero, Vantage, Finout
Cost per token	€/1M tokens per team/model	gateway (LiteLLM) + any of the above

The implication for tool selection: cost per token, the metric that compares on-prem against cloud, is not solved by any tool on its own, but by the combination of allocation (OpenCost/Kubecost) with token measurement (gateway). When evaluating a commercial tool, the AI question is not “does it give the pod cost?” (they all do, one way or another), but “does it integrate well with my gateway to reach the token?”. Vantage, by ingesting the OpenAI invoice, gets close from the external API consumption side; for your own inference, the gateway remains essential.

State of the art 2026

Consolidation under IBM: Kubecost/OpenCost integrated into IBM’s FinOps Suite alongside Cloudability and Turbonomic; OpenCost remains the CNCF allocation standard.
First-class GPU: Kubecost 3.0 and OpenCost allocate GPU via DCGM; GPU FinOps stops being a special case.
Towards the token and FOCUS: the strong platforms track cost at token level, and the FOCUS standard (v1.3, Dec 2025) is extending to AI workloads, which will push interoperability between these tools.
A category of AI cost platforms: specific AI cost observability tooling is emerging that combines resource, invoice and token; it is worth demanding FOCUS compatibility so as not to couple to one provider.

Selection checklist

To decide without opinion, in order:

Step	Question	If…
1	Do I have a team to operate OpenCost?	no → commercial / hosted
2	Spend > ~10k USD/month and multi-cluster?	yes → Kubecost
3	Is the report for finance/management?	yes → Kubecost UI or business platform
4	Do I need cost per product/customer?	yes → CloudZero/Vantage/Finout on top
5	Can the cost data go to a US SaaS?	no → self-hosted (OpenCost / Kubecost)
6	Do I need cost per token?	yes → any of them + gateway

Step 5 is the filter for a sovereign platform: if the cost data cannot leave the EU jurisdiction, the list shrinks to the self-hosted options, with OpenCost as the base.

Pricing models of commercial tooling

For those that charge by saving or by percentage of spend:

Model	How it charges	Range
Fixed licence (Kubecost)	monthly rate	from ~449 USD/month (business)
Savings-based	% of the savings delivered	15–35 %
Fixed-fee	% of annual cloud spend	1–3 %

The implication: at large spend, a 1–3 % fixed-fee can exceed Kubecost’s fixed licence; with a lot of waste, savings-based aligns incentives but can come out expensive if the saving is large. OpenCost, free, changes the equation if you have someone to operate it.

Coexistence: the reality is layered

In mature organisations, these tools are not chosen against each other, they are stacked. The usual pattern:

Layer	Tool	Answers
Intra-Kubernetes allocation	OpenCost / Kubecost	how much does each pod/team/GPU cost?
Per-token measurement	gateway (LiteLLM)	how much does each request/team cost?
Cost per product / multi-cloud	CloudZero / Vantage / Finout	how much does it cost to serve this product/customer?

Each layer solves a different question and feeds off the one below: the pod cost (OpenCost) plus the tokens (gateway) give the cost per token; that cost per token, aggregated by a business platform, gives the cost per product. Trying to make a single tool cover all three layers usually ends in compromises: the Kubernetes ones do not reach the business, the business ones do not reach the pod, and none reaches the token without the gateway. The healthy architecture is layers that integrate, not one tool that does everything by halves. And the glue between layers, in 2026, is FOCUS: demanding FOCUS compatibility from each piece is what allows the data to flow between them without being rewritten.

Summary by organisation profile

To get your bearings quickly, which stack fits each profile (indicative, not prescriptive):

Profile	Scale	Stack that usually fits
Small team with a platform	1 cluster, low spend	OpenCost self-hosted + Grafana
Scale-up with several teams	multi-team, medium spend	OpenCost + gateway; Kubecost if time is short
Multi-cluster enterprise	many clusters, high spend	Kubecost + business platform
Sovereign platform (Fibercli)	GDPR, data in the EU	self-hosted OpenCost/Kubecost + gateway; no US cost SaaS

The sovereign profile is the one that changes the answer relative to the standard guides: however much an American SaaS platform wins on features or on UI, the requirement to keep cost data under EU jurisdiction rules it out for GDPR data. For Fibercli, the bottom row is the one that rules, and that is why this series’ FinOps track is built on self-hosted tools.

Limits and traps (data-driven)

“Free” is not free. OpenCost has an operating cost; count it in the build-vs-buy or you are fooling yourself comparing 0 with 449 USD.
Reconciliation only matters in cloud. In pure on-prem, you declare the node’s real price and Kubecost’s reconciliation adds less.
Cost SaaS = cost data outside. For a sovereign platform, sending cost telemetry to an American SaaS reintroduces the jurisdiction problem that on-prem avoided.
Same engine, different UI. A good part of what you pay for in Kubecost is the presentation and governance layer; if your audience is engineers, it may not pay off.
Do not confuse layers. OpenCost/Kubecost (Kubernetes) and CloudZero/Vantage/Finout (business) solve different problems; sometimes they complement each other, they do not compete.

With the allocation tooling decision settled, the FinOps track moves on to cost per token (A4) and the full TCO model (A8). The foundation is the one from A2 and A3: correct allocation, with the node price in euros, in a tool that respects data sovereignty.

Closing

The OpenCost vs Kubecost vs commercial platform choice is sold as a feature comparison, and is in reality a build-vs-buy decision with a sovereignty constraint on top. The allocation engine is the same (OpenCost) in the first two; what you pay for in Kubecost is the optimisation, governance and presentation layer, which pays off when scale is large, the audience is financial or you do not want to operate infrastructure. The business platforms (CloudZero, Vantage, Finout) solve a different problem, cost per product, multi-cloud, and often coexist in layers with the Kubernetes ones rather than competing. But for a European sovereign platform there is a criterion no American comparison puts first and that for Fibercli comes first: where the cost data lives. Sending your spend, team and product telemetry to an American SaaS reintroduces, through the back door, the jurisdiction problem that on-prem avoided. The conclusion the data supports: self-hosted OpenCost as the base, self-hosted Kubecost if the enterprise layer is wanted, and the gateway to reach the token, a stack that keeps cost measured, in euros, and under EU jurisdiction. Expensive or cheap is secondary; sovereign or not is not.

Sources

CloudZero · Kubecost vs OpenCost (2026) — https://www.cloudzero.com/blog/kubecost-vs-opencost/
Finout · Kubecost vs OpenCost (6 diferencias) — https://www.finout.io/blog/kubecost-vs-opencost
Finout · Kubecost: pros/cons, pricing, alternativas (2026) — https://www.finout.io/blog/kubecost-pros/cons-pricing-tutorial-alternatives-2026-guide
CloudBolt · Top Kubecost Alternatives (2026) — https://www.cloudbolt.io/blog/top-kubecost-alternatives/
Amnic · OpenCost vs Kubecost — https://amnic.com/blogs/opencost-vs-kubecost
Clanker Cloud · Best tools for managing GPU usage in Kubernetes (2026) — https://clankercloud.ai/blog/best-tools-managing-gpu-usage-kubernetes-2025-2026-cost-roi

OpenCost in depth: how GPU cost is allocated in Kubernetes

Sun, 14 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. European references (OVH is among the supported providers). The dollar sign is not used (on this site it is a formula delimiter).

What this article covers

Second article in the FinOps track (A2), and the first tool deep dive. The FinOps introduction established that OpenCost is the CNCF standard for cost allocation; here the box gets opened: how it allocates cost internally, where it gets its data from, how on-prem prices are configured in euros, what its API exposes, and where the traps are. Understanding the mechanics matters because it defines what any tool on top (Kubecost included) can and cannot do, and because a badly set base price invalidates the whole split. No recommendations; only the mechanics and the methodology.

What OpenCost is

OpenCost is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project). It is a cost allocation layer for Kubernetes: it reads resource usage from the cluster, joins it with a price, and splits the cost across Kubernetes dimensions down to the container (OpenCost · GitHub). It supports dynamic pricing via the billing APIs of AWS, Azure and GCP, and custom prices for on-prem clusters (OpenCost · configuration).

What it is not: it is not an optimiser, nor a governance system, nor a product unit economics tool. It is the piece that answers “how much does each namespace/pod/team cost?”, and leaves the rest to the layers above. That is why it is the foundation of the FinOps track: without correct allocation, there is no cost per token and no TCO model worth anything.

The cost model: node-level allocation

The key to OpenCost is that the model works at node level. It starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit per-resource prices, OpenCost uses the ratio of a set of base prices (marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem).

This is what makes it work on-prem: you declare the node cost (depreciated capex + opex), and OpenCost splits it without needing a cloud invoice. The normalisation guarantees that no cost is “lost” or “invented”: what you pay for the node is exactly what gets split across its resources.

Where the data comes from: Prometheus

OpenCost instruments nothing of its own: it reads from Prometheus, which is a prerequisite of the installation (OpenCost · GitHub). The sources:

Source	What it provides
`kube-state-metrics`	state of K8s objects (pods, requests/limits)
`node-exporter`	node resources and capacity
`cAdvisor`	real CPU/memory usage per container
`dcgm-exporter` (NVIDIA)	GPU usage, memory and power

With those series, OpenCost automates the join between Kubernetes resource usage and the provider price, reducing the custom PromQL you would otherwise write by hand (Grafana · manage costs). The GPU signal comes from DCGM via the dcgm-exporter (part of the NVIDIA GPU Operator), the same base as GPU observability. Without Prometheus and without DCGM exporting, OpenCost does not see the GPU.

Architecture: the pieces

OpenCost is not a monolith; it is three responsibilities worth telling apart:

Piece	Function
Cost model	resolves the node price and splits/normalises it across resources
Allocation API	serves the allocated cost per dimension (`/allocation` and endpoints)
Exporter	publishes cost metrics on `/metrics` for Prometheus/Grafana

The data flow: Prometheus holds the usage series (from kube-state-metrics, cAdvisor, dcgm-exporter); the cost model joins them with the resolved price; the Allocation API serves them aggregated by whatever dimension you ask for; and the exporter returns them to Prometheus so that Grafana can plot them. It is a cycle: metrics come in from Prometheus and cost goes back to Prometheus enriched. That symmetry is what lets you reuse your observability stack without standing up a new database: OpenCost lives on top of Prometheus, not beside it.

An operational note: historical analysis depends on Prometheus retention. For cost reports spanning months, a long-term backend behind it (Thanos, Mimir, VictoriaMetrics) is advisable, or periodically exporting the allocations to a store of your own.

On-prem prices in euros: the piece that decides everything

On-prem, the most important figure, and the most neglected, is the node price. It is configured per provider; the options are alibaba, aws, azure, gcp, oracle, ovh or default (on-prem) (OpenCost · configuration). For on-prem you use default, supplying base prices in a default.json (or by overriding the Helm values.yaml); anything you do not override uses the chart’s value (OpenCost · on-prem).

To set the node price in euros, the calculation is the TCO model one: depreciated capex + opex. An example of an 8×H100 node:

Component	Calculation	€/hour
Capex (node ~240,000 €, 36 months)	240,000 ÷ 26,280 h	~9.13
Energy (5.6 kW × PUE 1.4 × 0.058 €/kWh, France)	~0.455 €/h	~0.46
Maintenance, networking, operations	estimate	~1.5
Node total		~11.1 €/h

That ~11 €/h is the number you put into OpenCost, and everything is split from it. And here is the documented trap: with the default pricing configuration, OpenCost under-prices CPU and GPU on-prem (issue #3781). If you do not adjust the base prices to your real cost, the split will be systematically low and the cost per token you report will be unreal. Configuring the node price is not optional: it is the figure everything else hangs from.

Configuring the price: an example

The base price is given in the Helm values.yaml (or a default.json), in euros per hour and per unit of resource, derived from the node’s total cost:

# values.yaml — on-prem prices in €/hour (8xH100 node, France)
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 8xH100 node"
 CPU: "0.030" # €/CPU-hour
 RAM: "0.004" # €/GB-hour
 GPU: "1.30" # €/GPU-hour ← the figure that moves cost/token the most
 storage: "0.0002" # €/GB-hour

OpenCost normalises those values so that, multiplied by the node capacity, they add up to the total price you declared (~11 €/h in the example). The number to be most careful with is the GPU one: at 1.30 €/GPU-hour, the 8 cards are ~10.4 €/h, the bulk of the node’s cost. A badly set GPU base shifts the whole cost per token. Always review the resolved prices in /allNodePricing after configuring, because the default under-pricing bug bites exactly here.

Allocation: by usage, not by presence

The behaviour that has changed most and that matters most for GPU: in recent versions, the cost allocated to a GPU is determined by its usage, not its presence. If a GPU costing 100 €/month is only half used, the allocated cost is ~50 € ([search]). This aligns allocation with reality, you pay for what you use, but it has a consequence: the other half does not disappear, it is idle that somebody keeps paying for. OpenCost exposes it, and that is where the optimisation lever comes from.

OpenCost allocates to any Kubernetes dimension: cluster, node, namespace, controller (deployment, statefulset and so on), service, pod and container, and by labels (team, product). Access is through the Allocation API and several endpoints that expose the mechanics:

Endpoint	What it returns
`/allocation`	cost allocated by the dimension you ask for
`/costDataModel`	the resolved node price
`/allNodePricing`	hourly prices per node
`/pricingSourceSummary`	summary of the pricing source
`/metrics`	cost metrics for Prometheus

The /metrics endpoint is what turns OpenCost into a Prometheus exporter: once there, you can write PromQL to calculate the cost and efficiency of any Kubernetes concept and build panels in Grafana (OpenCost · exporter).

Allocation API parameters

/allocation is controlled by a handful of parameters worth knowing, because they define which number you get:

Parameter	What it controls	Example
`window`	time range	`7d`, `today`, specific dates
`aggregate`	aggregation dimension	`namespace`, `label:team`, `pod`
`accumulate`	sum the range or break it per interval	`true` (total) / `false` (series)
`idle`	include idle cost or not	`true` / `false` / `separate`
`filter`	filter by namespace, label and so on	`namespace:"llm-prod"`

The idle parameter is the most revealing: with idle=separate, OpenCost returns the idle cost as a separate row, which lets you see at a glance how much is being paid for unused capacity. A query with aggregate=label:team and idle=separate over window=30d is, literally, the monthly chargeback report with the waste highlighted.

GPU in depth: DCGM, usage, MIG and idle

The GPU is the expensive resource, so its allocation deserves detail:

Usage signal: dcgm-exporter gives DCGM_FI_DEV_GPU_UTIL (utilisation), DCGM_FI_DEV_FB_USED (memory), and power. OpenCost uses utilisation to allocate by usage.
MIG (Multi-Instance GPU): an A100 is split into up to 7 isolated instances, which allows cutting idle from 50 % to almost 0 % by sharing one card across small workloads ([search]; see sharing the GPU: time-slicing, MPS and MIG). The cost allocation of each MIG instance follows the same usage logic.
Idle: the reliable detection pattern combines three pieces, DCGM metrics in Prometheus, an alert that fires when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, and routing of that alert to the team that owns the namespace ([search]).

An attribution nuance worth knowing: MIG creates isolated instances that Kubernetes sees as distinct resources, so OpenCost allocates them cleanly, each one to its pod. Time-slicing, by contrast, shares the same physical GPU across several pods with no compute isolation: there, per-pod utilisation is harder to separate, and cost attribution becomes approximate (the GPU is split across the pods sharing it, according to their measured usage). The practical rule: if you need exact per-team chargeback on a shared GPU, MIG gives cleaner attribution than time-slicing, at the price of rigid fixed partitions. It is a trade-off between cost accuracy and flexibility that is better decided before, not after, building the split.

Efficiency: the number that triggers action

With cost allocated by usage, OpenCost lets you calculate efficiency: how much of the allocated cost corresponds to useful work against the total. A GPU at 30 % utilisation has an efficiency of 30 %: the remaining 70 % is idle cost that somebody pays without receiving work. That figure, per team and per namespace, is what turns a dashboard into a chargeback conversation: “your namespace has 5 GPUs allocated at 25 % efficiency; either you raise utilisation or you release three cards”. It connects with FinOps and multi-tenancy: allocation locates the waste; scheduling and co-residency recover it.

Worked example: the allocation report

On the ~11 €/h node (8×H100), three teams share the cluster for a month. What a /allocation?aggregate=label:team&idle=separate&window=30d query returns:

Team	GPU-hours	Avg. util.	Allocated cost (€)	Efficiency
A · chat-prod	2,880 (4 GPU)	78 %	~3,744	high
B · batch	1,440 (2 GPU)	55 %	~1,872	medium
C · experimentation	1,440 (2 GPU)	22 %	~1,872	low
idle (separate)	—	—	~1,100	—

What the report reveals: the cost allocated to A, B and C comes out of their GPU-hours and the node price; but the idle row (~1,100 €/month) is capacity nobody used and everyone pays for in the pro rata. And the efficiency column points at C: 2 GPUs at 22 % is ~1,560 € a month of which only ~340 € is useful work. That is the actionable figure: C does not have a model problem, it has a utilisation problem. Without OpenCost, those ~1,100 € of idle and C’s inefficiency stay diluted in an aggregate invoice nobody questions. With it, they are a named row and a number in euros.

Useful PromQL queries

As a Prometheus exporter, OpenCost lets you build the panels by hand. Some reference queries (exact metric names vary by version; check in your /metrics):

# Cost per node and hour (resolved price)
node_total_hourly_cost

# GPU cost allocated per namespace (€/hour)
sum by (namespace) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
)

# Average GPU utilisation per namespace (for efficiency)
avg by (namespace) (DCGM_FI_DEV_GPU_UTIL)

# Idle GPUs: utilisation < 10 % sustained
DCGM_FI_DEV_GPU_UTIL < 10

The last one, wrapped in an alert with for: 15m and routed to the namespace owner, is the standard idle detection pattern. With these series in Grafana you have cost per team, efficiency and waste on one dashboard, the “Inform report” of the FinOps phase in real time.

Dynamic cloud prices vs custom on-prem prices

The difference is worth understanding, because it changes how reliable the figure is:

Cloud (aws/azure/gcp/oracle/ovh): OpenCost obtains the price dynamically from the provider’s billing API, so the allocated cost reflects the real rate (including discounts, spot and so on) without you declaring anything.
On-prem (default/custom): you declare the price, derived from your TCO model. The advantage is total control; the risk is that a badly calculated price skews everything. That is why the TCO model (depreciated capex + opex) is not an academic exercise: it is literally OpenCost’s input.

The ovh provider among those supported is relevant for a European platform: if part of the workload goes to a European sovereign cloud, OpenCost can allocate its cost with dynamic pricing, and mix it with on-prem in the same allocation view.

From pod cost to cost per token

OpenCost gets as far as “this vLLM pod cost X €/hour”. The cost per token, the metric that compares on-prem against cloud, needs one more piece: the gateway (LiteLLM) that counts tokens per request and per team, as covered in the FinOps introduction. The full chain is: OpenCost gives the pod cost by usage → the gateway gives the tokens per team → dividing them gives the cost per token per team. OpenCost is the hardware half of that equation; without it, you would know the tokens but not their real cost. That is why A2 (allocation) is the foundation the whole cost per token of the proposal hangs from.

Deployment: Helm + Prometheus

The typical deployment is via Helm, with Prometheus as a prerequisite (for scraping and series storage). Conceptual steps:

Have Prometheus with kube-state-metrics, node-exporter, cAdvisor and dcgm-exporter.
Install OpenCost with Helm, pointing it at your Prometheus.
Configure the node price in euros (default.json / values.yaml), the step most often neglected.
Expose /metrics and build panels in Grafana with PromQL over the cost metrics.

As a Prometheus exporter, OpenCost lets you write PromQL for the cost and efficiency of any concept and create bespoke dashboards (OpenCost · exporter). It reuses infrastructure a GPU cluster already has for observability (DCGM, Prometheus, Grafana): OpenCost only adds the pricing and allocation layer.

OpenCost vs Kubecost (just the essentials)

OpenCost is the free base; Kubecost (IBM) is the commercial product built on it. Briefly, what the commercial one adds:

Capability	OpenCost	Kubecost
Cost allocation (CPU/GPU/mem/PV)	✓	✓
GPU via DCGM	✓	✓ (3.0)
Long historical retention	depends on your Prometheus	included
Automatic rightsizing	—	✓ (Turbonomic)
Governance, alerts, enterprise RBAC	basic	✓
Commercial support	community	✓ (IBM)

For most on-prem clusters with a platform team, OpenCost covers allocation; Kubecost adds value when you want automated optimisation and enterprise features without operating it yourself. The detail of the comparison, and when paying is worth it, is article A3.

Implementation checklist

For OpenCost to give a defensible per-team cost, order matters:

Step	Action	Verification
1	Prometheus with kube-state-metrics, node-exporter, cAdvisor	series present
2	NVIDIA GPU Operator + dcgm-exporter	`DCGM_FI_DEV_GPU_UTIL` in Prometheus
3	Install OpenCost (Helm) pointing at Prometheus	`/allocation` responds
4	Configure the node price in € (capex+opex)	`/allNodePricing` shows the real one
5	Label pods/namespaces by team and product	`aggregate=label:team` splits correctly
6	Grafana panels for cost, efficiency and idle	the waste is visible
7	Idle alert (`util<10` `for: 15m`) to the owner	the alert reaches the team
8	(optional) long-retention backend	months of history available

Step 4 is the one most often skipped and the one that most invalidates the result. If you only do one thing right, make it that one: without the correct node price in euros, steps 5 to 7 split a wrong cost with great precision.

Limits and traps (data-driven)

Default under-pricing. With the default pricing config, OpenCost underestimates on-prem CPU and GPU (#3781). Adjust the base prices to your real cost or the whole split will come out low.
It depends on Prometheus and DCGM. Without the right series (especially dcgm-exporter), there is no GPU allocation. Input data quality rules.
Allocation by usage ≠ total cost. A GPU at 50 % charges half; the other half is idle that still costs. Do not confuse “allocated cost” with “cost paid”.
Prometheus retention. Historical cost analysis depends on how much your Prometheus retains; for long series, plan a long-term backend.
Allocation, not per-token measurement. OpenCost reaches the pod; cost per token needs the gateway (LiteLLM) above it, as covered in the FinOps introduction.

With OpenCost’s mechanics clear, the next article (A3) compares what Kubecost and the commercial alternatives add, to decide which FinOps stack to adopt. But the foundation, correct allocation with the node price properly set in euros, is this one.

Closing

OpenCost looks like “installing a cost tool” and is, in reality, declaring what your hardware is worth and letting it split itself. That simplicity is its strength, it lives on top of the Prometheus and DCGM you already have, with no new database, and also its trap: the split is only as good as the node price you declare, and the default value under-prices on-prem GPU. Properly configured in euros, OpenCost turns the cluster’s aggregate invoice into a split per team, per product and per GPU, separates used cost from idle, and exposes the efficiency that triggers chargeback conversations. It is the hardware half of the cost per token equation, the other half, the tokens, is supplied by the gateway, and without this half there is no TCO model or on-prem vs cloud comparison that holds up. For a sovereign architecture proposal, OpenCost is the piece that makes cost stop being an intuition and become a table in euros, reproducible and auditable, with the idle GPU named and identified. The Inform phase of GPU FinOps starts here, and everything else hangs on this number being right.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · configuración (proveedores: aws/azure/gcp/oracle/ovh/default) — https://opencost.io/docs/configuration/
OpenCost · configuración on-prem (precios personalizados) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exportador de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
OpenCost · issue #3781 (infra-precio CPU/GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
NVIDIA · dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
Grafana · manage costs (OpenCost + Kubernetes) — https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/manage-costs/

GPU/LLM FinOps: frameworks, metrics and state of the art (tool by tool)

Sat, 13 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The references are European (providers and prices from FR/DE/ES), as this is a sovereign proposal; when a source quotes dollars it is marked “USD”. The dollar sign is not used (on this site it is a formula delimiter).

What this introduction covers

Second article in the data series, and the first pillar deep dive. Here the FinOps tooling for GPU/LLM infrastructure is inventoried with the detail an architecture decision needs: what metrics each tool handles, how it allocates cost internally, what GPU support it has, under what licence and pricing model it operates, and where its limits are. It is the natural continuation of the opening article, which established that cost per token is the unit that allows on-prem and cloud to be compared. No recommendations: the final choice is settled in the synthesis article with the Pareto table; here there are only facts and methodology.

Why GPU FinOps is a different problem

Classic cloud FinOps was born to split CPU, memory and storage, cheap and elastic resources. The GPU breaks three assumptions at once, and that is why it needs its own treatment:

It is expensive and discrete. An H100 costs on the order of 2.7 to 7 €/hour in European cloud (Scaleway from 2.73 €/h; OVHcloud somewhat more), or its depreciated capex on-prem; it is not a resource that gets split in trivial fractions. A 10 % allocation error across a GPU fleet is real money.
It is easily underused. Unlike the CPU, a GPU draws its power even when idle, and average occupancy in ungoverned clusters is notoriously low. The cost of idle, the GPU powered on without working, is waste number one, and it is invisible if it is not measured.
It is hard to attribute. Who owns the cost of a GPU shared by time-slicing or MIG across several pods from several teams? Without an allocation layer, GPU spend is a black hole nobody claims and nobody optimises.

The goal of GPU FinOps is to turn that black hole into an attributed, measured and actionable invoice: knowing what each team, each model and, in the end, each token costs.

The FinOps metrics

Metric	Definition	Unit
CPM (cost/1M tokens)	cluster cost ÷ tokens produced	€ / 1M tok
Cost per request	cost charged to a complete request	€ / req
€/GPU-hour	hourly cost of a GPU (depreciated or rented)	€/h
Utilisation	fraction of the GPU actually used	% (MFU, GPU-hour util)
Cost of idle	GPU powered on without useful work	€/h wasted
Showback	reporting the cost to each team (without charging)	€ / team
Chargeback	charging the cost to each team	€ / team
Cost efficiency	real cost ÷ cost if it were at 100 %	%

The metric that closes the loop with the business is cost per token (or per request): it is the only one that can be compared across providers, across models and against the price of an external API. All GPU FinOps tooling exists to reach, one way or another, that number.

The three FinOps phases (FinOps Foundation)

The FinOps Foundation framework organises the work in a three-phase cycle. Applied to GPU:

Phase	Goal	Typical GPU action
Inform	visibility and allocation	measure cost per namespace/team/model/token
Optimize	reduce spend	rightsizing, spot, quotas, switch off the idle, quantise
Operate	continuous governance	budgets, idle alerts, automated chargeback

The common mistake is jumping to Optimize without having done Inform: GPUs get bought or replicas get adjusted without knowing where the money really goes. Allocation (the Inform phase) is the prerequisite for everything else, and it is where the tooling comes in.

How cost is allocated in Kubernetes: OpenCost’s mechanics

OpenCost is the de facto standard, so it is worth understanding how it allocates cost, because that defines what any tool on top can and cannot do. It is a vendor-neutral, Apache 2.0 project, originally built by Kubecost and donated to the CNCF (incubating project) (OpenCost · GitHub).

The model works at node level: it starts from the node’s resource capacity (CPU, RAM, GPU, storage) and its total price, and splits that price across the resources. When the provider does not give explicit CPU/GPU/RAM prices, OpenCost uses the ratio of a set of base prices (the provider’s marginal rates, customisable) and normalises them so that the sum of the components equals the node’s total price (OpenCost · on-prem). This is key on-premise: you define the node cost (depreciated capex + opex) and OpenCost splits it.

It obtains utilisation by scraping Prometheus: kube-state-metrics, node-exporter and cAdvisor give it the real consumption per pod, and with that it allocates cost per cluster, node, namespace, controller, service or pod (OpenCost · exporter). For the GPU, the signal comes from DCGM (via the NVIDIA GPU Operator) exported to Prometheus, the same base as GPU observability. A common idle detection pattern: an alert when DCGM_FI_DEV_GPU_UTIL < 10 for more than 15 minutes, routed to the team that owns the namespace.

Frameworks, tool by tool

OpenCost — the allocation standard (CNCF, Apache 2.0)

What it measures: allocated cost of in-cluster resources (CPU, GPU, memory, volumes) by any Kubernetes dimension. Method: node-level pricing model + Prometheus scraping (above). GPU: yes, via DCGM. Licence: Apache 2.0, free. It can run as a metrics exporter to Prometheus with no further dependencies. Limit: it is the allocation layer, it brings no optimisation, governance or product unit economics; for that you put something on top of it.

Kubecost — the commercial product on OpenCost (IBM)

What it measures: what OpenCost does (it is built on it) plus enterprise capabilities. Kubecost 3.0 (2025) added GPU monitoring via NVIDIA DCGM and integration with IBM Turbonomic for automatic rightsizing, and extended its scope from Kubernetes to cloud service cost. IBM acquired Kubecost and integrated Kubecost/OpenCost into its FinOps Suite alongside Cloudability and Turbonomic (CloudZero · Kubecost vs OpenCost). Differentiator: rightsizing, governance, support. Limit: it is a commercial product; the bulk of the value over free OpenCost is the optimisation and enterprise layer.

CloudZero — unit economics and cost per product

What it measures: it maps cloud cost to features, products, teams and customers, not just to resources. Method: it ingests multi-cloud billing and models it in business dimensions. Differentiator: the deepest allocation and the unit economics approach (cost per business unit). Limit: less focused on intra-Kubernetes mechanics than OpenCost/Kubecost; it is the “cost to business” layer.

Vantage — multi-cloud with many integrations

What it measures: multi-cloud cost with more than 20 native integrations (AWS, Azure, GCP, Kubernetes, Snowflake, Datadog, OpenAI and so on). Differentiator: breadth of sources, including the invoice from LLM providers (OpenAI), which brings it closer to end-to-end AI cost. Limit: the depth of intra-cluster allocation is lower than that of K8s-native tools.

Finout — virtual tagging, fast deployment

What it measures: multi-cloud cost with virtual tagging: it applies cost labels without modifying the real resources, which allows spend that was not properly tagged at source to be allocated. Differentiator: fast deployment and flexible reallocation without touching the infra. Limit: like the others in its category, it depends on the quality of the billing data it ingests.

Comparison table

Tool	Scope	GPU	Licence / model	Differentiator	Layer
OpenCost	Kubernetes	Yes (DCGM)	Apache 2.0 (CNCF), free	allocation standard	resource
Kubecost	K8s + cloud	Yes (DCGM, 3.0)	commercial (IBM)	rightsizing, enterprise	resource+optim
CloudZero	multi-cloud	indirect	commercial	unit economics per product	business
Vantage	multi-cloud (20+)	via K8s/provider	commercial	breadth (incl. OpenAI)	business
Finout	multi-cloud	via K8s/provider	commercial	virtual tagging	business

Pricing models of commercial tooling

They are worth knowing because the cost of the FinOps tool is also FinOps:

Model	How it charges	Range
Savings-based	% of the savings delivered	15–35 %
Fixed-fee	% of annual cloud spend	1–3 %

(CloudZero · FinOps Tools). The implication: at large cloud spend, a 1–3 % fixed-fee can exceed the savings-based model; at small spend with a lot of waste, savings-based aligns incentives. OpenCost, being free, changes the equation for whoever has a team to operate it.

FOCUS: the cost data standard

The cross-cutting problem of multi-source FinOps is that each provider bills in its own format. FOCUS (FinOps Open Cost and Usage Specification) is the open technical specification that defines requirements so that providers produce uniform billing datasets (FOCUS · FinOps Foundation). The committee ratified FOCUS v1.3 on 4 December 2025.

What is relevant for this series: at FinOps X 2026 the focus has been put on extending FOCUS to AI workloads, with token economics pushing the specification, and the expansion requests include AI, datacenter and SaaS/PaaS workloads (SiliconANGLE). In other words, the standard that normalises cloud cost is being stretched to cover AI cost per token. For an architecture proposal, betting on tools that emit and consume FOCUS is betting on future interoperability.

From resource to token: cost per token with a gateway

Resource allocation (OpenCost) gets as far as “this vLLM pod cost X €/hour”. To get to “this request from this team cost Y” you have to intercept the inference traffic. That is where the gateway comes in.

Tools such as LiteLLM sit between the application and the LLM provider/engine, and intercept every request to record tokens, latency and cost in real time. Layers above (for instance OpenLM’s AI Gateway) generate usage logs compatible with FOCUS (v1.0 to 1.3) and map spend to team, product, customer or feature, enabling showback or chargeback (OpenLM · token attribution). This links up with FinOps and multi-tenancy of the GPU cluster, where the gateway is the piece that splits cost between tenants.

Worked example: chargeback of a multi-tenant cluster

To see the two halves joined, a split over an example node (4×H100, depreciated cost 12 €/hour) shared by three teams via namespaces and MIG:

Team	GPU-hours allocated (OpenCost)	Hardware cost (€/h)	Tokens/day (gateway)	Cost/1M tok
A · chat product	50 %	6.00	8M	~0.75 €
B · nightly batch	30 %	3.60	3M	~1.20 €
C · experimentation	20 %	2.40	0.5M	~4.80 €

How it comes out: OpenCost splits the node’s 12 €/h according to the GPU-hours each namespace consumes (half for A → 6 €/h). The gateway supplies the tokens per team. Each one’s cost per million tokens is its hardware cost divided by its output, and it reveals something neither half would see alone: team C pays 6× more per token than A, not because its model is worse, but because its GPU is underused (a lot of allocated GPU-hours for few tokens). That 4.80 €/1M is the chargeback signal that triggers an optimisation conversation: either C raises its utilisation, or it releases the GPU. Without crossing allocation and tokens, that waste stays hidden in a cluster average.

The hidden cost: utilisation

Optimisation lever number one is not changing GPU, it is stopping paying for an idle GPU. As seen in the opening article, the electricity cost per token at 80 % utilisation is a quarter of what it is at 20 %, and utilisation spreads all the fixed cost (capex + energy) over more tokens. Measuring idle is, therefore, the highest-return action of the Optimize phase:

Signal	Source	Typical threshold
Idle GPU	`DCGM_FI_DEV_GPU_UTIL`	< 10 % for > 15 min
Unused GPU memory	`DCGM_FI_DEV_FB_USED`	reserved but not used
Pods with no traffic	gateway metrics	0 requests, GPU allocated

Allocation (OpenCost) locates whose the idle GPU is; scheduling and co-residency (sharing the GPU: time-slicing, MPS and MIG) recover it. FinOps closes the loop: measure → attribute → optimise → govern.

Optimize: the saving levers (with data)

Once cost is allocated (Inform), the Optimize phase has a bounded repertoire of levers. Ordered by typical return:

Lever	Typical saving	Mechanism	Cost/risk
Recover idle	the largest	switch off/share the idle GPU	requires measuring utilisation
Quantisation (FP8/INT4)	raises throughput, lowers VRAM	more tokens per GPU-hour	possible quality cost
Reserved commitment	20–40 % over on-demand	reserve capacity	less elasticity
Spot/preemptible	the deepest discount	interruptible capacity	you have to tolerate cuts
Rightsizing	variable	match GPU type/count to the SLO	requires benchmarks
Autoscaling (HPA/KEDA)	variable	scale replicas with demand	tuning metrics

On commitment, the 2026 cloud data is conclusive: reserved plans give 20–40 % saving against on-demand, and spot the deepest discount in exchange for interruptibility; the price of the H100 fell 64–75 % between Q4 2024 and the start of 2026 (Spheron · GPU Cloud Pricing). Rightsizing and autoscaling connect with cluster scheduling: fitting the workload onto the right GPU and scaling replicas with demand are, at the same time, performance and cost levers. Quantisation appears here because it raises throughput, and, by the identity from the opening article, lowers cost and energy per token at the same time.

GPU FinOps maturity

A simple model to place where an organisation is, and what it is missing:

Level	State	What is missing to move up
0 · blind	aggregate invoice, no attribution	instrument OpenCost + DCGM
1 · visibility	cost per namespace/team	per-token measurement (gateway)
2 · unit economics	cost per token and per product	idle alerts and budgets
3 · optimisation	idle recovered, commitment, rightsizing	automated chargeback
4 · governance	chargeback + budgets + FOCUS	continuous improvement

Most organisations with GPUs are at level 0 or 1: they see a big invoice but do not know whose each euro is. The value jump is in reaching level 2, cost per token and per product, which is exactly where the tooling in this article stops being optional. The series’ FinOps track walks that ladder up to the full TCO model and the comparable cost/token that underpins the architecture proposal.

The minimum stack to reach level 2

Bringing the previous pieces together into a concrete toolchain, all open source, that takes you from blindness (level 0) to cost per token and per product (level 2):

Piece	Function	Alternative
NVIDIA GPU Operator + DCGM	exports GPU metrics (usage, memory, power)	—
Prometheus	stores the usage and cost series	VictoriaMetrics
OpenCost	allocates cost per resource and dimension	Kubecost (commercial)
Gateway (LiteLLM)	counts tokens per team/model	OpenLM AI Gateway
Grafana	cost, utilisation and idle panels	—

It is, deliberately, infrastructure many clusters already have for observability (DCGM, Prometheus, Grafana); GPU FinOps reuses that base and adds OpenCost and the gateway to it. It is not a new product, it is a layer on top of what exists.

KPIs to watch

KPI	What it indicates	Typical target
Average GPU utilisation	waste	>70–80 % sustained
Cost per 1M tokens (per model)	economic efficiency	compare against cloud rental
% of GPU-hours idle	money thrown away	minimise (<10–15 %)
Cost per team/product	attribution	fair split, no surprises
Deviation against budget	governance	alert before it is exceeded

These five KPIs are what turn GPU FinOps from a pretty dashboard into a decision tool: if you watch them, you know at all times what each thing costs and where the waste is.

State of the art 2026

Consolidation in Kubernetes: OpenCost is the CNCF allocation standard; Kubecost (IBM) the reference commercial product, already with GPU via DCGM and rightsizing through Turbonomic.
From resource to token: the strong platforms of 2026 track cost at token and GPU level, combining resource allocation (OpenCost) with gateway measurement (LiteLLM); it is the only route to a comparable cost/token for on-prem against cloud.
FOCUS v1.3 (Dec 2025) as the interoperability layer, extending to AI at FinOps X 2026: the cloud cost standard absorbs token economics.
Mature GPU FinOps: instrumentation leans on DCGM (the same base as observability), and idle detection by a utilisation threshold is common practice.

Cost per token: the bridge to the business decision

All the previous tooling exists to produce a number the business understands: cost per token (or per request). It is the one that lets you answer the three questions that underpin an architecture proposal:

Build or buy? On-prem cost/token (depreciated capex + opex, at real utilisation) against the price of an external API or of cloud rental. Below the volume threshold (~2M tokens/day) buying usually wins; above it, building.
How do you price a product? If a feature consumes N tokens per use and each million costs C, the marginal cost per use is N·C/10⁶, the basis of any margin.
Where is the waste? Cost/token per team (the chargeback example) points at whoever underuses the GPU before the aggregate invoice hides it.

The trap: comparing cost/token across scenarios without fixing the assumptions (utilisation, precision, ownership model). An on-prem cost/token calculated at 80 % utilisation is not comparable with one at 20 %, nor an FP16 with an FP8. That is why allocation (Inform) is not an end in itself: it is the raw material of a cost model with explicit assumptions, which is what the synthesis article turns into the “build vs buy” argument with defensible numbers. GPU FinOps is not accounting; it is the quantitative basis of the architecture decision.

Limits and traps (data-driven)

Allocation ≠ per-token measurement. OpenCost splits the resource cost; without a gateway you do not reach the request. They are two halves that have to be joined explicitly.
Badly set base prices = badly set allocation. On-prem, OpenCost splits the cost you declare for the node; if the depreciated capex/opex is wrong, the whole split is wrong. Input data quality rules.
Invisible idle. Without DCGM exported to Prometheus and without utilisation alerts, waste number one does not appear on any dashboard.
Cost of the tool. A 1–3 % fixed-fee on large spend is real money; compare the tooling cost with the saving it delivers (it is FinOps on top of FinOps).
Proprietary formats. Tools that do not emit/consume FOCUS tie you to their data model; in 2026 the robust bet is FOCUS interoperability.

The next article in the series (A2) goes into OpenCost in depth; this one fixes the map. With allocation settled, the rest of the FinOps track builds the TCO model and the cost/token the proposal needs.

Sources

OpenCost · GitHub (CNCF, Apache 2.0) — https://github.com/opencost/opencost
OpenCost · documentación on-prem (modelo de precio por nodo) — https://opencost.io/docs/configuration/on-prem/
OpenCost · exporter de Prometheus — https://opencost.io/docs/integrations/opencost-exporter/
CloudZero · Kubecost vs OpenCost (2026) — https://www.cloudzero.com/blog/kubecost-vs-opencost/
CloudZero · FinOps Tools: Definitive Guide (2026) — https://www.cloudzero.com/blog/finops-tools/
FOCUS · especificación (FinOps Foundation) — https://focus.finops.org/focus-specification/
SiliconANGLE · FOCUS y la economía de tokens de IA (FinOps X 2026) — https://siliconangle.com/2026/06/08/focus-specification-ai-cost-accountability-finopsx/
OpenLM · atribución de tokens en tiempo real (LiteLLM + FOCUS) — https://www.openlm.com/enable-ai-finops-with-real-time-token-attribution/
Finout · Best AI Cost Observability Tools (2026) — https://www.finout.io/blog/best-ai-cost-observability-tools-in-2026

FinOps and multi-tenancy on the GPU cluster: who pays for what

Thu, 11 Jun 2026 12:20:00 +0000

Third instalment in an operational series on how to squeeze a generic on-premise LLM inference cluster of 4×H100 SXM 80 GB. The sibling pieces are Multimodal VLM on-premise with vLLM, serving vision-language models on the same cluster, and Speeding up cold start with Tensorizer, cutting the model load time from disk to HBM. This post closes the economic question the other two leave open: once the cluster serves several models for several teams, who pays for what?

TL;DR

An on-premise GPU cluster does not have the property that makes FinOps easy in the cloud: no monthly invoice arrives telling you what each compute hour cost. The cost has to be manufactured from capex and energy. The right unit is the token, and it is derived in two steps. First the €/GPU-hour: capex depreciation spread over the hours of useful life, plus energy (power × PUE × price_kWh). For an H100 SXM with an assumed capex of 30,000 € and 4 years of life at 90 % availability, that gives ≈ 1.07 €/GPU-hour (0.95 of depreciation + 0.12 of energy at 0.12 €/kWh and PUE 1.4). Second, the €/1M tokens: divide that €/GPU-hour by the sustained throughput. At 2,000 useful tok/s, ≈ 0.148 €/1M tokens; but that number only holds if the GPU is at 100 % useful utilisation. At 20 % utilisation the same token costs 4× more (0.74 €/1M), because capex and energy keep being paid even if the GPU is idle. That is the central thesis: the FinOps goal is not to negotiate the price of the token down, there is no provider to negotiate with, but to raise useful utilisation, which is the only term that moves cost by an order of magnitude. Attribution (chargeback / showback) is made real with LiteLLM virtual keys: per-team and per-user budget, RPM/TPM rate limit, automatic spend tracking and tags to assign spend to cost centres. Isolation between tenants is achieved with MIG (hard GPU partitioning), namespaces, ResourceQuota and PriorityClass. The FinOps Foundation rule worth carving in stone: showback always, chargeback only if the organisation’s accounting policy supports it; neither of the two is “more mature”. And the governance metric of the shared cluster is not the €/token but DCGM_FI_DEV_GPU_UTIL.

The problem the cloud hides from you and on-premise forces you to solve

In the cloud, the cost of a GPU is an observable fact: AWS, GCP or Azure charge you per GPU-hour and the invoice arrives itemised. FinOps in that world consists of reading an invoice that already exists and attributing it. On-premise there is no such invoice. The GPU was bought once, eighteen months ago, on a purchase order nobody remembers; the electricity bill arrives for the whole building with no breakdown of what fraction was the cluster’s; and the datacenter PUE is a number the facilities team knows but the AI team never asked about.

The first job of on-premise FinOps, then, is not to attribute a cost but to manufacture one. You have to build, from explicit assumptions, the equivalent of the “price per GPU-hour” that in the cloud is given to you. And once manufactured, you have to understand that this number is not a property of the hardware, like VRAM or TDP, but a function of how the hardware is used. An idle H100 costs exactly the same as an H100 at 100 %: the capex is already paid and idle energy is not zero. The only thing that changes is how many useful tokens that fixed cost produced. That is where the whole thesis of this post comes from.

The analogy: the coworking space with metered services

A coworking space manages a finite physical space and rents it to teams. The cost model has exactly the structure of the GPU cluster, and the analogy holds down to the last detail.

The desk-hour is the spread capex. The manager paid the rent on the premises, the furniture and the refurbishment, a one-off outlay or a fixed monthly cost, and spreads it over the available desk-hours in the month. Each desk has a base cost that does not depend on whether anybody uses it: the walls, the table and the chair cost the same empty as occupied. This is the depreciation of the GPU: capex divided by the hours of useful life.

The metered kWh is the energy. On top of the desk-hour, the coworking space measures the actual electricity consumption, the plugged-in laptop, the screen, the air conditioning in that room, and bills it separately. This is the energy of the GPU: power × hours × price_kWh, multiplied by the datacenter’s PUE, which is the coworking space also charging you for the fraction of the building’s air conditioning that cools your room.

The empty desk is paid for by whoever booked it. Here is the heart of the matter. If a team books ten desks for the whole month and only uses two, it pays for all ten anyway. The empty desk still takes up space the manager cannot resell, still depreciates furniture, still counts as committed capacity. In the cluster: a GPU assigned to a tenant that uses it at 20 % costs the same as if it used it at 100 %, and the idle 80 % is wasted capex that somebody is paying for.

The manager optimises occupancy, not tariff. A competent coworking manager does not obsess over lowering the price of the desk-hour, it is a sunk cost, the premises are already paid for. They obsess over the occupancy rate: every empty desk is margin lost forever, because an unsold desk-hour cannot be stored. The FinOps equivalent: the goal is not to lower the €/token, it is to raise utilisation, because the €/token falls only as a consequence of utilisation rising. There is no provider to squeeze; the only margin lies in not leaving GPUs idle.

The analogy has an honest limit worth naming: in a coworking space, two people cannot physically share the same chair. On a GPU you can, with time-slicing, MPS or MIG (see Sharing a GPU), and that is precisely the technical lever that lets you raise occupancy beyond what the physical coworking space would allow. But the economics of fixed cost + metered cost is identical.

The token as the unit of cost

Why the token and not the GPU-hour, or the request, or the model? Because the token is the only unit comparable across heterogeneous workloads. A GPU-hour of Llama 8B and one of Llama 70B produce radically different amounts of “useful work”; a RAG request with a 50-token answer and one generating a 3,000-token report are not comparable. The token, specifically the pair (input_tokens, output_tokens), normalises all of that. It is the unit LiteLLM accounts for natively, the one that appears in the OTel attributes gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (see LLM tracing with OpenTelemetry GenAI), and the one commercial providers already use to charge. Adopting the token as the internal currency makes on-premise chargeback directly comparable with the cloud alternative, which is exactly the comparison management wants to see.

The two derived metrics that matter:

Tokens/€ — how many useful tokens each euro of total cluster cost buys. It rises when utilisation rises. It is the metric of economic efficiency.
Tokens/W — how many tokens each watt consumed produces. It is the metric of energy efficiency and, in a datacenter with limited power (the usual case on-premise), it is often the real hard constraint: you cannot add more GPUs because there are no more kW in the rack, so every watt has to perform.

Cost model of a GPU: from capex to €/token

The hourly cost of a GPU has two addends. Let us put all the assumptions in writing, because, as in capacity planning, a cost calculation without written assumptions is a disposable calculation.

Depreciation (spread capex)

$$\text{depreciation cost/h} = \frac{\text{capex}_{\text{GPU}}}{\text{hours of useful life}}$$

where the hours of useful life are the depreciation years × 8,760 h/year × the effective availability (the GPU is not available 100 % of the time: there is maintenance, restarts, update windows). Explicit, generic assumptions (they are not figures from any provider or any real purchase):

Capex per GPU: 30,000 € — a reasonable assumption that includes not just the GPU but its pro-rata share of the server, power supply, cooling, NVLink/InfiniBand network and rack. A “bare” H100 SXM costs less, but honest FinOps spreads the cost of the whole node across its GPUs.
Accounting useful life: 4 years. That is aggressive-realistic for a datacenter GPU; some depreciate over 3, others over 5.
Effective availability: 90 %.

$$\text{hours of life} = 4 \times 8{,}760 \times 0.90 = 31{,}536 \text{ h}$$ $$\text{depreciation cost/h} = \frac{30{,}000}{31{,}536} \approx 0.95 \text{ €/GPU-hour}$$

Energy (power × PUE × price_kWh)

$$\text{energy cost/h} = P_{\text{GPU}} \cdot \text{PUE} \cdot \text{price kWh}$$

Assumptions:

GPU power: the TDP of the H100 SXM is 700 W according to the NVIDIA datasheet (H100 datasheet). Under sustained inference load it hovers around that value; we use 0.7 kW as the average power under load. (At idle it drops a lot, 60–100 W, but the FinOps of reserved capacity reasons about power under load, which is what limits the rack.)
PUE (Power Usage Effectiveness): how many watts enter the datacenter for every watt that reaches the chip. A modest, well-managed on-premise datacenter sits at 1.4; a bad one at 2.0; the hyperscalers boast of 1.1. We use 1.4.
Price per kWh: 0.12 €/kWh — a generic assumption for an industrial tariff; it varies enormously by country and contract.

$$\text{energy cost/h} = 0.7 \cdot 1.4 \cdot 0.12 \approx 0.1176 \approx 0.12 \text{ €/GPU-hour}$$

The total €/GPU-hour

$$\boxed{\text{€/GPU-hour} = 0.95 + 0.12 \approx 1.07 \text{ €/GPU-hour}}$$

Two important readings of this number. First: depreciation dominates (89 % of the cost); energy is 11 %. This inverts many people’s intuition, who believe “the expensive part is the electricity”. With these assumptions, the expensive part is having bought the GPU, and that is why leaving it idle hurts so much. Second: the number is per-GPU; for the 4×H100 node it is ≈ 4.28 €/hour, and for the complete generic cluster of 4 nodes (16 GPUs), ≈ 17.1 €/hour ≈ 150,000 €/year of fixed cost that is paid whether it is used or not.

From €/GPU-hour to €/token

Now we divide the hourly cost by the tokens the GPU produces in an hour. Here sustained throughput comes in. Let us suppose, an order-of-magnitude figure, checkable with vllm bench serve, that a Llama 70B replica on 4×H100 (TP=4) sustains 2,000 useful tokens/s aggregated under good concurrency.

$$\text{tokens/hour} = 2{,}000 \times 3{,}600 = 7.2 \times 10^6 \text{ tok/h}$$

The cost of that replica (4 GPUs) is $4 \times 1.07 = 4.28$ €/hour. Therefore:

$$\text{€/1M tokens} = \frac{4.28}{7.2} \approx 0.59 \text{ €/1M tokens}$$

Per individual GPU, if we take a smaller replica (e.g. Llama 8B FP8 on 1 GPU at ~2,000 tok/s):

$$\text{€/1M tokens} = \frac{1.07 \text{ €/h}}{7.2 \times 10^6 \text{ tok/h}} \times 10^6 \approx 0.149 \text{ €/1M tokens}$$

These numbers are the theoretical floor at 100 % utilisation. Nobody operates at 100 %. And that is the whole story.

Utilisation is the lever (with numbers)

The €/token in the previous section assumes the GPU produces 7.2 M tokens every hour, hour after hour. In practice it produces that only during peak hours. The rest of the time it is partly idle: at night, at weekends, between traffic peaks. The useful utilisation $u$ is the fraction of the theoretical capacity that really turns into billable tokens.

The hourly cost is fixed (1.07 €/GPU-hour is paid come rain or shine), but the tokens produced scale with $u$:

$$\text{€/1M tokens}(u) = \frac{\text{€/GPU-hour}}{\text{max tok/hour} \cdot u} = \frac{1.07}{7.2 \times 10^6 \cdot u} \times 10^6 = \frac{0.149}{u}$$

The requested contrast, worked out:

Useful utilisation $u$	Real tokens/h	€/1M tokens	Multiplier vs 80 %
100 %	7.2 M	0.149 €	0.80×
80 %	5.76 M	0.186 €	1.00× (baseline)
50 %	3.6 M	0.298 €	1.60×
20 %	1.44 M	0.744 €	4.00×
10 %	0.72 M	1.488 €	8.00×

Direct reading: going from 20 % to 80 % utilisation divides the cost per token by four ($0.744 \to 0.186$). No tariff negotiation, no quantization, no hardware change gives you that factor of 4 so cheaply. Quantizing from BF16 to FP8 can double throughput, and therefore halves the €/token, but it degrades quality and demands evals (see Quantization for inference); raising utilisation from 20 to 80 % does not touch the quality of a single token.

This reformulates GPU cluster FinOps in a single sentence: the €/token is not something you negotiate, it is something you earn by filling the GPUs. And raising utilisation is an engineering problem that already has its pieces:

Autoscaling that switches replicas off when traffic drops, so as not to pay for idle GPU (see LLM autoscaling on Kubernetes with KEDA). Switching off a 4-GPU replica for eight hours overnight saves ≈ 34 € a day of real cost; multiplied by replicas and by days, that is the difference between a profitable cluster and one that bleeds.
Sharing a GPU between small workloads with MIG or time-slicing, so that two tenants that individually would use 30 % together fill a GPU to 60 % (see Sharing a GPU).
Overnight batch that fills the off-peak hours with non-urgent work (re-embeddings, evals, light fine-tuning) instead of leaving the GPUs switched off or ticking over.
Reducing cold start so that scaling to zero and starting again is cheap and therefore viable as a utilisation strategy (see the sibling piece Speeding up cold start with Tensorizer).

The diagram: the €/token cascade by utilisation

The cost per token is not a number, it is a cascade that multiplies as utilisation falls:

Attributing the cost: LiteLLM virtual keys

With the €/token manufactured, it remains to attribute it to whoever consumed it. The piece that already sits in front of the engines, the L7 inference router, is also the natural place to count. LiteLLM Proxy makes chargeback real with four mechanisms, all documented:

Virtual keys with a budget. Each team or user receives a virtual key with a max_budget and a budget_duration. You can define several independent budget windows (for example one of 24 h and another of 30 d) that reset on their own cycle (Virtual Keys). When a team exhausts its monthly token budget, the proxy rejects with a budget exceeded error instead of carrying on spending GPU-hours nobody will be able to charge anywhere.

RPM/TPM rate limit. Each key carries requests per minute (RPM) and tokens per minute (TPM) limits (Budgets, Rate Limits). This is not just anti-abuse protection: it is the tool that stops a noisy tenant monopolising the cluster’s utilisation at everybody else’s expense. The rate limit is the capacity quota; the budget is the spend quota. They are orthogonal and you use both.

Automatic spend tracking. LiteLLM accounts for the spend of all known models by key, user and team, recording the API key, the user, the team_id, the request’s tags, the end user, the model group and the token counts (Spend Tracking). For the cluster’s self-hosted models, the cost per token is configured with the €/token we manufactured above, and that is the hook between the cost model and the real accounting.

Tags for cost centres. The tags let you track spend and set budgets per label, categorising costs by project, department or cost centre (Setting Tag Budgets). A tag is attached when the key is created and every request made with that key inherits the tag automatically; the proxy applies the label’s budget. That way chargeback maps cleanly onto the organisation’s accounting hierarchy without every team having to remember to tag by hand.

A declarative catalogue fragment, which fits into the litellm-config of the router post, materialising the manufactured cost and two tenants:

litellm_settings:
 # €/token manufactured in this post for the self-hosted model.
 # input/output in €/token (not per 1M); 0.186 €/1M = 1.86e-7 €/token at u=80%.
 model_cost_map:
 llama-70b-onprem:
 input_cost_per_token: 0.000000186
 output_cost_per_token: 0.000000186

# Virtual keys per team (via the /key/generate API or config):
# equipo-datos: max_budget=500 €/month, tpm_limit=200000, tags=["cc-datos"]
# equipo-soporte: max_budget=150 €/month, tpm_limit=60000, tags=["cc-soporte"]
# The proxy counts tokens, multiplies by model_cost_map, deducts from the budget,
# and attributes the spend to the tag → showback per cost centre, without touching the engine.

The complete attribution flow, from the request to the per-team report:

Showback vs chargeback: the distinction the FinOps Foundation insists on

The two words are constantly confused and the difference is one of accounting formality, not technology:

Showback — giving each team visibility of what it consumed and its cost, without really billing it to their budget. The report arrives, the team sees it, but the money does not move between cost centres.
Chargeback — really transferring the cost to the team’s or product’s budget, putting it in their P&L. The spend stops being a central IT cost and becomes an allocated cost.

The FinOps Foundation is explicit on two points worth carving in stone (Invoicing & Chargeback, Chargeback vs Showback). First: showback is a requirement of any FinOps practice; chargeback depends on the organisation’s accounting policy. Not all organisations can or want to move money between departments for GPU consumption; showback you can always do. Second, and counterintuitive: neither of the two is “more mature” than the other. The narrative that chargeback is the “grown-up” version of showback is false according to the framework itself. The practical recommendation: start with showback, giving visibility, then build cost allocation aligned to the organisational hierarchy, and only then, if the accounting policy supports it, switch chargeback on.

For the on-premise GPU cluster this means: the LiteLLM machinery (keys, tags, spend tracking) always produces the showback. Turning it into chargeback is a decision for finance, not for the platform team. The platform team guarantees that the numbers are correct and reproducible; who really pays is a governance decision.

Isolation: making the cost genuinely attributable

Chargeback is only honest if consumption is isolable. If two tenants share a GPU without partitioning and one saturates the HBM, the other suffers degradation it did not cause but which contaminates its cost attribution. Isolation has two planes.

Hard GPU isolation: MIG. Multi-Instance GPU physically partitions an H100 into up to 7 instances with dedicated memory and SMs; in a 7-way partition each instance has ~10 GB of HBM3 and its own SMs. MIG gives the strongest isolation: tenant A in its MIG instance cannot touch tenant B’s performance, and attribution is trivial because each instance is accountable separately (see Sharing a GPU for the detail of MIG vs time-slicing vs MPS). The cost: MIG instances are fixed, they are not resized hot, and if they sit empty they are fragmented and wasted capex, the coworking desk subdivided into booths nobody rents.

Logical Kubernetes isolation. On top of the cluster:

Namespaces per tenant — the boundary for RBAC, NetworkPolicy and quotas.
ResourceQuota — limits how many nvidia.com/gpu (or nvidia.com/mig-1g.10gb) a namespace can request. It is the GPU capacity quota at the scheduler level: the tenant cannot claim more GPUs than its quota grants, which bounds its maximum cost by construction.
PriorityClass — defines which workloads can evict others under pressure. The overnight re-embeddings batch runs with low priority and gives way to interactive inference; that way it fills the utilisation valleys without risking the paying tenant’s SLO. It is the piece that stops “raising utilisation with batch” from cannibalising quality of service.

The combination that works in the generic cluster: MIG to partition the GPUs between tenants that need hard isolation and clean attribution, ResourceQuota per namespace to bound each tenant’s cost, and PriorityClass so that filler work raises utilisation without touching the priority tenants.

Applied to the generic 4×H100 cluster

Let us bring it all down to the series’ cluster: 4 nodes × 4×H100 SXM 80 GB, 16 GPUs, ≈ 17.1 €/hour of fixed cost (≈ 150,000 €/year). Three teams share it: Data (production RAG, office-hours traffic), Support (ticket assistant, daytime peaks) and Platform (evals and re-embeddings batch, no urgency).

Splitting the cost. The cluster’s fixed cost (150,000 €/year) is split through LiteLLM’s spend tracking in proportion to the tokens consumed by each team, valued at the manufactured €/token. If in one month Data consumed 1,700 M tokens, Support 480 M and Platform 320 M, at 0.186 €/1M the showback is ≈ 316 € / 89 € / 60 €. The residue, the fixed cost of the GPUs nobody used because average utilisation was, say, 45 %, is the cost of idleness, and the governance decision is whether it is split between the tenants (penalises everybody equally) or charged to Platform as “cost of unsold capacity” (gives Platform an incentive to raise utilisation). The FinOps Foundation would say: make it visible first (showback of the cost of idleness), decide the split afterwards.

Quotas per tenant. Each team has its namespace with ResourceQuota: Data can claim up to 8 GPUs (2 replicas at TP=4), Support up to 4, Platform up to 4 but with a low PriorityClass, giving up its GPUs when Data or Support need them at peak. In LiteLLM, each team has its virtual key with a monthly max_budget and a tpm_limit that reflects its capacity quota.

MIG to isolate and attribute. For the small models (embeddings, reranker, a utility Llama 8B), an H100 split 7-way with MIG gives seven separately attributable instances: three for Data’s embeddings, two for Support’s reranker, two free for filler. Each instance is an independent accounting “desk”; the per-tenant showback comes straight from the scheduler.

The governance metric is utilisation, not the €/token. Here we close the circle. The KPI the cluster owner should look at daily is not “what the token costs”, that number only falls as a consequence, but DCGM_FI_DEV_GPU_UTIL, the DCGM exporter metric that indicates what fraction of the time the GPU is not idle (DCGM exporter). And with a critical nuance that already appeared in capacity planning: DCGM_FI_DEV_GPU_UTIL measures “not idle”, it does not measure useful work. A GPU can show 100 % GPU-util with the HBM saturated and low SM occupancy, producing few tokens. That is why serious FinOps crosses three signals: DCGM_FI_DEV_GPU_UTIL (is the GPU busy?), DCGM_FI_PROF_SM_OCCUPANCY and DCGM_FI_PROF_GR_ENGINE_ACTIVE (is it doing real work?), and vllm:gpu_cache_usage_perc (is the HBM well used?). The economic governance metric is useful tokens per GPU-hour, and it is watched from the GPU observability with DCGM dashboard.

An end-to-end assembly detail, LibreChat on top of LiteLLM with RAG, which closes the flow from tenant to cost, is covered in the (forthcoming) post on the end-to-end sovereign assistant; here it is enough to know that the metering point is always the proxy, never the engine.

Four traps of on-premise GPU FinOps

Trap 1 — comparing the on-premise €/token with the commercial API’s list price. The commercial API has margins, SLA and scale the on-premise cluster does not, but on-premise has the cost of idleness the API hides from you (they fill their GPUs with thousands of customers). The honest comparison is on-premise at its real utilisation vs the API at its real negotiated price, not the fantasy of on-premise at 100 %.

Trap 2 — forgetting the cost of idleness in the showback. If you only charge teams for the tokens they consumed, the cost of the idle GPUs disappears from the report and nobody sees it. That cost exists and somebody is paying it. Making it visible is the first step to reducing it.

Trap 3 — confusing high GPU-util with efficiency. A GPU at 100 % DCGM_FI_DEV_GPU_UTIL with the HBM saturated and low SM occupancy spends capex without producing proportional tokens. The goal is not “GPU at 100 %”, it is “useful tokens per euro”. Always cross util with SM occupancy and real throughput.

Trap 4 — chargeback before showback. Switching chargeback on, moving money, before the teams trust that the numbers are correct generates disputes that burn the whole FinOps programme. First visibility, then trust in the data, then, if the accounting policy allows it, the money.

References

NVIDIA — H100 Tensor Core GPU Datasheet (TDP 700 W, HBM3 3,35 TB/s): resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306.
LiteLLM — Virtual Keys (docs.litellm.ai/docs/proxy/virtual_keys), Budgets & Rate Limits (/docs/proxy/users), Spend Tracking (/docs/proxy/cost_tracking), Setting Tag Budgets (/docs/proxy/tag_budgets), Team Budgets (/docs/proxy/team_budgets).
FinOps Foundation — Invoicing & Chargeback Capability (finops.org/framework/capabilities/invoicing-chargeback/) y Data Analysis and Showback (finops.org/framework/previous-capabilities/analysis-showback/).
CloudZero — Chargeback vs. Showback: Cloud Cost Allocation Models Explained (cloudzero.com/blog/chargeback-vs-showback/).
NVIDIA — DCGM Exporter (docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html): DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_GR_ENGINE_ACTIVE.

The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

Sat, 06 Jun 2026 08:00:00 +0200

The close of the “under the engine” series. We saw the cable between GPUs and the host: NUMA, hugepages and CPU isolation done by hand. Here is the piece that makes it declarative and at scale: how the RKE2 kubelet pins each vLLM pod to the right NUMA node without a single script.

TL;DR

Pinning NUMA with numactl/isolcpus/taskset, the subject of the previous post, does not scale to a cluster where pods come and go and there are dozens of nodes. The kubelet automates it with three components that work as Hint Providers for a central coordinator, the Topology Manager: the CPU Manager (assigns exclusive CPUs to containers of Guaranteed pods with whole CPUs), the Memory Manager (NUMA-local memory and hugepages) and the Device Manager/GPU plugin (which knows which GPU is on which NUMA node). With the single-numa-node policy, the Topology Manager only admits the pod if its CPUs, its memory and its GPU fit in the same NUMA domain; if they do not fit, it rejects the pod, strict admission, like the maître d’ who will not seat a party of eight unless there is a table for eight. On RKE2 all of this is configured with kubelet-arg in /etc/rancher/rke2/config.yaml. This post explains the mechanism, gives the 10 knobs and takes apart the gotchas that break pinning silently: the cpu_manager_state file you have to delete when changing policy, the QoS that has to be exactly Guaranteed, and the reserved-cpus that must match the host’s isolcpus. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the orchestration that materialises the host

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

A restaurant has tables of different sizes and, house rule, tables are never pushed together. A party of eight arrives. The maître d’ looks for a single table where all eight fit. If there is one, he seats them; if only tables for four are left, he does not accept them. He would rather turn the booking away than seat the group split across two separate tables, because he knows a split dinner goes badly.

That maître d’ is the Topology Manager under the single-numa-node policy. The “party” is an inference pod asking for CPUs, memory and a GPU. The “table” is a NUMA node. The maître d’ asks three assistants (are there free CPUs on any node? the CPU Manager; is there free memory? the Memory Manager; is there a free GPU? the Device Manager) and only admits the pod if all three resources fit on the same node. If not, he rejects it (the pod ends up Failed with TopologyAffinityError), and the scheduler will try another node.

The difference from the previous post: there you were the maître d’, seating each process by hand with numactl. Here the maître d’ is the kubelet, and it does it for every pod, on every node, automatically, rejecting whatever does not fit. That is what turns artisanal pinning into a declarative property of the cluster.

The mechanism: Hint Providers and the coordinator

The Topology Manager does not assign resources; it coordinates the ones that do. The flow, when a Guaranteed pod arrives at a node:

The three managers are Hint Providers: each one tells the Topology Manager on which NUMA node(s) it could satisfy its part. The Topology Manager computes the intersection and, depending on the policy, decides:

none (default): no coordination; each manager does its own thing without aligning. No NUMA guarantee.
best-effort: it tries to align on one node; if it cannot, it admits anyway (on whichever node). Better than nothing, no guarantee.
restricted: if it fails to align, it rejects the pod. Strict, but it allows multi-node affinity if the intersection provides it.
single-numa-node: it requires everything to fit on a single NUMA node, or it rejects. The strictest one, and the one that genuinely guarantees the locality of the previous post.

And two preconditions without which none of this kicks in:

The pod has to be QoS Guaranteed: requests == limits on CPU and memory, and whole CPUs (not 500m). Only then does the CPU Manager assign exclusive CPUs.
The CPU Manager has to be on the static policy (not none).

Without those two, the Topology Manager has nothing to align and pinning does not happen, even with the policy in place. That is gotcha no. 1.

How it is configured on RKE2

RKE2 passes arguments to the kubelet with the kubelet-arg key in /etc/rancher/rke2/config.yaml. The reference configuration for GPU inference nodes:

# /etc/rancher/rke2/config.yaml (on every agent node with GPUs)
kubelet-arg:
 - "cpu-manager-policy=static"
 - "topology-manager-policy=single-numa-node"
 - "topology-manager-scope=pod"
 - "memory-manager-policy=Static"
 - "reserved-cpus=0-1,64-65" # housekeeping; must match the host
 - "system-reserved=memory=8Gi"
 - "kube-reserved=memory=4Gi"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi" # required by Memory Manager Static
node-label:
 - "fibercli.local/numa-pinned=true"

After deploying it: systemctl restart rke2-agent. Critical gotcha: if the node has already run with cpu-manager-policy=none, there is a state file /var/lib/kubelet/cpu_manager_state that pins the old policy; changing the arg without deleting that file makes the kubelet fail to start or ignore the new policy. You have to stop the agent, rm /var/lib/kubelet/cpu_manager_state, and start again. (The same applies to memory_manager_state.)

And the vLLM pod, to be eligible, Guaranteed with whole CPUs:

resources:
 requests:
 cpu: "16" # whole, not a weird fractional "16000m"
 memory: "200Gi"
 nvidia.com/gpu: "2" # TP=2 → 2 GPUs on the same NUMA node
 hugepages-1Gi: "16Gi"
 limits:
 cpu: "16" # == requests → Guaranteed QoS
 memory: "200Gi"
 nvidia.com/gpu: "2"
 hugepages-1Gi: "16Gi"

With this, on a node with the config above, the kubelet assigns 16 exclusive CPUs from the NUMA node where the 2 requested GPUs live, its local memory and the hugepages, or it rejects the pod if they do not fit together. The artisanal pinning of the previous post, now declarative.

The 10 knobs to reach for

Ordered by dependency (the first ones are a precondition for the later ones). The canonical reference is the Kubernetes Topology Manager documentation and the RKE2 configuration.

Knob 1 — `cpu-manager-policy=static`: the foundation

Without this there are no exclusive CPUs and nothing else kicks in.

kubelet-arg: [ "cpu-manager-policy=static" ]

Gotcha: changing it requires deleting /var/lib/kubelet/cpu_manager_state and restarting the kubelet, or startup fails. It is cause no. 1 of “I set the policy and it does not pin”.

Knob 2 — `Guaranteed` QoS + whole CPUs: the pod’s precondition

This is not node config, it is pod config, but without it knob 1 does nothing for that pod. requests == limits on CPU and memory, and whole CPUs. A cpu: 500m or a requests != limits downgrades the pod to Burstable and it loses the pinning. Plenty of people set the node policy and forget the pod’s QoS.

Knob 3 — `topology-manager-policy=single-numa-node`: strict admission

kubelet-arg: [ "topology-manager-policy=single-numa-node" ]

The strict maître d’. For GPU inference it is the right policy: it guarantees that CPU+memory+GPU share a node. best-effort guarantees nothing (it admits misaligned); restricted allows multi-node affinity. Start with single-numa-node and drop to restricted only if you have admission problems.

Knob 4 — `topology-manager-scope=pod`: group the whole pod

kubelet-arg: [ "topology-manager-scope=pod" ]

With container scope (the default), each container is aligned separately; with pod scope, the whole pod goes to the same node. For a vLLM pod with sidecars (metrics, proxy), pod scope stops the sidecar from dragging the main container onto another node. Recommended for inference.

Knob 5 — `memory-manager-policy=Static` + `reserved-memory`: NUMA-local memory

kubelet-arg:
 - "memory-manager-policy=Static"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi"

Memory Manager Static forces cpuset.mems so that the pod’s memory comes out of the right node (and so do the hugepages). It requires declaring reserved-memory per node, or the kubelet does not start. It is the declarative equivalent of the previous post’s --membind.

Knob 6 — `reserved-cpus`: the housekeeping cores (must match `isolcpus`)

kubelet-arg: [ "reserved-cpus=0-1,64-65" ]

It reserves cores for the system and the daemons; the rest are left for exclusive pods. Key point of the series: these reserved-cpus must be the same cores you left out of isolcpus on the host (previous post). If the host isolates 2-31 but RKE2 reserves 0-3, there is a mismatch: isolated cores that the kubelet assigns to pods without them actually being quiet. Coordinate the two layers.

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

The Device Manager can only give a correct NUMA hint if the GPU plugin exposes which node each GPU is on. The NVIDIA device plugin / GPU Operator does this, but you have to verify that the topology information arrives (in some versions it requires flags). Without a GPU hint, the Topology Manager aligns CPU and memory but not the GPU, and GPU locality is precisely the one that matters most.

Knob 8 — hugepages as a pod resource

resources:
 limits:
 hugepages-1Gi: "16Gi" # the node must have them pre-reserved (post 2, knob 4)

The hugepages you reserved at host boot (previous post) are requested as a resource. The Memory Manager assigns them NUMA-local. If you request them without having reserved them on the node, the pod does not get scheduled.

Knob 9 — `system-reserved` / `kube-reserved`: do not oversubscribe

kubelet-arg:
 - "system-reserved=cpu=500m,memory=8Gi"
 - "kube-reserved=cpu=500m,memory=4Gi"

It reserves resources for the system and the K8s components so the node does not run out of air under load. Badly calibrated, either the node chokes (too little reserved) or you waste capacity (too much). It must be consistent with reserved-cpus.

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

# GPU node: taint to repel anything that does not need a GPU
node-taint: [ "nvidia.com/gpu=present:NoSchedule" ]
node-label: [ "fibercli.local/numa-pinned=true" ]

Keep the NUMA-pinned nodes for inference and push out whatever does not need it (databases, the Langfuse backend, runners). A ClickHouse stealing memory bandwidth from a carefully pinned vLLM pod throws away all the work of the previous nine knobs. Workload isolation is the closing move.

Summary table

#	Knob	Where	Function
1	`cpu-manager-policy=static`	kubelet-arg	exclusive CPUs (foundation)
2	`Guaranteed` QoS + whole CPUs	pod spec	pinning precondition
3	`topology-manager-policy=single-numa-node`	kubelet-arg	strict NUMA admission
4	`topology-manager-scope=pod`	kubelet-arg	group the whole pod
5	`memory-manager-policy=Static`	kubelet-arg	NUMA-local memory/hugepages
6	`reserved-cpus`	kubelet-arg	housekeeping (match isolcpus)
7	GPU plugin with topology	GPU Operator	NUMA hint for the GPU
8	`hugepages-1Gi`	pod spec	hugepages as a resource
9	`system/kube-reserved`	kubelet-arg	do not oversubscribe
10	taints + labels	node config	isolate GPU workloads

Verifying that the pinning really happened

Do not trust that the config “is in place”. Check:

# Is the active policy the one you set?
cat /var/lib/kubelet/cpu_manager_state | jq .policyName # "static"

# Which exclusive CPUs does the container have?
kubectl exec <pod> -- cat /sys/fs/cgroup/cpuset.cpus.effective

# Inside the pod: is the assigned GPU local to those cores?
kubectl exec <pod> -- nvidia-smi topo -m

# Were there topology rejections?
kubectl describe pod <pod> | grep -i TopologyAffinityError

A pod in Failed with TopologyAffinityError is not a bug: it is the maître d’ doing his job. That node did not have a table where CPU+memory+GPU fitted together. The answer is to review the sizing of the pod or the node, not to relax the policy casually.

How it connects with the rest of the stack

With the host (previous post). This post is the declarative automation of that one. cpu-manager-policy=static materialises the taskset; memory-manager-policy=Static materialises the --membind; reserved-cpus must match isolcpus. The two layers are one single decision seen from two places: the host executes it, the kubelet declares it. Uncoordinating them (isolcpus 2-31 vs reserved-cpus 0-3) breaks both.

With the interconnect (post 1). The Topology Manager pins the right GPU to the pod, but if you ask for 2 GPUs for TP=2, you will want those two to share NVLink. The NUMA policy guarantees they are on the same socket; that they are NVLink-connected is guaranteed by the baseboard hardware (post 1, knob 1). The two things together are what makes TP=2 perform.

With autoscaling. When KEDA scales vLLM pods, each new replica goes through Topology Manager admission. If the node has no free “table”, the pod stays pending. Pod autoscaling and node autoscaling (cluster-autoscaler) have to account for NUMA granularity, not just aggregate CPU/memory.

With capacity planning. Sizing changes: it is not “128 vCPU per machine”, it is “128 minus the reserved-cpus, in blocks that fit per NUMA node”. A machine with 2 sockets × 64 cores does not serve a pod asking for 80 cores under single-numa-node: they do not fit at one table. Planning has to reason per NUMA node, not per machine.

With service coexistence. The knob 10 taint is what keeps Langfuse, databases and runners off the inference nodes. Without that boundary, all the fine-grained pinning gets eaten by a noisy neighbour. Observability goes on its own nodes; inference, pinned, on its own.

Traps and things that are not what they seem

Policy in place, QoS forgotten. The most common mistake: cpu-manager-policy=static on the node but the pod is Burstable (requests != limits or fractional CPU). Pinning does not happen and nobody warns you. Guaranteed QoS with whole CPUs is a necessary condition.

Fossilised cpu_manager_state. Changing policy without deleting /var/lib/kubelet/cpu_manager_state (and memory_manager_state) makes the kubelet fail or ignore the change. Stop agent → delete file → start.

reserved-cpus ≠ isolcpus. If the host isolates some cores and RKE2 reserves others, the managers assign cores to pods that are not really quiet, or they leave isolated cores idle. The two lists have to be consistent. It is the coordination failure between the previous post and this one.

GPU plugin without NUMA topology. If the device plugin does not expose each GPU’s NUMA node, the Topology Manager aligns CPU and memory but leaves the GPU to chance, and GPU locality is the one that weighs most. Verify that the GPU Operator publishes the topology.

single-numa-node that rejects too much. If the pods ask for more resources than fit on one node (for example, more cores than a socket has), rejection is constant. The answer is not to drop to best-effort (which silences the problem by serving misaligned), but to size the pod so it fits at one table, or to accept restricted with full knowledge of what you are doing.

Believing best-effort “is nearly the same”. best-effort admits the pod even if it fails to align: it gives you a false sense of NUMA-awareness while you serve from the wrong socket. For inference with a tail SLO, single-numa-node or restricted; best-effort only if the alternative is scheduling nothing at all.

Conclusion

The previous post pinned by hand; this one does it at scale and with a guarantee that artisanal numactl did not give: strict admission. The kubelet, via CPU Manager, Memory Manager and Topology Manager, acts as a maître d’ who only seats the pod if its CPUs, its memory and its GPU fit at the same NUMA table, and who rejects what does not fit instead of serving a split dinner. Of the ten knobs, the first two, cpu-manager-policy=static and Guaranteed QoS with whole CPUs, are the precondition without which the other eight do nothing, and they are exactly the ones most often forgotten; the rest tune the policy, the memory, the hugepages and the coexistence. The thread that closes the series: the inference performance that looked like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a cable problem (NVLink not used), a host problem (remote NUMA, jitter) or an orchestration problem (pinning that never happened because the QoS was wrong). Going down a level is not infrastructure snobbery: it is where the root causes live that no application-layer dashboard is ever going to point out for you.

References

Kubernetes, Control Topology Management Policies on a node: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/.
Kubernetes, Control CPU Management Policies on the Node: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/.
Kubernetes, Control Memory Management Policies on a Node: https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/.
RKE2, Configuration Options (kubelet-arg in config.yaml): https://docs.rke2.io/install/configuration.
RKE2, Advanced Options and Configuration: https://docs.rke2.io/advanced.
rancher/rke2, discussion #3034 CPU Management Policies for RKE2 (the cpu_manager_state gotcha): https://github.com/rancher/rke2/discussions/3034.

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

Mon, 01 Jun 2026 16:00:00 +0200

This post complements those on GPU observability for LLM inference (where the metrics that feed the HPA come from), Capacity planning (what ceiling and what head-room the autoscaler assumes) and Continuous batching (which explains why num_requests_waiting is the primary metric).

TL;DR

Classic Kubernetes autoscaling, HPA on cpu or memory, is no use for LLM inference. The reason: the vLLM pod consumes little CPU (the GPU does the work) and the process RSS is flat; both metrics can sit at 30 % while the GPU is saturated and the request queue grows unchecked. The four viable signals that do respond to the real load are: vllm:num_requests_waiting (the queue, the primary metric), vllm:gpu_cache_usage_perc (pressure on the KV cache pool), TTFT P95 via the vllm:time_to_first_token_seconds_bucket histogram (the SLO guarantee) and the batch fill ratio num_requests_running / max_num_seqs (utilisation of the concurrency ceiling). For an HPA to consume Prometheus metrics you need an adapter; as of May 2026 there are two mature options: prometheus-adapter (cluster-wide sigma, static configuration, external.metrics.k8s.io output) and KEDA (ScaledObject with a Prometheus trigger, configurable polling, optional scale to zero, cron integration). KEDA is the dominant option for LLM on a generic cluster because it solves the “warm pool + cron + engine metric” pattern in a single CRD. The dominant operational challenge is not the scaling logic but the cold start: a vLLM pod with Llama 70B BF16 (140 GB) takes between 90 seconds (model pre-cached on a local PV) and 6 minutes (image pull + model download from object store) to serve the first token. The five levers that cut it down are an image pre-pulled via DaemonSet, the model cached on a regional PV or tmpfs, a warm pool with minReplicaCount > 0, predictive scaling via KEDA cron when the traffic pattern is predictable (offices 9–18 h), and parallel model download. The three specific LLM scale-down pitfalls: cutting streaming SSE connections halfway through a response (graceful drain with terminationGracePeriodSeconds ≥ 60 s), scale-out/in oscillation from a badly calibrated stabilization window, and forgetting that the HPA only scales pods; GPU nodes are scaled with cluster-autoscaler over labelled nodepools. This post includes the minimal YAML manifests.

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

An artisan bakery has three wood-fired ovens. Each oven takes 25 minutes to reach temperature from cold. Once hot, it bakes bread continuously with a run of 18 minutes per batch. The manager wants to maximise bread sold per day without burning wood for nothing, and she knows three things: that there is a demand peak at 7:30 every morning, that Mondays sell almost nothing, and that when the bread on the counter runs out the customers go to the supermarket next door.

The cheap strategy, lighting ovens when there is a queue in the shop, does not work. By the time the queue grows and the manager lights the second oven, that oven will not be ready until 25 minutes later; the customers in that window are lost. The “queue at the counter” signal arrives late.

The smart strategy: light the second oven at 6:55, ahead of the predictable 7:30 peak, and leave it running until 10:00 even though the queue drops at 8:15. Keep the third oven off from Monday to Wednesday because demand does not arrive; light it proactively on Thursdays at 12:00 because historically it goes up. Keep a stock of pre-proved raw dough in the chiller so that when the oven is ready the bread goes in within 30 seconds and there is no two-hour proving wait.

Autoscaling an LLM inference cluster works the same way:

Lighting ovens from cold = reactive scale-out when the queue grows (slow, loses customers).
Proactive cron = predictive scaling when the pattern is known (working hours, expected peaks).
Pre-proved dough = warm pool of replicas with the model loaded but under 0 load.
Switching off ovens with no bread inside = scale-down respecting the active streams (you do not shut the oven with bread in it).

The key metric, “how many customers are in the queue”, is called num_requests_waiting. The metric that says “the oven is about to run out of dough for new loaves” is called gpu_cache_usage_perc. And the quality-of-service metric, “how long the first loaf takes to come out when a new customer walks in”, is called TTFT.

Why HPA on CPU is no use

The classic Kubernetes HPA looks at the pod’s resource.cpu. For a conventional HTTP service, Node.js or a REST API, CPU moves linearly with traffic and the HPA scales with reasonable accuracy. For a vLLM or SGLang pod on GPU, the pod’s CPU typically lives between 5 % and 15 % regardless of whether the GPU is at 30 % or 99 % load: the real work is done by the device, not the process. Result: the CPU-based HPA never triggers scale-out even though the GPU is bursting, and clients pile up in the queue until TTFT P95 crosses the SLO. The operator discovers the problem through the TTFT alert, not through the HPA.

memory is no use either: the vLLM process RSS is flat after startup (model plus buffers loaded in one go); it does not reflect the real pressure on the GPU. The only things that rise and fall with the useful inference load are metrics the engine publishes explicitly: request queue, KV cache pool, SLO latencies. Without an adapter that makes them visible to the HPA, autoscaling is blind.

The four viable signals

Signal 1, vllm:num_requests_waiting (queue). It is the most direct metric: how many requests are waiting to enter the batch. It reacts the instant the target concurrency saturates. It is robust against model changes (the number of requests is the same concept whether it is Llama 7B or 70B). It is the primary metric of the LLM HPA. Typical threshold: target = 5 requests waiting on average; if the queue grows above 5 sustained for 2 minutes, scale out.

Signal 2, vllm:gpu_cache_usage_perc (KV pool). It moves before the queue does: the KV pool fills up while the batch slots are still free, until the engine starts rejecting new requests for OOM prevention and the queue forms. It is therefore predictive: it triggers scale-out before the client notices degradation. Typical threshold: target = 0.85 (85 % of pool used).

Signal 3, TTFT P95. The contractual guarantee. If TTFT P95 leaves the SLO, scale out even though the queue and the KV pool look reasonable (there may be a spike of long prompts). It is reactive, it leaves the SLO before your HPA reacts, but it works as a final guardrail.

Signal 4, batch fill ratio. The num_requests_running / max_num_seqs quotient (the latter is engine config, not a metric). Useful for scale-down: if the ratio stays below 0.4 for 10 minutes, there is spare capacity and replicas can be reduced safely.

The recommended policy combines all four: the queue and the KV pool trigger scale-out (whichever arrives first), TTFT confirms it as a guardrail, and the batch fill ratio manages scale-down. Implementing that in a single HPA demands external metrics; KEDA makes this manageable.

The wiring: KEDA as the Prometheus adapter

KEDA introduces two main CRDs: TriggerAuthentication (how to authenticate against the source) and ScaledObject (which deployment to scale with which triggers). For a vLLM deployment with Prometheus as the source:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-llama70b-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama70b
 minReplicaCount: 2 # warm pool
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300 # 5 min before scale-down
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600 # large window to avoid oscillation
 policies:
 - type: Pods
 value: 1
 periodSeconds: 120
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - type: Pods
 value: 2
 periodSeconds: 60
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue_depth
 threshold: "5"
 query: |
 avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv_cache
 threshold: "0.85"
 query: |
 avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_ttft_p95
 threshold: "1.5"
 query: |
 histogram_quantile(0.95,
 sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{deployment="vllm-llama70b"}[5m])))

Three non-obvious operational details:

minReplicaCount: 2. This is the warm pool. Keeping at least two replicas guarantees availability if a node is lost and absorbs spikes without waiting for the cold start of the first scale-up. Dropping it to 0 saves GPU off-peak but introduces 90 s–6 min of latency for the first new client.

stabilizationWindowSeconds: 600 on scale-down. Ten minutes. Models are not nginx: if a replica closes prematurely and two minutes later there is another peak, the cold start of a new pod is what the client waits. Better to keep extra replicas around for twice as long as you would for a normal web service.

scaleUp: stabilizationWindowSeconds: 30. Thirty seconds. Scale-out has to be fast: the new pod’s cold start adds its own delay, and if on top of that the HPA waits several more minutes before triggering, the SLO is already broken.

The big operational problem: cold start

A vLLM pod loading Llama 70B goes through these phases before serving the first token:

Phase	Typical time	Can be accelerated with
Image pull (4–6 GB)	30–90 s	DaemonSet pre-pull
Model download (140 GB BF16)	60–300 s	Cached regional PV, S3 + multi-thread
Loading the model into HBM	30–90 s	tmpfs or local NVMe
CUDA graph capture	20–60 s	`--enforce-eager` (slower at runtime but fast startup)
PagedAttention warmup	5–15 s	—
Health check ready	10–30 s	probe tuning

Total without optimisation: 4–10 minutes. That is how long a new replica takes to absorb traffic. With all the levers combined: 45–90 seconds. The difference between those two numbers is the main platform work for LLM autoscaling.

The five levers

Lever 1, pre-pulled image. A trivial DaemonSet runs ctr image pull (or crictl pull) on the GPU nodes as soon as they join the cluster. The inference engine image stays on disk; new pods skip the 30–90 s of pull. Cost: around 6 GB of disk per node.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: vllm-image-warmer }
spec:
 selector: { matchLabels: { app: vllm-warmer } }
 template:
 metadata: { labels: { app: vllm-warmer } }
 spec:
 nodeSelector: { workload: gpu }
 initContainers:
 - name: pull
 image: vllm/vllm-openai:v0.10.0
 command: ["/bin/true"]
 containers:
 - name: pause
 image: registry.k8s.io/pause:3.10

Lever 2, model on a regional PV. The model download (140 GB BF16 or 35 GB FP8) from central object storage is the dominant component of the cold start. Caching the model on a zone or rack PV, Rook-Ceph RBD or local NVMe provisioned by the operator, cuts 60–300 s down to 5–15 s. The antipattern: downloading the model on every startup from external S3.

volumeMounts:
 - name: model-cache
 mountPath: /models
 readOnly: true
volumes:
 - name: model-cache
 persistentVolumeClaim:
 claimName: llama70b-fp8-pvc  # RWX shared, filled offline

Lever 3, warm pool. minReplicaCount > 0 keeps pre-loaded replicas idle. The cost is idle GPU; the benefit is 0 s of cold start for the first client of a peak. For production clusters with continuous traffic: a warm pool of 2–3 replicas. For nightly batch clusters with 0 traffic: warm pool 0 and accept the cold start, or KEDA with a cron that switches on 10 minutes earlier.

Lever 4, predictive scaling with cron. When the pattern is predictable (offices 9–18 h):

triggers:
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5" # 8:30 Monday–Friday
 end: "0 19 * * 1-5" # 19:00
 desiredReplicas: "6"

Combined with reactive triggers. The HPA scales according to the maximum of the signals: if the cron asks for 6 and the queue asks for 10, the result is 10.

Lever 5, parallel download and an efficient format. For PVs that are not pre-loaded, tools such as nvidia-modelmanager, s5cmd or aria2c parallelise the model download. Going from serial download (~150 MB/s) to parallel with 8 threads (~1.2 GB/s) divides the time by 8. And formats such as safetensors load into HBM faster than the original PyTorch pickle.

When to scale nodes, not just pods

The HPA scales pods. If the cluster has no free GPU nodes, the new pod stays Pending for lack of resources. To scale nodes, you need cluster-autoscaler with a specific GPU nodepool, labelled:

# nodepool config (Karpenter or cluster-autoscaler equivalent)
labels:
 workload: gpu
 gpu-model: h100-sxm-80gb
taints:
 - key: nvidia.com/gpu
 effect: NoSchedule
limits:
 min: 2 nodes
 max: 8 nodes

Without this, the HPA can ask for 10 replicas but the cluster only delivers those that fit on nodes already up. The cold start of a new node (bare metal or cloud provisioning, PXE, OS boot, NVIDIA drivers, cluster join) is much longer than the cold start of a pod: typically 5–15 minutes on preconfigured bare metal, 30–60 minutes on real provisioning. For on-premise clusters, the nodepool must be always sized to the expected maximum, and the “scaling” is only on the pod side. The concept of reactive node scale-out only applies to clouds; on-premise you have to buy for the peak.

Three pitfalls specific to LLM scale-down

Pitfall 1, cutting streaming SSE connections. When a replica enters Terminating, Kubernetes sends SIGTERM to the pod and, by default, kills it 30 seconds later. For vLLM that means cutting streaming SSE connections halfway through the response. The client gets a 502 error with the partial output lost. Solution: terminationGracePeriodSeconds: 120 plus a preStop hook that tells the engine not to accept new requests but to finish those in flight:

spec:
 terminationGracePeriodSeconds: 120
 containers:
 - name: vllm
 lifecycle:
 preStop:
 httpGet:
 path: /shutdown
 port: 8000

This requires the engine to expose a graceful shutdown endpoint; vLLM v1 supports it via --enable-graceful-shutdown. Without it, scale-down breaks the SLO even though the metrics do not capture it (cut requests never enter the TTFT histogram).

Pitfall 2, scale-up/scale-down oscillation. If the scale-down stabilizationWindowSeconds is short (~60 s default), the next drop in the queue triggers scale-down, and two minutes later the next peak triggers scale-up. The system oscillates, pays repeated cold starts, and never reaches a stable regime. Solution: scale-down with a window of 10 minutes minimum and conservative policies (type: Pods, value: 1, periodSeconds: 120, at most one replica fewer every 2 minutes).

Pitfall 3, vllm:num_requests_waiting with avg when there is rebalancing. If two replicas are unbalanced (one with queue 20, the other with queue 0), avg gives 10 and the HPA triggers scale-out when the right move would be to rebalance via the load balancer. To detect it: add an alert on stddev(vllm:num_requests_waiting) per deployment. If the dispersion is high, the problem is not capacity but routing.

Full example manifest

For a vLLM deployment with Llama 70B FP8 on 4×H100 SXM per replica, KEDA with warm pool 2:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama70b
 namespace: inference
spec:
 replicas: 2 # managed by KEDA afterwards
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata:
 labels: { app: vllm-llama70b, deployment: vllm-llama70b }
 spec:
 terminationGracePeriodSeconds: 120
 nodeSelector: { workload: gpu, gpu-model: h100-sxm-80gb }
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args:
 - --model=/models/llama-3.3-70b-fp8
 - --tensor-parallel-size=4
 - --max-num-seqs=64
 - --enable-prefix-caching
 - --enable-graceful-shutdown
 ports:
 - { name: http, containerPort: 8000 }
 - { name: metrics, containerPort: 8000 }
 resources:
 limits:
 nvidia.com/gpu: "4"
 memory: 200Gi
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 60
 periodSeconds: 10
 failureThreshold: 30 # tolerates the warmup
 lifecycle:
 preStop:
 httpGet: { path: /shutdown, port: 8000 }
 volumeMounts:
 - { name: model-cache, mountPath: /models, readOnly: true }
 - { name: dshm, mountPath: /dev/shm }
 volumes:
 - name: model-cache
 persistentVolumeClaim: { claimName: llama70b-fp8-pvc }
 - name: dshm
 emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: vllm-llama70b-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: vllm-llama70b } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: vllm-llama70b-scaler, namespace: inference }
spec:
 scaleTargetRef: { name: vllm-llama70b }
 minReplicaCount: 2
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600
 policies:
 - { type: Pods, value: 1, periodSeconds: 120 }
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - { type: Pods, value: 2, periodSeconds: 60 }
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue
 threshold: "5"
 query: avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv
 threshold: "0.85"
 query: avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5"
 end: "0 19 * * 1-5"
 desiredReplicas: "6"

This set is the minimum viable one for LLM autoscaling on a generic cluster with the NVIDIA GPU Operator. Each team adapts it to its own SLO.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB per node, 4 GPU nodes:

Each node hosts one vLLM replica with TP=4 and Llama 70B FP8 (one model per node, they are not shared).
Warm pool of 2 replicas off-peak; the KEDA cron raises it to 4 during working hours.
Cluster-autoscaler does not apply (4 physical nodes bought; scaling is pods only). The number of concurrent replicas is at most the number of available nodes (if each replica uses all 4 GPUs of the whole node).
If the sizing requires more simultaneous replicas than nodes, there are two routes: (a) lower the TP of each replica so two fit per node, (b) expand the physical nodepool. The decision is dictated by capacity planning, see Capacity planning for on-premise LLM inference.

KEDA event volume: around 5 evaluations/min per ScaledObject. For 10 models served in parallel, 3,000 evaluations/h. Manageable with one KEDA operator per cluster.

What we have not covered (upcoming articles)

Cluster-autoscaler for on-premise GPU nodes: how to orchestrate bare metal provisioning (Tinkerbell, Metal³) as a function of demand.
Multi-cluster autoscaling: scaling across clusters in different DCs for geographic resilience.
Cost-aware autoscaling: prioritising nodes by hourly energy cost (on clusters with indexed tariffs).
Predictive ML-based scaling: instead of a static cron, training a model that predicts demand 30 minutes ahead.
Multi-tenant quotas and fairness: KEDA with namespace quotas so that one tenant does not monopolise the HPA.

References

KEDA project — keda.sh (documentación oficial de triggers Prometheus y cron).
Kubernetes — Horizontal Pod Autoscaler walkthrough (kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale).
NVIDIA — GPU Operator on Kubernetes (Helm chart oficial con DaemonSet de drivers y DCGM).
vLLM project — production_monitoring/ (métricas Prometheus expuestas por el servidor).
Karpenter — NodePool spec (etiquetado y taints para nodepools GPU).
Cluster Autoscaler — Scaling GPU nodes (caveats de descubrimiento de recursos GPU).
Kubernetes — Pod lifecycle and termination (preStop, terminationGracePeriodSeconds).

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

Mon, 01 Jun 2026 15:30:00 +0200

This post complements those on LLM tracing with OpenTelemetry GenAI (the tracing layer above the metrics), Capacity planning (what was sized and what must be watched) and Continuous batching (the mechanism that explains several of the engine metrics).

TL;DR

Observability for an LLM inference cluster is built on two complementary sources: the GPU hardware metrics exposed by the DCGM (Data Center GPU Manager) Exporter, part of the NVIDIA GPU Operator, and the metrics from the inference engine (vLLM, SGLang, TensorRT-LLM) exposed on a Prometheus-compatible /metrics. Neither is enough on its own. The classic nvidia-smi metric called GPU utilization is misleading for LLMs: it reads high whenever any kernel is running, without distinguishing tensor cores burning from SMs waiting on HBM. The full cockpit has twelve DCGM metrics in four families (compute: DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE; memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL; thermal-power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS; health: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_RETIRED_DBE) and five vLLM engine metrics (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds). Each one has a defensible green/amber/red threshold, a PromQL query for alerting, and at least one common false reading that confuses the junior operator. The six critical alerts that any production cluster must fire are: HBM > 92 %, thermal or power throttle, XID error, ECC double-bit, KV cache pool > 95 %, and TTFT P95 outside SLO for 5 minutes. The point of having this panel: that the operator on shift diagnoses the origin of a degradation in under five minutes, without opening an SSH console to the GPUs. When that holds, the cluster has moved to professional operation; until then, it is operated on intuition.

You are here: OBSERVE (the other half of tracing)

Tracing, already covered in LLM tracing with OpenTelemetry GenAI, answers what happened in this specific request. Metrics answer what is happening across the cluster in aggregate. They are complementary: an alert on the metrics side tells you “the cluster is degrading”, tracing tells you “and here is the specific trace that proves it”. A cluster without tracing but with metrics operates; a cluster without metrics but with tracing does not operate, it debugs.

The analogy: the cockpit of a modern aircraft

In a modern commercial aircraft, the pilot’s instrument panel has more than 70 active indicators. If there were only one, the altimeter, say, the aircraft would fly into the ground at the first moment of low visibility. You need the altimeter and the attitude indicator, and the airspeed indicator, and the turn indicator, and the fuel gauge, and the oil pressure gauges for each engine, and the turbine exit temperatures. Each answers a different question. And together they cover the operational question: is the aircraft healthy, is it where it should be, and is it going where we want?

Observability for an LLM inference cluster works the same way. A single metric, “GPU utilization 99 %”, answers nothing useful. It is like looking only at the car’s odometer to diagnose why the engine is making a noise. The full cockpit is twelve instruments on the hardware side plus five on the inference engine side, organised into families that answer different questions:

Compute and efficiency: are the tensor cores doing the work we expect, or are they waiting?
Memory: is there VRAM left for new requests, or are we on the edge of OOM?
Thermal and power: is the hardware healthy, or is it silently limiting throughput?
Health and errors: is there hardware degradation under way (ECC, XID, NVLink)?
Inference engine: is the queue growing, is the KV pool saturated, is the SLO being met?

The first four answer “is the GPU fine?”. The fifth answers “is it delivering the service we promised?”. The two questions are different and both must have an answer at a glance.

Why `nvidia-smi` `GPU-Util` misleads with LLMs

The classic metric that appears in nvidia-smi as GPU-Util corresponds to DCGM_FI_DEV_GPU_UTIL. Its official definition: “percentage of time during which one or more kernels were executing on the GPU”. The problem with LLMs: the decode phase is memory-bound, not compute-bound. When the inference engine decodes token by token, the GPU spends 90 % of the time waiting for the HBM to finish delivering the model weights and the KV cache. There is a kernel running (the HBM read), so GPU-Util reports values close to 100 %. But the tensor cores are idle: the bottleneck is memory, not compute.

Practical result: the operator sees “GPU-Util 99 %” in Grafana and assumes “GPU saturated, no more load can go in”. But the reality may be “compute at 25 %, HBM saturated at 95 %”, which changes the operational decisions (quantisation, batch size, parallelism). The classic metric lies by oversimplifying.

The right move is to look at the three DCGM profiling metrics of the _FI_PROF_* subsystem:

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps over the maximum per SM. Is there parallel work?
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores effectively active. Is compute working?
DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring. Is memory saturated?

A typical decode-bound GPU running Llama 70B on an H100 shows: SM occupancy 35–55 %, tensor active 15–30 %, DRAM active 80–95 %. That is the real “saturated GPU” for LLMs. The three together tell the regimes apart; none of them alone says anything actionable.

How they are wired up in production

The platform side is covered in Five maturity levels (level 4, the GPU plane) and Seven deployment phases (phase F5). For the observer, the key pieces are:

NVIDIA GPU Operator. Helm manifests that deploy on each GPU node: drivers, container toolkit, MIG manager and DCGM Exporter. The last one exposes /metrics in Prometheus format with all the DCGM_FI_* listed above. It is scraped from the cluster’s internal Prometheus.

Inference engine. vLLM exposes /metrics on port 8000 (default) with vllm:* metrics. SGLang exposes it too, with the sglang: prefix. TensorRT-LLM exposes it through Triton Inference Server with the nv_inference: prefix. The basic naming convention is similar across the three engines; the thresholds and queries in this post assume vLLM, but they translate.

ServiceMonitor / PodMonitor. The Prometheus operator resource that says what to scrape. Minimal example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: vllm-inference
spec:
 selector:
 matchLabels: { app: vllm }
 podMetricsEndpoints:
 - port: metrics
 interval: 15s

Dashboards. The NVIDIA operator publishes reference Grafana dashboards for DCGM in nvidia/dcgm-exporter (the official repo). vLLM publishes one in vllm-project/vllm (the examples/ folder). Both work as a base; each team adds the panels specific to its own SLO.

The twelve DCGM metrics organised by family

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps per SM over the maximum possible. Value between 0 and 1.

Green: 0.30–0.70 (typical LLM decode regime).
Amber: < 0.20 sustained (batch too small, GPU under-used in parallelism).
Red: 0.95 sustained with low DRAM_ACTIVE (pathological kernel saturating the SMs).

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores executing. The key metric for “is compute producing?”.

Green in prefill: 50–80 %.
Green in decode: 15–30 % (decode is memory-bound, this is not a symptom of a problem).
Red: < 5 % sustained in prefill, or the engine is not using the tensor cores at all (bad config, incompatible format).

DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring data. The key metric for detecting memory saturation.

Green in decode: 60–85 %.
Amber: > 90 % sustained (HBM is a firm bottleneck, which explains high TPOT).
Red: > 95 % sustained with KV cache pool < 70 % (something other than the engine is asking for HBM; investigate leaks).

Family 2 — Memory

DCGM_FI_DEV_FB_USED: Frame Buffer (HBM) used, in MiB.

Green: 70–85 % of total.
Amber: 86–92 %.
Red: > 92 % (risk of OOM on the next paged-attention allocation).

PromQL for the cluster-wide percentage: 100 * sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_TOTAL).

DCGM_FI_DEV_FB_FREE: free Frame Buffer. Complementary to the previous one; useful for absolute alerts (< 4096 MiB free).

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: aggregate NVLink bandwidth in MB/s. For TP (tensor parallel) topologies that cross GPUs over NVLink, this metric reveals whether the parallelism split is saturating the bus.

Green: varies with topology. On 4×H100 SXM with NVLink 4.0, theoretical capacity is 450 GB/s per GPU. A typical TP=4 regime: 50–150 GB/s sustained.
Red: > 90 % of capacity sustained (check whether the model would fit with lower TP or with pipeline parallel).

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP: die temperature in °C.

Green: < 75 °C.
Amber: 75–82 °C.
Red: > 83 °C (close to the H100’s automatic thermal throttle; check ventilation, air flow, rack inlet temperature).

DCGM_FI_DEV_POWER_USAGE: draw in watts. For the H100 SXM, nominal TDP is 700 W. Useful for three things: spotting an unusually low workload (suspect idle or a stall), billing real energy cost, and firing alerts if the draw approaches the PDU limit.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: encoded bitmap with the active throttle reasons. This is the metric that silently explains TPOT degradations.

Relevant bits:

0x0000000000000001: Idle (not a problem).
0x0000000000000002: App clocks setting.
0x0000000000000004: SW Power Cap (software limit, e.g. from nvidia-smi -pl).
0x0000000000000008: HW Slowdown.
0x0000000000000010: Sync Boost (NVIDIA Sync).
0x0000000000000020: SW Thermal Slowdown (software thermal limit).
0x0000000000000040: HW Thermal Slowdown (hardware thermal limit, an emergency).
0x0000000000000080: HW Power Brake Slowdown (PSU voltage drop).
0x0000000000000100: Display Clock Setting.

Any throttle other than Idle with a sustained value > 0 is an alert. TPOT degradation with DRAM_ACTIVE already high and thermal throttle active is the classic “the rack is badly ventilated, it is not the engine’s fault”.

Family 4 — Health

DCGM_FI_DEV_XID_ERRORS: cumulative counter of driver XID errors. XIDs are critical event codes that NVIDIA documents exhaustively (XID 13: graphics engine exception; XID 31: GPU memory page fault; XID 43: reset channel verif error; XID 79: GPU has fallen off the bus; XID 95: uncontained ECC error; and so on). Any increment is an immediate alert: many XIDs require a node reset or an RMA of the GPU.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC errors (uncorrectable). Unlike single-bit ones (which ECC corrects silently and which are counted in DCGM_FI_DEV_ECC_SBE_*), double-bit errors corrupt data. Any value > 0 is a critical alert: the GPU must be drained and inspected.

DCGM_FI_DEV_RETIRED_DBE: physical HBM pages retired because of accumulated double-bit errors. NVIDIA retires faulty pages automatically to prevent future corruption. More than 4–8 retired pages on one GPU suggests silicon degradation: document it and plan a replacement in the next maintenance window.

The five inference engine metrics (vLLM)

The DCGM metrics answer “is the GPU healthy?”. The engine metrics answer “is the service meeting the SLO?”. Without them, you know the hardware works but you do not know whether the clients are happy.

vllm:num_requests_running: requests currently in the batch. If it reaches the configured --max-num-seqs and does not come down, the engine is saturated on concurrency (check VRAM and rebalance through the autoscaler, see Autoscaling LLM on Kubernetes).

vllm:num_requests_waiting: requests queued, not yet in the batch. Any value > 0 sustained over minutes means the cluster is not scaling with the load. This is the primary metric for HPA.

vllm:gpu_cache_usage_perc: % of the KV cache pool in use.

Green: 50–80 %.
Amber: 80–92 %.
Red: > 92 % (risk of preempt-on-OOM: vLLM will drop requests to free memory, which raises TTFT visibly).

vllm:time_to_first_token_seconds: histogram of TTFT per request. Consumed as histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket[5m]))). Compared against the TTFT P95 SLO, it fires the primary service alert.

vllm:time_per_output_token_seconds: histogram of TPOT. Equivalent to the previous one but for streaming smoothness. Compared against the TPOT P95 SLO, it fires the secondary alert.

The six alerts that must page in production

Any serious production cluster fires these six alerts to a channel with an on-call rotation. Without them, the SLO is met by luck, not by process.

groups:
 - name: gpu-llm-critical
 rules:
 - alert: GpuHbmNearOom
 expr: 100 * (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 92
 for: 2m
 labels: { severity: critical }
 annotations:
 summary: "HBM on {{ $labels.gpu }} at {{ $value }}% — OOM risk"

 - alert: GpuThermalOrPowerThrottle
 expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0) and ignoring(reason) (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 1)
 for: 1m
 labels: { severity: warning }
 annotations:
 summary: "GPU {{ $labels.gpu }} throttling (reasons={{ $value }})"

 - alert: GpuXidErrorDetected
 expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
 labels: { severity: critical }
 annotations:
 summary: "XID error on GPU {{ $labels.gpu }} — investigate immediately"

 - alert: GpuEccDoubleBit
 expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
 labels: { severity: critical }
 annotations:
 summary: "ECC double-bit on GPU {{ $labels.gpu }} — drain node"

 - alert: VllmKvCachePoolNearFull
 expr: vllm:gpu_cache_usage_perc > 0.95
 for: 3m
 labels: { severity: warning }
 annotations:
 summary: "KV cache pool > 95% on {{ $labels.instance }}"

 - alert: VllmTtftP95OutOfSlo
 expr: histogram_quantile(0.95, sum by(le, instance)(rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.5
 for: 5m
 labels: { severity: warning }
 annotations:
 summary: "TTFT P95 over SLO ({{ $value }}s > 1.5s)"

These six cover 80 % of the incidents that affect the SLO. The remaining 20 % demand investigation with tracing (see LLM tracing with OpenTelemetry GenAI).

Master table: thresholds and queries

Metric	Green	Amber	Red	Base query (PromQL)
SM occupancy	0.30–0.70	0.15–0.30	< 0.10 sustained	`DCGM_FI_PROF_SM_OCCUPANCY`
Tensor active (decode)	15–30 %	< 10 %	< 3 %	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
DRAM active	60–85 %	85–95 %	> 95 % with low KV	`DCGM_FI_PROF_DRAM_ACTIVE`
FB used	70–85 %	86–92 %	> 92 %	`100 * DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL`
NVLink BW	< 70 % cap	70–90 % cap	> 90 % cap	`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`
GPU temp	< 75 °C	75–82 °C	> 83 °C	`DCGM_FI_DEV_GPU_TEMP`
Power usage	< 90 % TDP	90–98 % TDP	> 98 % TDP	`DCGM_FI_DEV_POWER_USAGE`
Throttle reasons	0 or Idle	App/SW	HW Therm/Power	`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`
XID errors	no change	—	any delta	`increase(DCGM_FI_DEV_XID_ERRORS[5m])`
ECC DBE	0	—	> 0	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
Retired pages	< 4	4–8	> 8	`DCGM_FI_DEV_RETIRED_DBE`
KV cache used	50–80 %	80–92 %	> 92 %	`vllm:gpu_cache_usage_perc`
Requests waiting	0	1–5 sustained	> 10 sustained	`vllm:num_requests_waiting`
TTFT P95	< SLO	80–100 % SLO	> SLO	see alert query above
TPOT P95	< SLO	80–100 % SLO	> SLO	`histogram_quantile(0.95, sum by(le)(rate(vllm:time_per_output_token_seconds_bucket[5m])))`

Three pitfalls that confuse the junior operator

Pitfall 1 — “GPU-Util at 99 % = saturated”. As explained at the start, DCGM_FI_DEV_GPU_UTIL lights up with any kernel. The right move is to look at the three _PROF_* metrics (SM occupancy, tensor active, DRAM active) together. GPU util 99 % + tensor active 8 % + DRAM active 92 % means “saturated by memory, not compute”; GPU util 99 % + tensor active 75 % + DRAM active 50 % means “saturated by compute, prefill heavy”. The two situations call for different levers.

Pitfall 2 — confusing single-bit ECC (SBE) with double-bit (DBE). Single-bit errors are corrected silently and are unavoidable in any HBM under load (cosmic radiation, voltage fluctuations). An SBE counter growing slowly is not an alert, it is physics. DBE is: it corrupts data. Telling the two metrics apart avoids false alarms and false negatives in equal measure.

Pitfall 3 — alerting on num_requests_waiting > 0 without context. An instantaneous value of 1 or 2 during a spike is normal. What matters is a sustained queue: use for: 5m with a threshold of 3–5. Without that window, the system floods the alert channel with noise.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter deployed through the NVIDIA GPU Operator, one DaemonSet per GPU node.
Internal Prometheus with 30-day retention for high-frequency metrics, 1 year for downsampled ones (Thanos/Mimir if the volume justifies it).
Grafana with three standard dashboards: GPU hardware (DCGM), engine (vLLM), SLO (TTFT/TPOT/RPS against written targets).
Alertmanager with an on-call rotation and rate limiting through silences grouped by node.
Cardinality kept under control: gpu (local id), node, pod, model. Do not add request_id or other high-cardinality labels to metrics; that is tracing’s job.

Estimated volume for a 16-GPU cluster scraped every 15 s: around 2 million samples/min, around 25 GB/day of raw Prometheus. Manageable with one Prometheus per cluster plus retention; if the team scales beyond 64 GPUs, consider a Thanos sidecar or VictoriaMetrics. See OSS LLMOps tooling catalogue for equivalent alternatives.

What we have not covered (upcoming articles)

Tracing of LLM workloads: already covered in LLM tracing with OpenTelemetry GenAI.
Autoscaling based on these metrics: see Autoscaling LLM on Kubernetes.
Incident response runbooks: how each of these alerts translates into concrete action (drain, restart, RMA, scaling, rollback).
Cost accounting: using DCGM_FI_DEV_POWER_USAGE and vllm:request_success_total for per-tenant cost showback.
Multi-tenant fairness monitoring: when several tenants share a cluster, which metrics detect that one of them is hogging the KV cache.

References

NVIDIA — DCGM Exporter (repo nvidia/dcgm-exporter, documented metrics and units).
NVIDIA — DCGM Field Identifiers reference (complete list of DCGM_FI_*).
NVIDIA — XID Errors documentation (catalogue of XID codes and remediation procedures).
NVIDIA — NVIDIA GPU Operator (official Helm chart).
vLLM project — examples/production_monitoring/ (reference PromQL and Grafana dashboards).
Prometheus — Histogram and summary best practices (for building defensible percentile queries).
NVIDIA — H100 Tensor Core GPU datasheet (TDP, HBM bandwidth, NVLink capacities).

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

Seven greenfield deployment phases for an on-premise LLM platform: from hardware in the room to the first production token

Sun, 31 May 2026 08:00:00 +0200

TL;DR

The two previous posts in this architectural trilogy settled the pieces: the seven layers of the LLM inference stack describe the components above the cluster, and the five maturity levels of the platform describe the strata below. This post settles the when: in what order each thing gets deployed when you start from zero, hardware bought, racks installed, physical cabling done, and you want to reach a cluster serving the first production token to a customer. Seven nominal phases F0 to F6 with no calendar commitments, organised by technical dependencies (you do not enter F3 without the F2 gate) and with an identifiable critical path. F0 hardware inventory and electrical/network connectivity. F1 bare metal OS + drivers + container runtime. F2 Kubernetes cluster with CNI and Ceph storage operational. F3 GitOps and infrastructure observability. F4 identity, TLS, secrets and policies. F5 GPU platform with LLM-aware observability. F6 the LLM stack operational and open to production traffic. For each phase: what gets built, what has to be ready first (dependencies between phases), the gate that validates its closure, and the typical trap that delays the critical path. The thesis: an on-premise LLM platform sinks far more often from sequencing badly than from choosing badly. The tools have all been invented; the order is the only thing every team rediscovers.

You are here: the seven phases and their dependencies

The phases do not run in pure series. F2 and F3 can be started at the same time to speed things up (installing Kubernetes and preparing the GitOps repo in parallel). F4 can overlap with the tail end of F3. F5 waits for F4 to close because GPU pods demand NetworkPolicy and RBAC from day one. F6 is a single atomic step: the cluster goes into production or it does not.

The red arrows are the critical path: the sequential bottleneck that no parallelism can shorten. The grey arrows are dependencies that admit partial overlap. Recognising where to overlap and where not is the difference between a three-month deployment and a six-month one for the same perimeter.

The analogy: the expedition to an eight-thousand-metre summit

An expedition to a high alpine summit is not a long trek. It is a series of camps that are set up in order, each with its altitude, its function and its validation gate: if you do not acclimatise properly at base camp, you cannot go up to C1 without risk; if C2 does not have its kitchen and its radio running, you cannot send people higher; if the summit push is attempted without porters in the high camps, there is no safe descent.

Greenfield deployment of an LLM platform works identically. F0 is the arrival of the material at base camp: boxes, sponsors, permits, first review. F1 is setting up an operational base camp: kitchen, tents, generator. F2 is the climb to C1: there is real altitude now (a k8s cluster running) and you breathe differently. F3 is C2: it adds communications, planning and operational acclimatisation. F4 is C3, the last night before the push: team roped up, oxygen ready, every protocol verified. F5 is the day of the summit push, intense effort, thin margins. F6 is the summit and the start of a safe descent: from here on the expedition is in day-to-day operation, no longer in construction.

The analogy holds two useful lessons: you do not skip a camp (going straight from base camp to the summit kills the team), and the gates are technical, not emotional (if the barometer says storm, you do not set out, however much enthusiasm there is). The platform team that follows those two rules reaches the summit. The one that negotiates them does not.

F0 — Hardware in the room: base camp

What gets built in this phase. An inventory of the hardware received (servers, switches, PDUs, BMC), racks assembled, electrical and data cabling terminated, physical labelling of each machine (rack/U/function), connectivity to the corporate network, management IPs assigned, BMC reachable over VPN with MFA, first ping to each node from the bastion.

Dependencies. None technical, this is the phase before the software. There are procurement dependencies (servers bought, switches bought), civil works dependencies (a room with enough cooling, a raised floor) and administrative ones (datacenter access for the technicians).

Validation gate that closes F0.

Every node appears in the inventory with (hostname, MAC, management IP, data IP, rack, U, function, owner).
Each node’s BMC responds to ipmitool power status and to the HTTPS UI from the management VPN.
The top-of-rack switch has its configuration versioned in git (even if there is no cluster GitOps yet, the switch configs are).
A command for h in $(cat hosts); do ping -c1 -W1 $h.mgmt; done returns 100 % success.

Typical trap. Cabling “however it comes out” with no physical labelling and no diagram. When F4 arrives and a NetworkPolicy has to be troubleshot, not knowing which physical interface carries which VLAN doubles the diagnosis time of every incident for ever.

Why F0 does not overlap with F1. Until every server has a management IP and a live BMC, OS bootstrap cannot be automated. Every hour invested in F0 saves hours in every later phase. It is the phase with the best ROI in the project and the only one that admits no shortcuts.

F1 — Bare metal: the operational base camp

What gets built. The operating system image (stable Debian or Ubuntu LTS) provisioned via PXE or cloud-init with the cloud-config versioned in git. Each node has: a coherent hostname, LVM partitions, kernel ≥ 6.6, the containerd container runtime, NVIDIA drivers on the GPU nodes, chrony synchronising against your own servers, the operator’s SSH key as the only way in, and nvidia-smi passing a smoke test on the GPU nodes.

Dependencies. F0 closed. It needs the management network working so PXE responds and so the bastion can reach every node.

Validation gate that closes F1.

ansible -i inventory all -m ping returns 100 % success (or the equivalent with Salt / Pulumi / etc).
Every GPU node passes nvidia-smi showing the expected GPUs with a consistent driver across nodes.
Each node’s clock drifts by less than 50 ms from the reference NTP.
A physical reboot of a node leaves it in exactly the same state after boot (idempotency).

Typical trap. NVIDIA drivers installed by hand with apt install or with NVIDIA’s .run script. It works on day one and breaks the day of the first kernel update. The operational rule already established in the post on the five levels: the drivers end up managed by the GPU Operator in F5; whatever is done now is only so that nvidia-smi passes the smoke test, not for production.

Possible overlap. F1 can start for some nodes while F0 is still being finished on others (real greenfield rarely delivers every server on the same day). The F1 gate is per cluster, not per individual node.

F2 — Kubernetes cluster operational

What gets built. RKE2 installed with three HA control plane nodes, all workers joined (CPU and GPU), Cilium as CNI with kubeProxyReplacement enabled and a BGP control plane pointing at the ToR switches from F0, Rook-Ceph deployed on the storage nodes to cover block (RBD), filesystem (CephFS) and object (S3-compatible RGW), kubectl get nodes returning every node Ready, a first test pod with a PVC mounting and data persisting after the pod restarts.

Dependencies. F1 closed (drivers + container runtime). Switches with BGP configured (closed in F0). NVMe disks partitioned or available raw for Ceph OSDs.

Validation gate that closes F2.

kubectl get nodes -o wide shows every node Ready with the expected Kubernetes version.
A Deployment with replicas=3 and per-node antiAffinity starts and the pods land on different nodes.
An RWO PVC (RBD) creates a volume, the pod writes data, the pod is deleted, another pod mounts it and reads the data.
An RWX PVC (CephFS) does the same with two pods writing simultaneously.
An RGW bucket via s3cmd or mc accepts put and get over TLS.
Hubble (the read side of the CNI) shows flow logs between two pods in different namespaces.
Chaos test: draining a non-GPU worker node; the workloads are rescheduled automatically.

Typical trap. Starting to kubectl apply real workloads in F2 without GitOps. The backlog of things-applied-by-hand grows faster than the capacity to migrate it to git afterwards. The rule: in F2 only the cluster prerequisites are applied (CNI, CSI, default storage class). Any application workload waits for F3.

Possible overlap. F2 ↔ F3. While the cluster is being built, the GitOps repo is prepared in parallel (directory structure, first Helm releases). When F2 closes, Flux is plugged into the repo and everything that was going to be kubectl apply is already a reconciled manifest.

F3 — GitOps and infrastructure observability

What gets built. Forgejo deployed first (it is a prerequisite for everything that follows). A gitops-infra repo with the initial structure (apps/, infrastructure/, tenants/, clusters/). Flux installed and reconciling that repo. The prerequisite workloads applied by hand in F2 are moved to the repo and reconciled (no operational kubectl apply remains). VictoriaMetrics + vmagent scraping metrics. Grafana with initial dashboards (USE/RED + cluster + Ceph + Cilium). Loki receiving logs via vector/fluent-bit. Alertmanager + Keep routing alerts to a chat channel. Barman Cloud backups for Postgres (future CNPG) and scheduled Ceph snapshots.

Dependencies. F2 closed. An RGW bucket to store backups (covered by the Ceph from F2).

Validation gate that closes F3.

A change applied to the repo is reflected in the cluster in under 5 minutes with no manual intervention.
A change applied with kubectl edit directly to the cluster is detected by Flux and reverted (binding drift detection, not merely observational).
Grafana shows dashboards for the cluster, Ceph, Cilium and GPU nodes (DCGM does not arrive until F5, but basic node metrics do).
A test alert sent to Alertmanager reaches the chat channel in under 1 minute.
Restoring a Postgres backup on a temporary cluster returns coherent data (the test defines the real RPO).

Typical trap. Having Helm charts in git but still applying them with helm install from the terminal. That is level 1 dressed up as level 2. F3 only closes when Flux is the only authority that applies changes and humans edit the repo, not the cluster.

Possible overlap. F3 ↔ F4. While F3 is being closed, the Defguard and cert-manager manifests can be prepared in the repo. When they reconcile they have somewhere to land.

F4 — Identity, certificates, secrets, policies

What gets built. Defguard deployed with its dedicated Postgres (CNPG). An initial realm with the platform operators enrolled with MFA and WireGuard. OIDC integrated into kube-apiserver (--oidc-issuer-url, --oidc-client-id, --oidc-username-claim), into Forgejo, into Grafana, into Alertmanager, one single SSO. cert-manager installed with an internal CA issuing internal certs for mTLS and with Let’s Encrypt ACME for edge certs. SOPS configured with a KMS (it can be a physical HSM, an age key in a vault, or an external Vault) and External Secrets Operator syncing secrets into the cluster. Kyverno deployed with initial policies in audit mode for a week, then promoted to enforce. A default-deny NetworkPolicy applied to every existing namespace. Tetragon enabled for runtime security. The kube-apiserver audit log sent to Loki with long retention.

Dependencies. F3 closed (Flux applies the manifests, VM/Loki ingest metrics and logs).

Validation gate that closes F4.

kubectl with a shared admin kubeconfig stops working; every operator uses their own OIDC token with MFA.
A secret in plain data: in a commit is rejected by the pre-commit hook (or by Kyverno admission).
A pod without securityContext.runAsNonRoot=true is rejected by Kyverno at admission.
A deliberately wrong NetworkPolicy (allow-all) in a tenant namespace is rejected.
An audit of the last day returns the complete list of actors and changes (minimum regulatory footprint).
Basic internal pen-test: an attacker with a forged kubeconfig fails MFA and is recorded.

Typical trap. Kyverno in permanent audit mode because “we do not want to break production workloads”. F4 closes when the policies are in enforce. Until then, you are still in F3 wearing an F4 face.

Why F4 does not overlap with F5. F5 introduces GPU pods that move a lot of VRAM and a lot of compute. Without default-deny NetworkPolicy, without OIDC RBAC, without Kyverno blocking insecure configurations, the GPU pods are the juiciest attack surface in the cluster. Any compromise in F5 with F4 not closed is near-total access to the expensive hardware.

F5 — GPU platform with LLM-aware observability

What gets built. NVIDIA GPU Operator via Flux with the driver version decided in F1 (no longer touched by hand). DCGM Exporter exposing GPU metrics to VictoriaMetrics. MIG manager configured for the nodes where it makes sense (for example, on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs and embeddings). Topology Manager with the single-numa-node policy. KEDA with the Prometheus scaler installed and an example ScaledObject pointing at a vLLM metric (vllm:num_requests_running). OpenTelemetry Collector with OTLP receivers, attributes processors (enriching spans with tenant_id, priority_tier), exporters to Langfuse and to Tempo. LeaderWorkerSet API enabled for tensor parallel topologies. OME (Operator Model Engine) or vLLM Production Stack deployed as a controller, still with no models loaded.

Dependencies. F4 closed (the GPU pods inherit default-deny NetworkPolicy, OIDC RBAC and Kyverno policies).

Validation gate that closes F5.

A test pod requesting nvidia.com/gpu: 1 is scheduled on the right node and nvidia-smi from inside the container sees the right GPU (whole or a MIG slice).
DCGM Exporter exposes metrics in Grafana (utilisation, VRAM, temperature, NVLink bandwidth) for every GPU.
A test vLLM Deployment starts with a small model (for example, a 7B FP16) loaded from Ceph RGW.
An OpenTelemetry span generated by that vLLM reaches Langfuse with correct gen_ai.* attributes.
KEDA scales the test Deployment from 1 to N replicas under synthetic load and back to 1 when it stops.
A GPU Operator upgrade to a new version drains and reschedules the GPU pods with no loss of service.

Typical trap. Loading the large model “just to try” before DCGM and OTel are green. When something fails, there will be no metrics to distinguish OOM, thermal throttling, a driver mismatch or a network problem, and you diagnose blind. The rule: small model first, golden path green, then the large model.

Possible overlap. None with F6. F6 is atomic.

F6 — LLM stack in production

What gets built. The seven layers of the inference stack described in the corresponding post, deployed in this order:

Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets). Some components already existed from F3 as data; here they specialise for RAG with their initial collections and schemas.
Embeddings + reranker (Infinity with multilingual-e5-large, TEI with bge-reranker-v2-m3). This is the layer that must be green before any large model, because RAG depends on it.
LLM inference (vLLM Production Stack with the general LLM and the code LLM). It loads models from Ceph RGW. The initial multi-LoRA pool is empty.
Gateway (Envoy AI Gateway) with Defguard OAuth, routing by body.model, rate-limit per tenant. This is the point that opens traffic to the outside.
LLM-aware observability (Langfuse plugged into the OTel from F5).
GitOps control plane and dependency tracking were already active since F3 and F4 respectively; here you simply add the catalogue of the new LLM services to them.

Dependencies. All the previous phases closed.

Validation gate that closes F6.

A curl to the public endpoint with a Defguard bearer token receives a correct chat completion response in technical Spanish, with trace_id propagated.
The trace appears in Langfuse with complete gen_ai.* attributes, a latency breakdown and its own tenant_id.
A 5 % traffic canary to the new model over 24 h degrades neither quality nor latency metrics.
A controlled traffic burst triggers KEDA, the replicas scale, and P95 latency stays within budget.
A deliberate failure of one vLLM pod does not affect endpoint availability (replicas + reschedule).
The internal operator demonstrates the complete path for revoking a tenant’s access in under 5 minutes (Defguard → Kyverno → closing the NetworkPolicy).

Typical trap. Opening real customer traffic before having the incident runbook signed off, the SLO negotiated and the continuity plan tested. F6 is technically closed; operationally, the platform is still an experiment until the first real postmortem proves the team knows how to respond.

The maths that matter: relative weight of effort per phase

Without committing to calendar weeks, we can quantify the relative weight of the engineering effort per phase in a typical greenfield. The curve is not uniform:

$$ \text{effort}_{F_i} \approx \text{base}_i \cdot (1 + \epsilon_i) $$

where $\text{base}_i$ is the nominal effort and $\epsilon_i$ is the surprises factor (wrong cabling, incompatible drivers, badly issued certificates, version conflicts). The table below gives the nominal relative weight and the typical surprise factor observed:

Phase	Nominal weight	Typical surprise factor ε	Mean effective weight
F0 — Hardware	8 %	0.5 (1× to 2×)	12 %
F1 — Bare metal	6 %	0.3	8 %
F2 — k8s cluster	12 %	0.4	17 %
F3 — GitOps + obs	14 %	0.5	21 %
F4 — Identity + policies	18 %	0.7	31 %
F5 — GPU plane	10 %	0.4	14 %
F6 — LLM stack live	8 %	0.3	10 %
Buffer / integration	24 %	—	—

Two operational observations. F4 concentrates more surprises than any other phase (OIDC federation across four or five apps with different configurations, Kyverno policies that knock over legitimate workloads, secrets broken by poorly tested encryption). F0 has a high surprise coefficient relative to its size because any cabling or labelling error is discovered late and paid for dearly. The two practical consequences: plan F4 with generous margin and do not skimp on time in F0, because every hour saved there costs five later.

Critical path and slack. The critical path is linear F0 → F1 → F2 → F3 → F4 → F5 → F6. The only real slack is the overlaps already identified:

F2 ↔ F3 (slack ~30 %): prepare the repo and initial dashboards while the cluster is being built.
F3 ↔ F4 (slack ~20 %): identity manifests ready as F3 closes, applied immediately.
Within F4: policies in audit mode running in parallel with the Defguard setup.

Nothing shortens the critical path by more than about 15 % of the total. Anyone promising a productive greenfield in half a reasonable timescale is selling something else: probably skipping F4, or loading F6 with an F5 that is green-but-not-validated.

Final diagram: the complete deployment schedule

The schedule is not decorative: each row defines what gets built in its phase and the gate that closes it. A phase is not considered finished until its gate is green. A phase with a yellow gate drags every later one; trying to jump to the next with a partially met gate is what produces, several weeks later, the incident that forces you to “go back to F4 with production running”, the most expensive situation in the whole cost matrix of the post on the five levels.

Typical planning mistakes

Patterns that delay or sink the greenfield deployment, regardless of the tools chosen:

1. Buying the LLM before the cluster. Starting the project with “which model are we going to serve” instead of “what platform can sustain any reasonable model”. The model is an interchangeable parameter; the platform is not.

2. Underestimating F0. “The network team does that.” Yes, but the output of F0 is consumed by every later phase. If the network team delivers late, the whole project is late, and nobody had marked it as the critical path.

3. Overlapping F4 with F5 “to save time”. It is the one dependency with no slack. If you try to overlap, F5 ends up operating with policies in permanent audit (you are not in F4) or without OIDC integrated (operators with a shared kubeconfig touching GPUs). Both antipatterns stay in production.

4. Skipping the small model smoke test in F5. “Let’s go straight for the 70B.” When something fails (and something will), there will be no baseline to diagnose against.

5. Treating F6 as “turning on vLLM”. F6 includes gateway, LLM-aware observability, runbook, SLO, continuity plan. Turning on vLLM takes five minutes; closing F6 is weeks of validation and sign-off.

6. Not defining gates in writing. If the gates are not written down, they are negotiable after the fact. “This already counts as F4” is the sentence that precedes the next six months of retrofit.

7. Assigning a phase to a single owner. Every phase needs at least two people who understand it. Staff turnover on long projects destroys knowledge; written gates + cross review preserve it.

8. Forgetting the descent route. The post focuses on going up. Day-to-day operation (the descent, in the analogy) is another story that also deserves planning: runbooks, on-call, upgrade capacity, end-of-life plan. Teams that only plan the ascent reach the summit and stay there with no oxygen.

Applied to typical on-premise hardware: 4×H100 SXM

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM per GPU node, 3 control plane nodes, 3-5 CPU worker nodes, 2 GPU worker nodes), the temporal split of the work is distributed like this:

F0 (hardware)
└─ 8 physical servers racked + switches + BMC + management IPs
 ├─ 3 nodes cp-01..03 — control plane (no GPU)
 ├─ 3 nodes worker-cpu-01..03 — CPU plane (Forgejo, Ceph, observability)
 └─ 2 nodes worker-gpu-01..02 — GPU plane (4×H100 SXM each)

F1 (bare metal)
└─ OS + drivers + containerd on all 8 nodes alike
 (NVIDIA drivers only on the 2 GPU nodes, `nvidia-smi` smoke)

F2 (k8s cluster)
└─ RKE2 control plane on cp-01..03 (HA with embedded etcd)
 workers joining: 3 CPU + 2 GPU
 Ceph OSDs on the 3 CPU worker nodes
 default pools: RBD-replicated-3, CephFS-replicated-3, RGW

F3 (GitOps + obs)
└─ Forgejo + Flux + VM/Grafana/Loki + Keep on the CPU plane
 first `gitops-infra` repo reconciles what came from F2

F4 (identity)
└─ Defguard on the CPU plane (StatefulSet with Postgres CNPG)
 OIDC in kube-apiserver, Forgejo, Grafana, Alertmanager
 Kyverno as a Deployment on the control plane

F5 (GPU plane)
└─ NVIDIA GPU Operator targets the GPU workers
 MIG manager: 1st GPU MIG 7g.80gb (= passthrough), 2nd 2×3g.40gb
 OTel Collector as a DaemonSet on the GPU plane + CPU plane
 first vLLM with a 7B FP16 model green

F6 (LLM stack)
└─ The 7 layers reconcile via Flux from a second repo `gitops-llm`
 first public endpoint with Defguard OAuth
 first production customer enrolled under an SLA

The physical distribution of the cluster exploits the isolation between planes defined in F0: the control plane never touches a GPU, the CPU plane concentrates relevant state (Forgejo, Ceph, Postgres CNPG, Langfuse, Defguard) and the GPU plane specialises as much as possible. That separation, decided in F0 before the first server is installed, conditions the success of the remaining phases. It is another reminder of why F0 matters more than it looks.

What we have not covered (upcoming posts)

This post walks the ascent route to the summit. Some pieces still deserve an article of their own:

The safe descent: day-to-day operation, runbooks per component, on-call, continuous capacity planning, the cluster upgrade cycle with no downtime.
Multi-site (the second summit): how two clusters are federated with Cilium Cluster Mesh and what extra phases that introduces. F3.5 (Cluster Mesh) and F4.5 (cross-site replication) are the missing phases.
The brownfield route: what changes when there is already a cluster with workloads. The phases stay the same, but the gates are applied retroactively and every step requires migration planning.
The real calendar cost: typical ranges in weeks for a platform team of 2-3 people, split by phase, with uncertainty bands.
The handoff to operations: how the platform is handed from the deployment team to the operations team, which documents are signed, what is inherited and what is renegotiated.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook documentation — rook.io/docs
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai

Five maturity levels of the platform under the LLM: from a Linux server to a cluster ready for vLLM

Sun, 31 May 2026 07:00:00 +0200

TL;DR

The post on the seven layers of the LLM inference stack took many pieces for granted: an operational Kubernetes cluster, GitOps reconciling, identities resolved, GPUs visible to the scheduler, observability capable of carrying gen_ai.*. Before vLLM makes any sense, you have to reach that starting point, and you reach it by levels. This post defines five maturity levels of the platform that lives under the LLM, from a bare metal server with Linux installed (level 0) to a cluster ready to run the inference layer (level 4) and the handoff to the previous post (level 5). Each level unlocks a concrete capability, running containers reproducibly, rebuilding the cluster from git, authenticating humans via OIDC, scheduling GPUs with MIG and DCGM metrics, demonstrating compliance with no manual intervention, and each has a validation test that decides whether you are really there or just telling yourself you are. For each level: which OSS pieces cover it in 2026 (Cilium, RKE2, Flux, cert-manager, Defguard, NVIDIA GPU Operator, KEDA, Trivy, Kyverno…), the deployment order within the level, the decisions that are expensive to skip, and the antipatterns that drop you a level when you thought you were higher. The thesis: moving up a level costs little effort if you do it in time, and a lot of refactoring if you try to skip it. LLM inference demands at least level 4; anyone trying to serve LLMs from level 1 or 2 ends up paying in night-time incidents what they saved on the platform.

You are here: the five levels at a glance

Before the detail, the ladder. Each rung adds a capability absent from the one below. The level’s test is the question whose honest answer says whether you are already there.

The levels are not interchangeable. A cluster at level 2 cannot run LLMs in production with any guarantee: technically the vLLM pod loads, but at the first night-time incident you discover there is no TLS, no identities, no alerting, no GPU metrics, and no way to know who changed what. Moving up a level after LLMs are already in production costs orders of magnitude more than moving up while the cluster is still empty.

The analogy: from the street stall to the starred restaurant

Picture the scale of a hospitality business. Level 0 is the street stall: a griddle, a gas bottle, a cook improvising. It can sell food, it works, but anything that deviates from a normal day (a health inspection, an allergic customer, an order for 200 portions) sinks the business. Level 1 is the tapas bar: a properly sized kitchen, a short repeatable menu, several shifts. The cook no longer improvises daily; he works from a written menu, even if the recipes live in the head of the boss. Level 2 is the restaurant with a set lunch menu: written procedures, fixed suppliers, stock control, an incident book. If the head cook falls ill, the second can run the service without carnage. Level 3 is the restaurant with an à la carte menu and formal service: traceability of every ingredient, allergens on the menu, health certification, contracts with suppliers, mandatory staff training. Level 4 is the kitchen specialised in a complex product (sushi, haute cuisine, artisan bakery): specific tools the normal restaurant does not need (wood-fired oven, special knives, proving chamber), calibrated processes, quality metrics. Level 5 is the restaurant with a Michelin star: the whole system works, the dish is the result of the organisation, not of one person’s talent.

The analogy holds to the end, including the most interesting detail: you can operate at any level, but the promises you can keep are different. The street stall cannot promise a consistent experience to 80 diners with a booking. The cluster at level 1 cannot promise productive multi-tenant LLM service with an SLA. In both cases the problem is not the technical capability of the last component (the griddle cooks; the pod starts); it is the organisational capability of the whole system.

Let us go level by level.

Level 0 — Chaotic: the server with Linux and nothing else

The capability it gives. Running containers with docker/podman, running binaries, connecting the server to the network. The operator can SSH in, do things, and see results.

The test for the level. “If I reinstall the server from scratch, can I leave it identical to how it was in one afternoon, using only saved notes?”. If the answer is no (because the steps are in the head of whoever built it, in .bash_history, in an out-of-date wiki), you are at level 0.

Minimum pieces to settle before moving up to level 1.

Piece	Suggested decision in 2026	Why it matters when moving up
Linux distribution	Stable Debian or Ubuntu LTS	Long, predictable support, recent kernel available
Kernel	Recent LTS (≥ 6.6) with BPF and modern schedulers	Cilium/eBPF and recent NVIDIA drivers demand it
NVIDIA drivers	The version that matches the CUDA of the LLM engine you will serve	A driver/CUDA mismatch blocks vLLM before you start
Container runtime	`containerd`	CNCF standard, integrated with RKE2/kubeadm
Root filesystem	XFS or ext4 + LVM thin pools	Snapshots, online expansion
Time synchronisation	`chrony` with your own servers	TLS, correlated logs and short certificates demand it
Management network	Dedicated VLAN, ACLs on the switch	Isolate the control plane from workload traffic
Cluster network	LACP + jumbo frames + BGP (if you are going with Cilium)	Intra-node NVLink does not rescue the service network
BMC / IPMI	Out-of-band access with TLS and MFA	Recovery when the operating system does not boot

Antipatterns that pin you at level 0.

Pet servers (with their own names, configured by hand, not replaceable).
Changes applied with vi directly over /etc/... with no commit to a repo.
Deployment with docker-compose with no healthchecks and no automatic restart.
An inventory that lives in an Excel sheet nobody updates.

Deployment order within the level. System image from PXE/cloud-init with the initial configuration (LVM, hostname, network, SSH key, chrony) → bastion/jump host bootstrap → inventory in Ansible (or an equivalent declarative tool, even if it is later replaced) → NVIDIA drivers + container runtime → smoke test (a CUDA container passes nvidia-smi). At this point, the server is ready for Kubernetes to come in.

Level 1 — Repeatable: an operational Kubernetes cluster

The capability it gives. Scheduling containers with a scheduler, network abstraction between pods, persistent volumes, workload lifecycle, manual horizontal scaling.

The test for the level. “Can I lose a node and have the workloads rescheduled without human intervention?”. If yes, you are at level 1. If not, because the pods are pinned to nodes, because there are no replicas, because the PVCs do not reattach, you are still at 0 with Kubernetes on top.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Main alternative
k8s distribution	RKE2 (CIS-hardened by default, no commercial surcharge)	k3s for very small edge, plain kubeadm for custom cases
CNI	Cilium with kube-proxy replacement, BGP, Gateway API	Calico (without BGP it does not compete with Cilium in 2026)
CSI block + filesystem + object	Rook-Ceph (RBD + CephFS + S3-compatible RGW)	OpenEBS Mayastor + Garage for small deployments
Ingress	Cilium Gateway API (better to unify with the CNI)	NGINX Ingress, Traefik
Basic certs	Self-signed bootstrap	(cert-manager comes in at level 3)
Workload handling	`kubectl apply` + Helm from the terminal	No GitOps yet
Container registry	Any internal registry (or a trusted external one) with TLS	(a managed internal registry comes in at level 2)

Antipatterns that drop you to level 0.

Services deployed with kubectl apply from one person’s terminal and with the YAML saved nowhere.
Persistent volumes with no backup policy.
A “single-node cluster” as permanent production, an architectural single point of failure.
A CNI with no NetworkPolicy available or no BGP when the network requires it.

Deployment order within the level. RKE2 installed on at least three nodes for an HA control plane → Cilium installed in kube-proxy replacement mode + BGP control plane → Rook-Ceph on at least three nodes covering block (RBD) + filesystem (CephFS) + object (S3-compatible RGW) with 3× replication or Erasure Coding depending on the pool → smoke test (a Deployment with a PVC starts, the pods reschedule when a node is cordoned, the data persists).

Level 2 — Defined: the cluster rebuilds from git

The capability it gives. The cluster state lives in a repository. Every change goes through a commit. Anyone can rebuild the cluster (or an equivalent one) from the repo and the backups. Basic observability warns when something breaks.

The test for the level. “If I lose the whole cluster, can I recreate it in X hours from the repo + the backups, with no manual intervention beyond the bootstrap?”. The hours are negotiable; what defines the level is that the repo + the backups are enough, not that the person-who-knows is available.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
Forge	Forgejo (or Gitea, GitLab CE)	Self-hosted OSS, community fork of Gitea, open governance
GitOps reconciler	Flux	CNCF graduated, native multi-tenancy, lightweight
Image registry	Forgejo Container Registry	Next to the code, no extra piece
Metrics TSDB	VictoriaMetrics + vmagent	Higher throughput than plain Prometheus, long retention, PromQL compatible
Visualisation	Grafana	The de facto standard
Logs	Loki or Vector	OSS, integrated with Grafana
Alerting	Alertmanager + Keep (OSS orchestrator)	Keep adds multi-channel routing with no lock-in
DB backups	Barman Cloud (Postgres)	The standard for CNPG
Object / dataset backups	Ceph RGW multisite + CephFS snapshots	Cross-pool and cross-site

Antipatterns that drop you to level 1.

kubectl apply run in production outside the repo (undetected drift).
main branches with write permissions for humans without review.
A monolithic repo with no tenant/infra/apps separation (cross changes that cannot be audited).
Metrics kept no longer than 7 days (no SLO observable over a month).
Alerting that fires for everything (fatigue) or for nothing (silence).

Deployment order within the level. Forgejo deployed first (it is a prerequisite for everything else) → Flux installed and pointing at the manifests repo → an initial repository with the Cilium and Rook-Ceph Helm releases reconciled by Flux (replacing the level 1 kubectl apply) → VictoriaMetrics + Grafana + Loki via Helm/Flux → Postgres backups and Ceph snapshots scheduled → smoke test (tear the cluster down, restore from repo + backup, the services come back).

Level 3 — Managed: identities, certificates, secrets and policies

The capability it gives. Every human operating the cluster does so with their own identity (no shared kubeconfig), with MFA and with limited permissions. Automatic internal TLS. Encrypted versioned secrets. Policies that reject insecure configurations before they reach the cluster. A complete audit of who did what.

The test for the level. “If an attacker gets hold of an administrator’s laptop, what can they do in production?”. At level 3 the answer is “little”: MFA blocks the second factor, Kyverno policies block destructive changes without approval, the NetworkPolicies prevent lateral movement, the secrets are encrypted with an external KMS, and the audit log remains. At level 2, “everything”.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
IdP / OIDC	Defguard	Spanish OSS, WireGuard + OIDC + 2FA, multi-org
Federation with the cluster	OIDC in kube-apiserver, OIDC in Forgejo, OIDC in Grafana	Consistent SSO
Internal PKI	cert-manager + Trust Manager	De facto standard, ACME and internal CA
External ACME	Let’s Encrypt for edge certs	No cost, automated
Secrets in git	SOPS + age or an external KMS	Versionable, encrypted in the repo
Secret syncing	External Secrets Operator	Pull from KMS / Vault into the cluster
Policy as code	Kyverno (or OPA Gatekeeper)	Kyverno has a shallower learning curve
NetworkPolicy	Cilium NetworkPolicy + L7	Default deny per namespace
Runtime security	Tetragon (Cilium)	eBPF, complements NetworkPolicy with detection
Vulnerability scanning	Trivy in the CI pipeline + admission	SBOM per image, blocking of critical CVEs
Audit log	kube-apiserver with `--audit-policy-file` sent to Loki	Regulatory traceability

Minimum Kyverno policies to keep alive.

Deny :latest images or images without a sha digest.
Deny pods without securityContext.runAsNonRoot=true.
Deny pods without resources.limits (CPU + memory).
Deny Services without an owner=<team> label.
Deny changes in critical namespaces (kube-system, flux-system) without an approval label.

Antipatterns that drop you to level 2.

A kubeconfig shared between administrators.
Secrets in plain data: in a manifest committed to the repo.
NetworkPolicy absent from new namespaces by default (implicit allow-all).
kubectl edit or kubectl patch in production without going through the repo.

Deployment order within the level. Defguard deployed and enrolled with WireGuard / OIDC → OIDC integration with kube-apiserver, Forgejo and Grafana → cert-manager installed and issuing internal certificates (your own CA for mTLS, Let’s Encrypt for the edge) → SOPS configured and External Secrets Operator installed → migration of secrets from plain to encrypted → Kyverno with initial policies in audit mode, then enforce → default-deny NetworkPolicy per namespace → Tetragon enabled → smoke test (try to bypass each policy and check that admission rejects it).

Level 4 — Optimised for GPU: the cluster now knows what an H100 is

The capability it gives. The Kubernetes scheduler sees the GPUs, distinguishes them, can partition them (MIG) or multiplex them (time-slicing), expose DCGM metrics, autoscale with KEDA using metrics from the LLM workload itself (vllm:num_requests_running, vllm:gpu_cache_usage_perc), and carry traces with the GenAI semantic conventions. Everything needed for the LLM inference stack to rest on a platform that understands its nature.

The test for the level. “If I deploy a pod asking for nvidia.com/gpu: 1, does it get scheduled on the right GPU, with the right slice, with DCGM metrics exposed, with GenAI observability ready to receive spans?”. If yes, you are at level 4. If the answer requires “it depends which node and who deploys it”, not yet.

Minimum pieces for the level.

Piece	Suggested decision in 2026	Why
GPU device plugin	NVIDIA GPU Operator	Deploys drivers, container toolkit, DCGM and MIG manager with one operator
HW partitioning	MIG (Multi-Instance GPU) on the H100 where it applies	Real hardware isolation, not time-slicing
GPU metrics	DCGM Exporter	SM utilisation, VRAM, temperature, throttling, NVLink bandwidth
LLM metrics	vLLM Prometheus endpoint + scrape	TTFT, TPOT, KV cache, prefix hit rate
Autoscaling	KEDA with a Prometheus ScaledObject	Scales on LLM metrics, not on CPU
LLM operators	vLLM Production Stack / OME (Operator Model Engine)	Declarative handling of models / adapters
Traces	OpenTelemetry Collector with OTLP receivers + processors + exporters	`gen_ai.*` semantic conventions (post)
LeaderWorkerSet	LeaderWorkerSet API (k8s 1.30+)	Tensor parallel topology coherent with NVLink
Topology Manager	enabled with `single-numa-node`	Pinning GPU pods to the right NUMA node

The key decision: MIG, time-slicing or passthrough.

MIG divides an H100 into 1g.10gb, 2g.20gb, 3g.40gb, 7g.80gb (slices with real HW isolation). Useful for serving several small models or reserving capacity per tenant with a guarantee. Limitation: up to 7 instances per GPU, predefined profiles.
Time-slicing shares a GPU between several pods with no HW isolation. Useful for dev/test, not for multi-tenant production with an SLA.
Passthrough assigns the whole GPU to one pod. Useful for tensor parallel across several GPUs on the same node (a large LLM with TP=4).

For a productive LLM platform, the practical rule: passthrough for the large models with TP, MIG for embeddings and small models that cohabit, never time-slicing in production.

Antipatterns that drop you to level 3.

Installing NVIDIA drivers by hand outside the GPU Operator (silent breakage when Kubernetes is upgraded).
Serving an LLM with requests.gpu: 1 without having decided on MIG / passthrough (you end up with idle GPUs from fragmentation or pods stepping on each other).
KEDA autoscaling on CPU (the classic HorizontalPodAutoscaler) on pods that are almost always at 10 % CPU but 95 % KV cache.
OpenTelemetry deployed but without the gen_ai.* semantic conventions (the traces are not LLM-aware).

Deployment order within the level. NVIDIA GPU Operator installed via Helm/Flux with the driver version that matches the chosen LLM engine → DCGM Exporter enabled and metrics visible in Grafana (NVIDIA dashboards imported) → MIG manager configured for the nodes where it makes sense (a typical mix on a 4×H100 SXM cluster: two GPUs with full passthrough for the general LLM at TP=4, two GPUs partitioned into 2×3g.40gb each for small LLMs + embeddings) → OpenTelemetry Collector with attributes processors to enrich spans with your own labels (tenant_id, priority_tier) + exporters to Langfuse and to Tempo → KEDA installed with an example ScaledObject pointing at vllm:num_requests_running → vLLM Production Stack or OME to declare models as a CRD → smoke test (a vLLM Deployment declared via CRD starts, serves a token, exposes metrics, the trace reaches Langfuse, KEDA scales under synthetic load).

Level 5 — Handoff: the cluster is an LLM platform, the seven layers go on top

Once at level 4, the cluster meets the contract that the post on the seven layers assumed as its starting point. Level 5 adds no infrastructure: it adds the LLM stack proper. For completeness, the seven components of level 5 are:

Gateway (Envoy AI Gateway) — it comes first, directing traffic to LLM inference and embeddings.
LLM inference (vLLM Production Stack or OME with vLLM) — on the GPUs already discovered by the level 4 GPU Operator.
Embeddings + reranker (Infinity, TEI) — a pod separate from the LLM, already covered in the previous post.
Vector store + relational data (Qdrant, PostgreSQL CNPG, Ceph RGW for weights and adapters, CephFS for datasets) — most of it already existed at level 2 as data; now it specialises for RAG.
LLM-aware observability (Langfuse) — it plugs into the level 4 OTel chain.
GitOps control plane — the one from level 2 remains the only legitimate authority.
Dependency tracking (Hubble flows + Otterize) — on the Cilium that already existed at level 1.

The criterion for promoting from level 4 to level 5 is not technical: it is contractual. The cluster already supports LLMs; the decision is when to open real customer traffic. Promotion demands: a green golden eval of the model, a signed incident runbook, negotiated SLOs, a continuity plan, and mapping to ENS / NIS2 / 42001 where it applies.

The maths that matter: what skipping a level costs

To quantify the thesis of the post, an order-of-magnitude estimate of the cost of moving up each level in time versus moving up after production is running. The figures are engineering time with a small platform team (2-3 people), assuming templates and prior experience.

Level	Time to build on an empty cluster	Time to retrofit with production running
0 → 1	1-2 weeks	1-2 weeks (little downstream refactoring)
1 → 2	2-3 weeks	4-8 weeks (migrating everything to git)
2 → 3	2-4 weeks	8-16 weeks (image rebuilds, secret migration, retroactive RBAC)
3 → 4	1-2 weeks	4-8 weeks (reconfiguring GPUs, moving models to MIG, instrumenting `gen_ai.*`)
4 → 5	1-2 weeks	2-4 weeks
Total 0 → 5	~10-15 weeks	~20-40 weeks if done in the wrong order

The typical multiplier observable in practice: 2× to 3× the cost if it is done in the wrong order. And that assumes it gets done at all, since many projects never get past level 2 because “the identity thing” can always wait for another sprint. By the time the incident arrives, it is too late to start.

Beyond time, the operational cost (night-time incidents, security escapes, invisible debt) grows exponentially with the gap between the actual level and the required level. A cluster at level 2 serving productive LLMs to regulated customers is a time bomb: technically it works, organisationally it does not.

Final diagram: the complete ladder with pieces

The ladder is not decorative: each level enables the next. You cannot have LLM-aware observability (level 4) without OTel deployed via Flux (level 2). You cannot have automatic internal TLS (level 3) without a root PKI that lives somewhere (registry and certificates managed from level 2). You cannot have KEDA scaling on vLLM metrics (level 4) without Prometheus / VictoriaMetrics scraping (level 2). The levels are not a conceptual hierarchy: they are a hierarchy of installation dependencies.

Typical design decisions that break progress

Mistakes seen repeatedly that drag the cluster back a level:

1. Jumping from level 1 straight to level 4. “We are in a hurry to serve the LLM, we will do the identity and GitOps stuff later.” Later is always two orders of magnitude more expensive and always arrives after the first incident.

2. Confusing Helm with GitOps. Having Helm charts is not level 2. It is level 1 with templates. Level 2 demands that a reconciler (Flux/ArgoCD) applies the charts from a repo, detects drift and warns.

3. cert-manager with no usage policy. Having auto-renewed certificates but using TLS only at the ingress, with no internal mTLS between services, leaves the TLS promise lame and turns level 3 into a cosplay of level 3.

4. NVIDIA drivers by hand. It works on day one and breaks the day of the first kernel upgrade. The rule: drivers always via the GPU Operator, never operating system packages.

5. Prometheus metrics but 7 days of retention. Without long retention (≥ 90 days) there is no honest SLO. VictoriaMetrics with a year of retention costs little more than Prometheus with 7 days, and it unlocks compliance and serious postmortems.

6. OIDC only for kube-apiserver. If Forgejo, Grafana, Defguard and vLLM each have their own auth system, you do not have SSO, you have islands. An honest level 3 demands federation.

7. Kyverno in permanent audit mode. Policies that do not reject are not policies, they are alerts. At some point you have to move to enforce. Until then, you are still at level 2 wearing a level 3 face.

8. MIG with no conscious decision on the profile. Configuring MIG with the default profile without having measured the size of the models that will cohabit leaves fragmented GPUs with slices nobody uses. The rule: MIG only if you have measured and decided the profiles in advance.

They all share one root: declaring the level without passing the level’s test. Saying “we already did GitOps” when things are still applied with kubectl edit in prod. Saying “we already did identity” when there is a shared admin kubeconfig. Saying “we are ready for LLMs” when there is no DCGM Exporter and no Langfuse plugged in.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), a reasonable setup after passing the five levels distributes the components like this:

control plane (3 nodes without GPU, hostnames cp-01..03)
├── kube-apiserver, etcd, controller-manager, scheduler
├── Flux, Forgejo, cert-manager, External Secrets, Kyverno
└── Tetragon (DaemonSet here too)

worker plane (≥ 3 nodes without GPU, hostnames worker-cpu-01..03)
├── Cilium agent (DaemonSet)
├── Rook-Ceph OSDs + MONs + MDS (CephFS) + RGW (S3)
├── VictoriaMetrics + Grafana + Loki
├── Defguard (StatefulSet)
└── Langfuse + OTel Collector

GPU worker plane (≥ 2 nodes with 4×H100 SXM, hostnames worker-gpu-01..02)
├── NVIDIA GPU Operator (driver + container toolkit)
├── DCGM Exporter (DaemonSet)
├── MIG manager (configuring the chosen profile)
├── vLLM (Deployment) — general LLM TP=4 occupies 4 GPUs (passthrough)
├── vLLM (Deployment) — code LLM TP=2 occupies 2 GPUs
├── Infinity (embeddings) — 2 replicas cohabit on 2 MIG slices
└── KEDA scaler listening to vLLM metrics

The operational rule: the control plane and the CPU plane are separated from the GPU plane. An incident on the GPU plane must not take the control plane with it (the control plane is what recovers the cluster). And the CPU plane concentrates everything that moves relevant state (Forgejo, Rook-Ceph, Postgres CNPG, Langfuse): it is the heart to protect.

The GPU hardware specialises as much as possible: GPU pods run only on GPU nodes, and GPU nodes run nothing CPU-bound beyond the operational overhead (Cilium, GPU Operator, DCGM). This is enforced with nodeSelector + taints/tolerations + a Kyverno policy that rejects pods with no GPU requests being scheduled onto GPU nodes.

What we have not covered (upcoming posts)

This post walks the vertical path upwards. Some horizontal and cross-cutting pieces still deserve an article of their own:

Multi-site active/standby: how two clusters are federated with Cilium Cluster Mesh and what changes at each level when there are two sites instead of one.
Migration between levels with real traffic: how a cluster already in production is retrofitted to the next level with no downtime.
Day-to-day operation: runbooks per level, which dashboards to look at each morning, which SLOs to define per component.
The cost plane: what each level costs in hardware, energy, engineering hours, optional OSS licences (commercial support from Rancher, Cilium Enterprise, etc.) and when each expense is justified.
Operationalised compliance: how levels 3 and 4 map to ENS High, NIS2 and ISO/IEC 42001 controls without turning the cluster into a paperwork exercise.

References

RKE2 Documentation — docs.rke2.io
Cilium documentation — docs.cilium.io
Rook-Ceph — rook.io
Flux GitOps toolkit — fluxcd.io
Forgejo — forgejo.org
cert-manager — cert-manager.io
External Secrets Operator — external-secrets.io
Kyverno — kyverno.io
NVIDIA GPU Operator — docs.nvidia.com/datacenter/cloud-native/gpu-operator
DCGM Exporter — github.com/NVIDIA/dcgm-exporter
KEDA — keda.sh
LeaderWorkerSet API — github.com/kubernetes-sigs/lws
vLLM Production Stack — docs.vllm.ai/en/latest/serving/production_stack.html
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
CIS Kubernetes Benchmark
NIST SP 800-207 — Zero Trust Architecture

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

Thu, 21 May 2026 07:15:00 +0200

TL;DR

Having an expensive GPU cluster and many different workloads that want to use it is not an infrastructure problem: it is an internal product problem. What separates “we have a cluster” from “we have an inference platform” are four layers the market consolidated in 2026: a gateway layer that centralises authentication, routing and policies (LiteLLM, Portkey, Kong AI Gateway); a GPU isolation model appropriate to the tenants’ profile (MIG hardware isolation for untrusted multi-tenant, MPS for processes from the same team, time-slicing only for dev); a quota and rate limiting system with budgets per tenant/team/project (LiteLLM does this in its core at team/user/api-key level with descriptive 429s); and a multi-tenant observability plane that enables real cost attribution (showback as an intermediate step, chargeback as the destination), per-tenant tracing and differentiated dashboards. Applied to a typical mid-scale GPU cluster (one node with 4-8 H100 SXM and NVLink, a common place to start in production), this translates into concrete decisions: with ~640 GB of aggregate VRAM across 8 GPUs and two typical production models (a large 70B+ model with tensor parallel and a replicated mid-sized model), the cluster serves between tens and low hundreds of simultaneous sessions depending on the mix; GPU isolation is usually resolved with MIG on smaller workloads and per-model dedication on large ones; and the platform’s success metric is effective utilisation, which in typical production sits at 30-40% and where the reasonable optimisation target is to push it to 60-70% without degrading SLAs.

This is the fifth post in the MLOps for LLMs series. It is the most operationally oriented one and cuts across several pipeline stages (Deploy + Observe plus cross-cutting concerns). The “you are here” marker flags the two active stages because the notion of a multi-tenant platform does not live in just one.

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

When a platform team buys expensive GPU hardware and starts building inference, the first version is almost always single-purpose: one model, one client, one target latency. It works. When the second team turns up asking for the same resource, that single purpose becomes internal politics: how many replicas do we give them? What do we do if the SLAs clash? Who pays for the tokens of team B’s experiment? And when the third one arrives, what was an SRE project becomes an internal product project.

The distinction is not technical, it is one of framing. A cluster is infrastructure. A platform is a service with customers, contracts and success metrics. The change of framing implies:

Identifiable customers (tenants), not anonymous users.
Contracts (latency SLA, guaranteed throughput, available models), not “whatever we manage”.
Success metrics that are not technical but product ones: adoption, satisfaction, cost per query per tenant, time to the first “hello world”.

This post walks through how that change of framing is carried out. It lands it on a mid-scale cluster (4-8 H100 SXM with NVLink in a single node), the usual configuration when starting with serious LLM inference; but the principles generalise to any topology, from a single node with two GPUs to multi-node clusters with InfiniBand.

The four layers of a multi-tenant inference platform

The canonical architecture that settled in 2026 has four layers that any serious multi-tenant platform implements, from the outside in:

Each layer solves a concrete problem. Let us take them one by one.

Layer 1 — AI Gateway: the single front door

The AI Gateway is the component your tenants see. It is an HTTP/gRPC API compatible with OpenAI (typically /v1/chat/completions, /v1/embeddings, /v1/models) that centralises everything that happens before touching the inference backends.

Why centralise

Without a gateway, tenants connect directly to vLLM or whatever the model is. Every change (rotating an endpoint, adding a model, changing credentials, applying a policy) means notifying every tenant. Every tenant has its own retry logic, its own logging, its own auth model. It is unworkable from the third customer onwards.

With a gateway, the change is made in one place. Tenants have a stable URL and a set of credentials; the rest is the gateway’s problem.

The three dominant options in 2026

LiteLLM is the most popular OSS option, Python-first, deployed as a proxy. It supports 100+ providers (OpenAI, Anthropic, Bedrock, self-hosted vLLM, Ollama, and so on) behind a unified OpenAI-compatible API. It has a native multi-tenant hierarchy with Organizations → Teams → Users → API Keys, each level with its own budget. The Apache 2.0 version covers the basics; RBAC, SSO, audit logs and team-level enforcement require the paid Enterprise version. Deployed on K8s with the official Helm chart.

Portkey is the most mature commercial / SaaS option. A single control plane that enforces budgets, quotas, permissions and compliance. Real-time spending tracking with alerting. RBAC, audit, workspaces and SSO included. The trade-off: dependency on an external service and a per-request pricing model.

Kong AI Gateway is the option for organisations that already have Kong as their API gateway. An AI plug-in on top of the existing Kong gateway, integrated with its plugin, consumer and rate-limit model. If your platform team already runs Kong, it is the lowest friction.

When to choose each one

Situation	Gateway
Pure OSS, self-hosted, Python-first team	LiteLLM
You need RBAC, SSO, audit log out of the box, and have budget	Portkey
You already run Kong as a corporate API gateway	Kong AI Gateway
Greenfield enterprise with strict compliance	Portkey (probably)
Mid-sized OSS-first company without regulated compliance	LiteLLM (typically)

The minimum the gateway has to do

Whichever option you pick, this is what any serious deployment must enforce:

Auth and identity: every request carries an API key resolvable to a tenant + user + team.
Routing by model: the tenant asks for model: "gpt-4o"; the gateway decides whether it goes to OpenAI, to Azure OpenAI, or to your vLLM with Qwen3 32B (a cheaper fallback), according to policy.
Rate limiting: RPS per tenant, TPM (tokens per minute), concurrency limits.
Caching of identical responses: 5-30% of RAG queries are repeats; caching saves latency and cost.
OTel emission: every call produces a span with gen_ai.* semantic conventions and tenant_id as an attribute. Covered in the Evals post and MCP observability.
Failover: if vLLM goes down, the gateway redirects to the OpenAI API. If OpenAI rate-limits, the gateway falls back to Anthropic. Configurable policy.

Example of a multi-tenant LiteLLM configuration

# litellm-config.yaml — simplified example
model_list:
 - model_name: llama-3-70b
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-llama3-70b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: qwen3-32b
 litellm_params:
 model: openai/qwen3-32b
 api_base: http://vllm-qwen3-32b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: gpt-4o
 litellm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

router_settings:
 routing_strategy: usage-based-routing-v2
 fallbacks:
 - llama-3-70b: [qwen3-32b, gpt-4o]  # if vLLM goes down, fall back to the external one

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL  # Postgres for budgets/keys

# Hierarchy: Organizations → Teams → Users → API Keys
# Created via the API, not in static YAML

Creating a team with a budget:

curl -X POST http://litellm/team/new \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_alias": "soporte-chat",
 "max_budget": 500, # 500 USD/month
 "budget_duration": "30d",
 "tpm_limit": 100000, # 100K tokens/min
 "rpm_limit": 1000, # 1000 requests/min
 "models": ["llama-3-70b", "qwen3-32b"] # access to these
 }'

And the team’s API key:

curl -X POST http://litellm/key/generate \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_id": "<team-id>",
 "duration": "30d",
 "metadata": {"environment": "production", "app": "support-bot"}
 }'

That API key is what the tenant uses. Every request made with it consumes from the team’s budget. When it runs out, LiteLLM returns HTTP 429 with a description.

Layer 2 — Policy & Quota Plane: what each tenant can do

The gateway is where enforcement happens. Policy is what gets enforced. Five axes of multi-tenant policy:

Technical quotas

TPM (tokens per minute): the hard consumption limit. For a Llama 3 70B at TP=5, ~3000 sustained output tokens/s = 180K aggregate TPM. If you have 10 tenants, assign 18K each as the ceiling.
RPS / RPM: load control, not consumption control. A 4K-token session counts as one request; so does a batch of 100 mini-completions. Useful against abuse.
Concurrency: how many simultaneous active requests per tenant. Important for latency SLAs: 100 RPS with concurrency=50 works out at 2 seconds per request.

Economic budgets

Monthly per tenant: a hard cap in USD.
Daily and hourly: soft caps to avoid a runaway in a single day.
Per project / API key: fine granularity within a single tenant.

LiteLLM has a max_budget field at every level of the hierarchy (organization, team, user, api key). Budgets are inherited and constrained downwards.

Model whitelist and blacklist

Tenants with critical workloads → only stable models (llama-3-70b, gpt-4o). Research tenants → access to experimental models as well.

Priority classes

Not all requests are equal. Three typical classes:

Guaranteed: workloads with an SLA, latency respected even under pressure.
Best-effort: normal workloads without a strict SLA.
Spot: batches that can wait, evictable if a guaranteed one arrives.

The Token Management in Multi-Tenant AI Inference Platforms paper (2026) formalises this with a model of token pools per priority class that has started to be adopted in production. It maintains guaranteed P99 latency for guaranteed workloads even under overload, with selective throttling of spot.

Admission control

Before accepting a request: is there capacity? If not, return 429 immediately instead of queueing and degrading everyone. It is the most underrated operational discipline. A cluster with admission control done properly has predictable latency; without it, catastrophic degradation when the peak arrives.

The typical pattern in 2026

# Conceptual policy for a "soporte-chat" tenant
tenant: soporte-chat
quotas:
 tpm: 50000
 rpm: 500
 max_concurrency: 30
budget:
 monthly_usd: 800
 alert_thresholds: [0.5, 0.8, 0.95] # warn when you get there
models_allowed:
 - llama-3-70b
 - qwen3-32b
priority: guaranteed
fallback_on_overload:
 - qwen3-32b  # if guaranteed fills up, fall back
 - gpt-4o-mini  # last resort, external model

Layer 3 — Isolation Plane: isolating workloads physically

This is the technically densest layer. You have a node with several H100 SXM GPUs interconnected by NVLink. How do you partition them between tenants?

MIG (Multi-Instance GPU) is the strongest isolation. It partitions the GPU into up to 7 instances with physically separate HBM memory and dedicated compute units (SMs). Tenants in different MIG instances cannot touch each other: one workload does not consume memory another needs, one does not degrade another’s throughput. Hardware isolation. Available on A100, H100, B100 and B200.

MPS (Multi-Process Service) is soft. Several processes share the GPU concurrently and NVIDIA distributes SMs according to usage. Good performance if all the processes are yours and you trust them. Worse for multi-tenant setups between customers who do not know each other, because a noisy process can degrade the others.

Time-slicing is the simplest: the GPU is assigned alternately, slot by slot, to different processes. Much worse latency (waits between slots); not recommended for production workloads with an SLA.

The choice for multi-tenant in 2026

According to the enterprise adoption survey: 80% use MIG for untrusted multi-tenant (different customers who do not know each other) and MPS for trusted environments (processes from the same team) where you want to maximise throughput. Time-slicing is only used in dev/staging so that every developer can touch a GPU without the cost of exclusivity.

An important limitation of MIG: it isolates compute and HBM memory, but the PCIe path remains shared. For PCIe-bound workloads (a lot of host↔device traffic), tenants in different MIG instances can still affect each other. For LLM inference the main path is HBM, so this is rarely a problem. But it is worth knowing.

MIG partitions on the H100

An H100 (80GB HBM3) can be partitioned into fixed profiles:

Profile	SM	Memory	Max instances per GPU
1g.10gb	14	10 GB	7
1g.20gb	14	20 GB	4
2g.20gb	28	20 GB	3
3g.40gb	42	40 GB	2
7g.80gb	98	80 GB	1 (the whole GPU)

For a mid-scale cluster with NVLink, MIG has a fundamental problem: when you partition with MIG, NVLink between GPUs is disabled. An H100 in MIG mode does not take part in multi-GPU tensor parallel. If you are going to serve a large model with tensor parallel (Llama 3 70B with TP=4 or TP=8, for instance), those GPUs have to be whole, without MIG.

That defines the architectural decision. There are two main approaches:

Approach A — A shared large model with quotas at the gateway

All the node’s GPUs serve a single large model with tensor parallel spanning the whole node. All tenants share that instance. Isolation happens at the gateway layer (quotas, rate limiting) and the policy layer (priority classes). The cluster’s kernel is one enormous vLLM instance with --max-num-seqs=128 or similar; internally, vLLM shares GPU time between the active requests with continuous batching.

Advantages: you use all the GPUs to the full, NVLink stays active, better KV cache utilisation. Disadvantages: soft isolation. A tenant that saturates does not degrade the others directly (vLLM batches), but it does compete for batch slots. You need serious priority classes.

Approach B — Dedicating GPUs per model / tenant

You split the GPUs into pools dedicated to different models. Examples on an 8-GPU node:

4 GPUs: a large 70B model with TP=4.
2 GPUs: a mid-sized 32B model replicated (2 independent instances) for tenants with a strict SLA.
2 GPUs: miscellaneous workloads (smaller models, experimentation).

Advantages: physical isolation between critical models / tenants. Disadvantages: worse aggregate utilisation; some GPUs idle while others saturate.

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

If you have small workloads (4B, 7B models), you can use MIG on 1-2 GPUs to serve them and dedicate the rest to tensor parallel for the large model. It combines strong isolation for the small workloads with full use of NVLink for the large model.

The operational choice: start with A, move up to C if you need to

In most deployments, Approach A (a shared large model plus quotas) is the right starting point. Utilisation is better, operation is simpler, and the gateway’s soft isolation works for reasonable workloads.

When there is a tenant with a strict SLA that cannot tolerate competing with others, you move to Approach B for that tenant in particular (dedicating GPUs to an instance of the model just for them), keeping the rest of the cluster shared.

Approach C is for when you have 10+ tenants with very heterogeneous profiles.

Isolation at the Kubernetes level

Independently of GPU isolation, pod-level isolation is applied in K8s:

Namespaces per tenant: tenant-soporte, tenant-legal, and so on.
ResourceQuotas and LimitRanges: CPU/memory limits per namespace.
NetworkPolicies: tenant A cannot talk to tenant B’s namespaces.
K8s PriorityClasses: classes with a numeric value defining preemption order if a more critical pod arrives.
PodDisruptionBudgets: how many pods of each deployment can go down simultaneously.

Layer 4 — Observability Plane: seeing what happens per tenant

The fourth layer: observability with a tenant dimension. Without it you cannot do cost attribution, you cannot debug incidents for a single tenant, you cannot show dashboards to stakeholders.

The four mandatory properties

1. tenant_id on every span. The AI gateway resolves the API key and attributes a tenant_id. That ID is propagated via params._meta or OTel headers to every downstream component (vLLM, retrieval, MCP servers, tools). Any span in any system carries that label. It is what lets you reconstruct tenant-specific traces.

2. Metrics labelled per tenant. gen_ai.usage.input_tokens{tenant="soporte-chat"} or equivalents. Prometheus, Grafana, groupable by tenant.

3. Real cost attribution. The sum of tokens × cost/token per tenant gives the cost. For self-hosted vLLM, the cost is per GPU hour plus the proportional share of tokens (you can compute an equivalent cost per 1k tokens).

4. An immutable audit log. Every API key used, every model invoked, every quota change, every budget exceeded. For compliance.

Showback vs chargeback

An important FinOps distinction that gained clarity in 2026:

Showback: visibility without consequence. “Support team, you have consumed 623 USD this month on LLMs”. Information, not an invoice. It lets you spot abuse without penalising anyone before the team understands what is going on.

Chargeback: the cost is charged to the team’s budget. When it runs out, it runs out. It changes behaviour.

The practice that works: 6-18 months on showback while tags are calibrated, misattributions are identified and teams are trained. Then chargeback, once the numbers are credible. Launching chargeback on day 1 when the costs are still dirty creates an immediate political fight; launching showback prepares the ground so that chargeback lands in an orderly way.

Only 14% of organisations have chargeback active according to a recent survey, which indicates this is still mostly showback in real production.

Tools

Kubecost: cost allocation per namespace, deployment and pod on Kubernetes. For the cost of the shared GPU, allocate proportionally to the tokens consumed per tenant.
Finout: a FinOps platform that combines cloud bills and LLM API costs in a unified view with virtual tagging.
Langfuse: already covered. Cost tracking per trace, groupable by user or session metadata.
LiteLLM native tracking: LiteLLM’s master DB keeps running spend per team, user and API key, accessible via the API or the UI.

The minimum multi-tenant dashboard

Any platform should have:

Summary per tenant: monthly spend, current RPS, TPM consumed, % of budget spent, active sessions.
Top users within each tenant (for detecting internal abuse).
p95 latency per tenant: SLA tracking.
429 / 503 errors: how many requests are being rate-limited or rejected because of overload.
Cost trend: the monthly trajectory with a projection.
Drift per tenant (from the post-tracing series): if a tenant starts getting worse results, raise an alert.

Sizing on mid-scale GPU clusters: concrete decisions

Let us get down to hardware. As a reference we take a node with N H100 SXM (between 4 and 8) with NVLink/NVSwitch, 80 GB HBM3 each. That gives between 320 GB and 640 GB of aggregate VRAM. Inter-GPU connectivity is 900 GB/s (NVLink 4) or 600 GB/s (NVLink 3) depending on generation. HBM bandwidth per GPU is 3.35 TB/s.

Default decisions

Start with Approach A: all the node’s GPUs serving a single large 70B model in BF16 with tensor parallel = N. Expected real capacity (computed for a standard 8-GPU HGX node as an example; it scales roughly linearly with N):

Model VRAM (70B BF16): ~140 GB (≈ 17.5 GB/GPU at TP=8).
vLLM overhead + activations VRAM: ~10 GB/GPU.
VRAM free for KV cache: ~52 GB/GPU. On an 8-GPU node that is ~416 GB aggregate; on a 4-GPU one, ~210 GB.
With --kv-cache-dtype=fp8 and a 70B GQA model: ~320 KB/token.
Aggregate cache capacity (8-GPU node): ~1.3M tokens to distribute between simultaneous sessions.

That translates into throughput and concurrency (indicative figures for an 8-GPU node):

Simultaneous sessions	Average context per session	Aggregate throughput (tokens/s)
32	16K	~5000
64	8K	~8000
128	4K	~12000

Typical latencies: TTFT ~150ms at low traffic, TPOT ~15-20 ms/tok. Under high concurrency, TTFT rises to ~500ms if the queue is saturated.

An example tenant scheme

A cluster with 4 tenants and a research pool:

Tenant	TPM cap	RPM cap	Concurrency	Budget	Priority	Models
Support chat	80K	800	50	1500 USD/month	Guaranteed	llama-3-70b, qwen3-32b
Legal RAG	30K	200	15	600 USD/month	Guaranteed	llama-3-70b
Code agent	50K	300	25	1200 USD/month	Best-effort	llama-3-70b, qwen-coder
Data extr. batch	40K	1000	40	400 USD/month	Spot	llama-3-70b, qwen3-32b
Research / notebooks	10K	100	5	200 USD/month	Spot	all

TPM total: 210K. Aggregate cluster capacity: ~180K sustained TPM. It is overcommitted by about 15%, assuming not every tenant hits the ceiling at the same time. That is normal and desirable; if they all do it at once, the priority classes degrade in an orderly way.

When to add hardware

Signals that the node has become too small:

Sustained p95 TTFT > 500 ms during peak hours → the queue is building up.
vllm:num_requests_waiting constantly > 20 → admission control starting to reject.
Sustained GPU utilisation > 80% at critical hours without dropping in quiet hours → there is no headroom.
429 rate on guaranteed tenants > 1% → the platform is breaking its SLA in production.

When several of these hold, the natural next step is to add another HGX node with internal NVLink and stand up a second vLLM instance of the same model. The gateway load-balances between the two instances. Aggregate throughput doubles; latency stays the same.

Common operational traps

A gateway without auth: a backdoor into the cluster

Your vLLM sits on a ClusterIP Service and the main app talks to it. Some tenant discovers the endpoint directly and hits it without going through the gateway. Quotas and costs are silently evaded. Use a strict NetworkPolicy: only the gateway can talk to the vLLM Services; the rest of the cluster cannot.

MIG and NVLink are incompatible

You enable MIG on a GPU thinking you will get isolation plus multi-GPU; you discover that MIG disables NVLink. Any large model with TP becomes unusable. Decide MIG vs NVLink globally per cluster, not per individual GPU.

Quotas pinned to the cluster’s ceiling

You add up the TPM of all the tenants and it comes to exactly the cluster’s capacity. When two tenants peak at the same time, both wait or one gets rejected. An overcommit of 10-20% is healthy (it assumes they do not all peak at once); more than that is dangerous.

No multi-tenant observability from day 1

You launch with quotas and isolation but without tenant_id on spans. Three months later, your CFO asks “how much does the support agent cost versus the legal one?” and you cannot answer. OTel with a mandatory tenant_id from the first version, even if there are no dashboards yet; having the data is worth more than having perfect dashboards with no data.

Showback that never reaches chargeback

You have been on showback for 18 months, the teams know the numbers, nobody changes behaviour. Without the pressure of real chargeback, the incentive dilutes. Set an explicit calendar for the transition to chargeback, with an owner and a deadline.

Non-whitelisted models eating the budget

A team discovers that LiteLLM has gpt-4o configured. They use it without permission. The budget burns on an external API when the idea was to use the cheap self-hosted one. Keep an explicit per-team whitelist of accessible models.

Badly calibrated priority classes

Everyone declares themselves “guaranteed”. At the first peak, there is nothing left to degrade and everything suffers. Priority classes only for critical cases, with justification. The majority should be best-effort.

No failover from the gateway

Your vLLM goes down. The gateway has no fallback configured and returns 503 to every tenant. Configure a fallback to another model, ideally an external one (OpenAI) for guaranteed workloads, even if it costs more per hour. Availability is worth more than cost per hour.

An operational roadmap for getting started

If you start from scratch with an empty GPU node, the minimum order is as follows. Each milestone is a day’s work with slack, not a tight one:

Day 1-2 — Base K8s infrastructure. NVIDIA GPU Operator + nvidia-device-plugin + dcgm-exporter + cluster-default NetworkPolicies. Validation: a basic pod with nvidia.com/gpu: 1 gets scheduled.

Day 3 — vLLM with a large model and tensor parallel across the whole node. The vLLM Production Stack Helm chart (or bare vLLM manifests). Model weights on a shared PVC (CephFS or NFS). Validation: a curl request against the internal Service responds.

Day 4 — AI Gateway: LiteLLM. Helm chart, Postgres for budgets, master key, a first model_list pointing at vLLM. Validation: an OpenAI-compatible request through LiteLLM responds with the same content as vLLM directly.

Day 5 — Basic multi-tenancy. Create teams, API keys, budgets, model whitelists. Test with two teams. Validation: the second team, using the model it does not have whitelisted, gets a 403.

Day 6 — Minimum observability. Prometheus + Grafana scraping vLLM and LiteLLM. A dashboard with TTFT, TPOT, throughput, num_requests_waiting and budget_consumed_per_team. Validation: visible in Grafana with real data.

Day 7-8 — Pilot customer. One real tenant (ideally a controlled internal one) starts using it. Measure real latencies, discover the first operational incidents.

Day 9-10 — Tuning. Adjust --max-num-seqs, --gpu-memory-utilization, priority classes and quotas based on what the pilot taught you.

Day 11-14 — Onboarding the second tenant plus iteration. Repeat. Every new tenant onboarded reveals new cases.

After two weeks you have an operational platform with two real tenants and data to decide whether it is ready for more. The line of progress from here on is horizontal (more tenants) until you saturate; from then on, vertical (more hardware).

What we have not covered (upcoming posts)

Continuous fine-tuning in production (post 6, already decided): LoRA/QLoRA/DPO, dataset curation, eval gates, A/B versioning with real traffic between model versions.
Constitutional AI and alignment at runtime: an option still on the table.
Edge LLMs: when an H100 cluster is too expensive for a specific workload, distilled models running on NPUs or consumer GPUs.
GPU networking deep dive: NCCL, InfiniBand, GPUDirect, RDMA. For multi-node clusters with cross-host tensor parallel.

References

Multi-tenancy and GPU isolation:

Multitenant GPU Infrastructure: 4 Powerful Design Rules — survey of enterprise patterns.
Run Multiple LLMs on One GPU: MIG, Time-Slicing, and MPS Guide (Spheron).
A Practical Guide to GPU Partitioning with MIG (Medium).
GPU Partitioning for AI Workloads: NVIDIA MIG with SUSE Virtualization (KubeCon EU 2026).
Predictable LLM Serving on GPU Clusters (arxiv 2508.20274).
Token Management in Multi-Tenant AI Inference Platforms (arxiv 2603.00356) — paper on priority + admission control.

AI Gateways:

Multi-tenant FinOps:

Cross-references:

Previous posts in series 4: MLOps for LLMs overview, RAG over Kafka, The six-stage pipeline, PostgreSQL + Qdrant.
Relevant posts from the inference series: vLLM on Kubernetes — the multi-GPU HGX node scenario we develop here. LLM operators on K8s — the vLLM Production Stack and OME that the gateway can direct.
Observability: Evals, MCP observability, eBPF + drift.

vLLM on Kubernetes: the LLM inference piece that actually scales

Mon, 18 May 2026 13:00:00 +0200

TL;DR

vLLM is the inference engine that turns a general-purpose GPU into a productive LLM server. Its value is not in running a model, since any transformers.pipeline does that in three lines of Python, but in squeezing the GPU down to the last gigabyte and the last cycle: PagedAttention for the KV cache, continuous batching to mix requests, its own scheduler to share GPU time between sessions. Kubernetes is its natural habitat because vLLM behaves like a modern UNIX process, with a health endpoint, Prometheus metrics, orderly draining and declarable resources, and K8s already knows how to manage those. But there are traps: the standard HPA does not scale vLLM well, the model takes minutes to load, and naive rolling updates cut sessions halfway through decoding. This article takes the engine apart and then fits it, with real manifests, into a cluster that can actually serve it.

This article is the natural continuation of KV cache: the working memory that holds up LLM inference. There we explained why every token consumes VRAM. Here we look at what you do with that VRAM when you want to offer it as a service.

The analogy: a multiprocess kernel for your GPU

Imagine you have a single processor and need to serve a hundred concurrent processes without any of them blocking the rest. Nobody in their right mind would write a while-true loop dispatching processes one by one: they would install an operating system. The kernel takes care of scheduling, memory paging, isolation, priorities and cleanup on exit. The “process” becomes a convenient abstraction and the kernel does the dirty work.

vLLM is, for your GPU, what the kernel is for your CPU. As far as the GPU is concerned, a conversation with an LLM is a process that lives across many decoding steps, occupies a slice of VRAM (its KV cache) and demands compute time every time a token has to be generated. You have a hundred of those processes at once. You need to:

Share GPU time between them without pausing any of them entirely (it would be disastrous if one long conversation monopolised the GPU).
Manage memory with paging because, just as in RAM, contiguous allocation is inefficient.
Queue new requests when the GPU is saturated and serve them in a reasonable order.
Reclaim resources when a session ends.

PagedAttention is the virtual memory of the KV cache. Continuous batching is the time-slicing scheduler that shares the GPU token by token. The OpenAI-compatible server is the uniform syscall interface. Calling it a “kernel” for the GPU is marketing, but it is marketing that captures the idea well.

What vLLM does inside

Continuous batching: stop waiting for the slowest

The naive inference engine does static batching: it groups N requests, processes them until all of them are finished, returns and starts another round. The problem is obvious: if one request asks for 8 tokens and another asks for 800, the other seven wait for the slow one. GPU utilisation collapses.

Continuous batching (Yu et al., 2022, popularised by vLLM) changes the model. At every decode step, which produces one token for each active session, the engine composes the batch out of the active tokens of ALL sessions alive at that instant. When a session finishes its generation it releases its slot immediately and another request from the queue takes it. The batch never waits for the slowest session because nobody is blocked: everyone advances at one token per step.

The original paper measured 5–23× more throughput than the equivalent static batching. The exact number depends on how variable the response lengths are, but the order of magnitude holds up in practice.

The consequence for the operator is counterintuitive: a single vLLM replica performs like three naive replicas. There is no point adding pods without justifying it with real metrics.

PagedAttention: virtual memory for the KV cache

We already flagged it in the KV cache article: the naive engine reserves one contiguous block per session, sized for the worst case (max_context_len), and wastes 60–80% of the VRAM because real sessions come nowhere near their ceiling.

PagedAttention borrows the solution operating systems have been using for half a century: split the VRAM into small blocks (16 tokens in the default implementation) and keep a logical → physical page table per session. A session with 273 tokens of context occupies 18 blocks (not necessarily contiguous), and grows block by block as it generates. The paper measured <4% waste, an order of magnitude better than contiguous allocation, and that translates into 2–4× more aggregate throughput on the same hardware, because more sessions fit at once.

There is a cost: every attention operation has to be indirected through the page table. But vLLM’s CUDA kernels are written so that this indirection is cheap, and the net result is massively positive.

Prefill vs decode: two phases with opposite profiles

An LLM request has two phases with radically different GPU profiles:

Prefill: processes the entire prompt in one go. It is compute-bound: it uses the tensor cores intensively, the GPU sits above 90%, and it lasts between hundreds of ms and a few seconds depending on prompt size.
Decode: generates token by token. It is memory-bound: the compute is modest but the whole KV cache has to be read for every token, and it lasts from tens of ms per token up to minutes for long responses.

A naive server treats each request as a unit and serves the two phases in series. vLLM decouples them: it mixes requests in prefill with requests in decode in the same step (a technique called chunked prefill when it also splits long prefills into chunks). The result: the GPU is always busy doing something, the tensor cores with prefills, the HBM bandwidth with decodes, instead of oscillating between phases.

Operational implication: the “% GPU utilisation” metric from nvidia-smi is misleading. A GPU at 100% doing prefills may have its HBM bandwidth idle. A GPU at 40% doing decodes may have its HBM saturated. For LLM serving, the useful metric is effective HBM bandwidth, not the compute percentage.

Tensor parallel: when the model does not fit on one GPU

Llama 3 70B in BF16 is ~140 GB. There is no single GPU on the market that can hold it. The solution is tensor parallel: split each model layer by columns and run the partitions on N GPUs in parallel, synchronising with an all-reduce after each layer.

For N=5 GPUs and a 70B model, each GPU sees roughly 28 GB of weights. That sounds fine until you remember that the all-reduce on every layer means reading and writing large tensors between GPUs. If the GPUs share NVLink/NVSwitch (300–900 GB/s), the all-reduce is cheap. If they only share PCIe (~32 GB/s gen4 x16), the all-reduce eats half the time and throughput collapses.

The implication for K8s, which comes next: the scheduler has to guarantee that the N GPUs are physically close. That translates into NodeAffinity to the right product (NVIDIA-H100-80GB-HBM3), a single pod with nvidia.com/gpu: N (not N pods sharing) and, if multi-node is unavoidable, InfiniBand with NCCL as transport.

The OpenAI-compatible server

On top of all of the above, vLLM exposes an HTTP server with endpoints identical to OpenAI’s: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models. It supports Server-Sent Events streaming. It supports tool calling. It supports logprobs.

The value of this is enormous and underrated: any client using the OpenAI SDK works without changes. Your application points at https://vllm.your-cluster.local/v1 instead of https://api.openai.com/v1, and everything else, the LangChain, LlamaIndex, OpenAI Python and OpenAI JS SDKs, just works. It is the main reason vLLM has gained traction over technically comparable alternatives: it is the boring option that works.

Why Kubernetes is the natural habitat

vLLM is a well-behaved process: it starts, exposes metrics, serves a health endpoint, takes SIGTERM with dignity and declares the resources it needs. Kubernetes has spent ten years perfecting the management of processes like that. The only thing K8s took a while to absorb properly was the GPU, and that is now solved.

The GPU as a primitive resource

The plumbing goes like this:

The node has the NVIDIA driver installed (or the GPU Operator installs it).
A DaemonSet, nvidia-device-plugin, registers the physical GPUs as nvidia.com/gpu resources with kubelet.
The Kubernetes scheduler sees those resources the way it sees CPU and memory, puts them in its accounting and assigns them to Pods that ask for them.
The nvidia-container-toolkit makes sure containerd injects the right devices into the container at startup.

For the pod, asking for a GPU looks like this:

resources:
 requests:
 nvidia.com/gpu: 1
 limits:
 nvidia.com/gpu: 1

Without MIG, MPS or time-slicing configured, a GPU is not shared between pods: you ask for it whole or you do not ask for it. For vLLM, which wants the entire GPU to itself, that is exactly what you want.

The vLLM Pod lifecycle

Differences from a typical webapp Pod:

Long startup. Loading 16 GB of weights into VRAM over the network takes 30 seconds in the best case and 5 minutes in the worst. A readinessProbe with initialDelaySeconds: 30 and failureThreshold: 3 kills the pod before it starts. Solution: a startupProbe with a high threshold before the livenessProbe starts evaluating.
Warm-up matters. The first prefill compiles CUDA kernels specific to the input shape. The first 2–3 requests are noticeably slower. If latency matters from second one, it is worth firing a warm-up POST after ready.
Draining is not instantaneous. SIGTERM must not kill in-flight sessions. vLLM, configured with --disable-graceful-shutdown false (the default), finishes active requests before closing. That can take 30–180 seconds. terminationGracePeriodSeconds has to accommodate it.
Hostile rollouts. A naive rolling update (maxUnavailable: 1) can leave you with no replicas serving if the new one is slow to load. Set maxSurge: 1, maxUnavailable: 0 so the new pod is Ready before the old one is drained.

Anatomy of a serious deployment

First of all: GPU Operator

Without the GPU Operator (or an equivalent manual installation), a Pod with nvidia.com/gpu: 1 stays Pending forever. What the operator installs as DaemonSets on every GPU node:

nvidia-driver-daemonset — the kernel-mode driver (if you do not have it installed at host level).
nvidia-device-plugin-daemonset — registers the GPUs as a kubelet resource.
nvidia-container-toolkit-daemonset — the containerd integration.
nvidia-dcgm-exporter — Prometheus metrics for the GPU (utilisation, temperature, ECC errors, memory).
gpu-feature-discovery — node labels: nvidia.com/gpu.product, nvidia.com/gpu.memory, and so on, essential for NodeAffinity.

The recommended installation is the official Helm chart. The delicate part is aligning the driver with the host kernel version: if the nodes run kernel 6.x, the operator needs a compatible driver branch.

A complete, annotated vLLM Deployment

The following deploys Llama 3 8B with an FP8-quantised KV cache, up to 32K of context, on an RTX 4090. It is the reference manifest; the comments explain the non-obvious decisions.

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 replicas: 1
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxSurge: 1
 maxUnavailable: 0 # never run out of replicas during the rollout
 selector:
 matchLabels:
 app: vllm-llama3-8b
 template:
 metadata:
 labels:
 app: vllm-llama3-8b
 annotations:
 prometheus.io/scrape: "true"
 prometheus.io/port: "8000"
 prometheus.io/path: "/metrics"
 spec:
 # Only nodes with the GPU we expect
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-GeForce-RTX-4090
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 # Pre-download the weights if they are not on the shared PVC
 initContainers:
 - name: model-download
 image: ghcr.io/huggingface/huggingface-cli:latest
 command: ["sh", "-c"]
 args:
 - |
 if [ ! -f /models/llama-3-8b/config.json ]; then
 huggingface-cli download meta-llama/Meta-Llama-3-8B-Instruct \
 --local-dir /models/llama-3-8b --local-dir-use-symlinks False
 fi
 env:
 - name: HF_TOKEN
 valueFrom:
 secretKeyRef:
 name: huggingface
 key: token
 volumeMounts:
 - name: models
 mountPath: /models
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-8b
 - --served-model-name=llama-3-8b
 - --tensor-parallel-size=1
 - --max-model-len=32768
 - --kv-cache-dtype=fp8
 - --enable-chunked-prefill
 - --enable-prefix-caching
 - --gpu-memory-utilization=0.92
 - --port=8000
 ports:
 - name: http
 containerPort: 8000
 - name: metrics
 containerPort: 8000 # same port as http; /metrics
 resources:
 requests:
 cpu: "4"
 memory: 8Gi
 nvidia.com/gpu: 1
 limits:
 cpu: "8"
 memory: 16Gi
 nvidia.com/gpu: 1
 startupProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 10
 failureThreshold: 60 # 10 min of grace to load the model
 readinessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 5
 livenessProbe:
 httpGet:
 path: /health
 port: 8000
 periodSeconds: 20
 failureThreshold: 3
 volumeMounts:
 - name: models
 mountPath: /models
 readOnly: true # no process should write here at runtime
 - name: shm
 mountPath: /dev/shm  # vLLM uses shared memory for IPC between workers
 volumes:
 - name: models
 persistentVolumeClaim:
 claimName: model-cache
 - name: shm
 emptyDir:
 medium: Memory
 sizeLimit: 4Gi
 terminationGracePeriodSeconds: 120 # accommodates draining of active sessions
---
apiVersion: v1
kind: Service
metadata:
 name: vllm-llama3-8b
 namespace: inference
spec:
 selector:
 app: vllm-llama3-8b
 ports:
 - name: http
 port: 80
 targetPort: 8000

Five things that are not visible on a first read:

/dev/shm in memory, 4 GB. vLLM launches worker processes (one per GPU in tensor parallel, plus the driver) that communicate over shared memory. Docker’s default (64 MB) blows up as soon as the model is mid-sized. Without this, the pod starts but fails the moment it serves the first complex request.
--enable-prefix-caching. If the prompts in your workload share structure (a common system prompt, few-shot examples), vLLM reuses the KV cache of the shared part. A free 30–60% gain in TTFT.
--gpu-memory-utilization=0.92. vLLM reserves the given percentage of the VRAM for itself. The remaining 8% leaves room for activations, CUDA kernels and the overhead that does not get counted. Lowering it buys safety; raising it above 0.95 invites OOM.
A ReadOnlyMany PVC is ideal. The model does not change at runtime. Several pods can mount the same PVC without contention.
No livenessProbe shorter than the terminationGracePeriodSeconds. If a drain takes 90s and liveness kills at 60s, rollouts lose sessions.

Multi-pod tensor parallel: LeaderWorkerSet

When the model needs more GPUs than a single node has, the pattern is a group of coordinated pods, one per GPU, that behave as a single replica. For years this was modelled with a StatefulSet plus init scripts; since Kubernetes 1.32, the idiomatic primitive is LeaderWorkerSet (LWS):

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
 name: vllm-llama3-70b
 namespace: inference
spec:
 replicas: 1
 leaderWorkerTemplate:
 size: 5 # 1 leader + 4 workers = 5 pods, 5 GPUs
 restartPolicy: RecreateGroupOnPodRestart
 leaderTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-leader
 image: vllm/vllm-openai:v0.6.3
 args:
 - --model=/models/llama-3-70b
 - --tensor-parallel-size=5
 - --distributed-executor-backend=ray
 # ...
 workerTemplate:
 spec:
 nodeSelector:
 nvidia.com/gpu.product: NVIDIA-H100-80GB-HBM3
 containers:
 - name: vllm-worker
 image: vllm/vllm-openai:v0.6.3
 # the workers join the leader's Ray cluster

LWS guarantees startup order (workers first, leader afterwards) and an atomic lifecycle (if a worker goes down, the whole group restarts, not a single pod). Without it, the coordination is manually fragile.

A simpler alternative, if all the tensor parallel GPUs fit on a single node (the case with HGX H100 boxes with 8 GPUs and an internal NVSwitch): one single Pod with nvidia.com/gpu: 5, --tensor-parallel-size=5, and vLLM handles everything internally. No Ray, no LWS, far simpler. It is the recommended path whenever it is available.

Autoscaling: the standard HPA is no use

An HPA on CPU% is useless for vLLM. The GPU does the work; the pod’s CPU sits at 5–10% even under maximum load. Nor is the GPU utilisation percentage from dcgm-exporter any use: a pod at 100% GPU% with gpu_cache_usage_perc=15% is serving one long session without being saturated, while a pod at 60% GPU% with gpu_cache_usage_perc=95% is on the edge of evicting sessions.

The right metrics are exported by vLLM itself at /metrics (Prometheus format):

Metric	What it says	When to scale
`vllm:num_requests_waiting`	Requests queued and not yet in the batch.	If it stays above 5–10.
`vllm:num_requests_running`	Active requests in the batch.	For capacity planning, not for scaling.
`vllm:gpu_cache_usage_perc`	% of the KV cache occupied.	If sustained above 80%, there is a risk of preemption.
`vllm:time_to_first_token_seconds`	Prefill latency (histogram).	If p95 exceeds your SLA.
`vllm:e2e_request_latency_seconds`	Total latency per request.	An output metric.

For the HPA to consume them there are two routes: Prometheus Adapter (exposes custom metrics to the K8s API) or KEDA (scales on Prometheus queries directly, far more convenient). With KEDA:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama3-8b
 minReplicaCount: 1
 maxReplicaCount: 8
 pollingInterval: 10
 cooldownPeriod: 120 # 2 min before scale-down (long sessions)
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.monitoring:9090
 threshold: '5'
 query: |
 sum(vllm:num_requests_waiting{app="vllm-llama3-8b"})

The long cooldownPeriod matters: if you drop replicas while there are sessions decoding, you kill them. Better to leave 2 minutes of slack.

Observability: the four metrics that matter

Of everything /metrics exports, a minimal dashboard needs these four:

TTFT p50/p95 (time to first token) — what the user perceives when they hit send.
TPOT p50/p95 (time per output token) — the “speed” of the streaming.
Aggregate throughput (tokens generated per second across the cluster) — for capacity planning.
Queue depth (vllm:num_requests_waiting) — the leading indicator: if it grows, everything is about to degrade.

To that, add HBM utilisation and free memory per GPU (from dcgm-exporter) to spot bandwidth saturation and fragmentation problems. A decent Grafana dashboard with those 6 charts anticipates 90% of incidents.

Two concrete scenarios

We reuse the same hardware as the previous article for continuity. The same cache arithmetic, now with the engine on top.

Scenario A — 1×RTX 4090 (workstation or small K8s node)

Topology: 1 Pod, --tensor-parallel-size=1, 1 GPU, 1 node.
Model: up to 8B BF16 (Llama 3 8B, Qwen3 8B, Mistral 7B) or up to 14B in FP8/AWQ.
PVC: the node’s local SSD. The 4090 reads 1 TB/s from HBM; an NVMe SSD at 5 GB/s takes 5 seconds to feed 25 GB of weights into VRAM, negligible next to initialisation.
HPA: irrelevant inside the 4090 (always 1 vLLM replica per GPU), but useful across nodes: 3 replicas on 3 nodes with a 4090 each, and the K8s Service spreads round-robin.
Useful concurrency: 4–8 simultaneous sessions with 8K of context, 1–2 with 32K.
Natural use case: PoC, small teams, departmental environments, edge.

The manifest above is sized for this scenario. Changing only the model and the args, the same Deployment serves Qwen, Mistral or whatever is next.

Scenario B — 5×H100 SXM (cluster with NVLink/NVSwitch)

Topology: 1 Pod with nvidia.com/gpu: 5 on an HGX node, --tensor-parallel-size=5. If the platform does not allow grouping 5 GPUs into a single Pod, a LeaderWorkerSet with 5 pods coordinated by Ray.
Model: up to 70B BF16 (Llama 3 70B) or up to 200B+ in FP8 with cache quantisation.
PVC: NVMe attached directly to the node, or fast network storage (Ceph with a 25/100 GbE network, Lustre, GPFS). Loading 140 GB of weights over a slow network takes 5 minutes per start.
HPA: irrelevant inside the 5-GPU cluster (the 5 are an indivisible unit), but useful for adding whole HGX nodes when the load passes a given threshold. This combines with Cluster Autoscaler if the underlying infrastructure allows it.
Useful concurrency: 32–128 simultaneous sessions with mid-sized contexts, 4–16 with huge contexts.
Natural use case: internal corporate service, public exposure with an SLA, multi-tenant.

A and B, side by side

Aspect	A (1×4090)	B (5×H100 SXM)
Pod topology	1 pod, 1 GPU	1 pod with 5 GPUs (or an LWS of 5)
Maximum BF16 model	8 B	70 B
TTFT @ 8K context, idle	~250 ms	~80 ms
TPOT, idle	~30 ms/tok	~15 ms/tok
Throughput @ concurrency 16	~50 tok/s/session	~200 tok/s/session
Session drain	30–60 s	60–180 s
Useful autoscaling	Replicas on peer nodes	Whole nodes via Cluster Autoscaler
Reasonable multi-tenancy	Limited: 4–8 sessions	Comfortable: 32–128 sessions
Indicative cost (hardware)	~2 K €	~250 K € (≈ 125×)

The asymmetry is the same one as in the previous article: 125× more expensive, only ~4× more throughput per session and ~10× more concurrency. What the cluster buys is not proportional; it buys access to models an order of magnitude larger and latencies low enough for interactive use at scale. If your workload is batch or asynchronous agents where latency is not critical, several 4090s come surprisingly close.

vLLM against TensorRT-LLM and SGLang

Honestly, all three are good engines. The choice depends on practical criteria, not technical ones. A decision map, not a benchmark:

Criterion	vLLM	TensorRT-LLM	SGLang
Supported hardware	NVIDIA, AMD ROCm, Intel Gaudi	NVIDIA only	NVIDIA, AMD ROCm
Pure latency (TTFT)	Good	Best: kernels compiled for the exact hardware	Good
Aggregate throughput	Excellent	Excellent	Excellent (RadixAttention)
Deployment	Trivial: Docker image + args	Complex: build an engine per model and per GPU	Moderate
OpenAI-compatible API	Native, complete	Yes, through Triton Inference Server	Yes
Support for new models	Days after release	Weeks (recompile the engine)	Days
Quantization	AWQ, GPTQ, FP8 cache	INT4/INT8/FP8, very mature	AWQ, FP8
Multi-modal	Yes (Llava, Pixtral, Qwen-VL)	Yes	Excellent, a priority
Function calling / tool use	Good	Limited	First class
Community / release cadence	Very active, weekly	Active, NVIDIA-driven	Very active, academic
Licence	Apache 2.0	Apache 2.0	Apache 2.0

When to choose each one:

vLLM: the “boring choice” that works. The path of least friction to production. If your team has no dedicated inference serving specialist, this one. It supports varied hardware, up-to-date models, a stable API and a huge community.
TensorRT-LLM: when per-request latency is the only metric that matters and your model is stable (trained in-house, not swapped every fortnight). The price of the performance is that every model plus every GPU plus every TRT version requires an engine rebuild, and that blocks fast iteration.
SGLang: for agent-dominated workloads (heavy tool calling) or complex multi-modal. Its RadixAttention, structural caching of prompts with shared prefixes, shines in ReAct-style patterns where the same system prompt is repeated thousands of times.

For most teams starting out with on-prem LLM serving, vLLM is the right answer until you have production data pushing you elsewhere.

Common operational traps

A list of gotchas that come up again and again:

The model is downloaded on every rolling update

Symptom: every deploy takes 5+ minutes to become available. Cause: there is no shared PVC. Every new pod downloads the model from Hugging Face from scratch. Fix: a ReadOnlyMany PVC on fast storage, or a local mirror of the registry (a Pod with huggingface-cli serving a directory over HTTP). In CI/CD, hydrating the PVC before the rollout is 1 line of bash.

Short readiness timeouts that kill pods while they load

Symptom: new pods go into CrashLoopBackOff during the first model load. Cause: a readinessProbe with too low a timeout fires before vLLM finishes loading; the livenessProbe finishes the job. Fix: a startupProbe with failureThreshold: 60 or more (10 minutes of grace) before liveness starts evaluating.

An unquantised KV cache, then OOM

Symptom: the pod starts fine, serves for five minutes, then gets OOMKilled when session number five arrives with a long context. Cause: a BF16 KV cache (the default) consumes twice what FP8 does. Fix: --kv-cache-dtype=fp8. Negligible quality loss in the vast majority of cases, double the capacity.

Confusing replicas with concurrency

Symptom: the HPA scales to 8 replicas under light real load and the cloud bill goes up. Latency does not improve. Cause: someone configured targetAverageUtilization: 50% on CPU, thinking that is “load”. The reality: a single vLLM replica serves dozens of simultaneous sessions. Fix: an HPA on vllm:num_requests_waiting. If the queue is empty, one replica is enough even if the GPU is at 90%.

Tensor parallel on GPUs without NVLink

Symptom: throughput 3× worse than expected, GPUs at 30%, a lot of PCIe traffic. Cause: tensor_parallel=4 on 4 GPUs connected only by PCIe; the all-reduce saturates the bus on every layer. Fix: either the GPUs share NVLink/NVSwitch (SXM/HGX models), or you use pipeline parallel (worse latency but less all-reduce), or you reduce TP and accept that the whole model does not fit.

Sessions cut off during a rolling update

Symptom: users see truncated responses during the deploy. Cause: terminationGracePeriodSeconds: 30 (the default) is not enough to drain long generations. Fix: terminationGracePeriodSeconds: 120–180. Combined with maxUnavailable: 0, rollouts are invisible to active users.

What we have not covered (upcoming articles)

vLLM with hot LoRA adapters: serving a base model plus N tenant-specific adapters without reloading weights.
Disaggregated serving: separating prefill and decode into specialised pods, each optimised for its GPU profile.
Quantization deep-dive: AWQ vs GPTQ vs dynamic FP8 vs FP4, the real trade-offs, and when to use each.
Gateway API + AI Inference Extensions: the sigwg proposal to make LLMs first-class citizens in K8s (routing by model, sticky sessions per conversation, multi-tenant fairness).
Multi-modal serving: the same runtime, a different kind of request — images, audio, embeddings.

References

Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — the original vLLM paper.
Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) — the paper that popularised continuous batching.
Official vLLM documentation — operational and well maintained.
NVIDIA GPU Operator — installation and troubleshooting of the GPU layer on Kubernetes.
LeaderWorkerSet — the primitive for coordinated workloads such as multi-pod tensor parallel.
KEDA — event-driven autoscaling, ideal for scaling on queue metrics.
TensorRT-LLM and SGLang — the two most serious comparables.
LMSYS Chatbot Arena — periodic benchmarks comparing the three engines.
Previous article on this blog: KV cache: the working memory that holds up LLM inference.

Parameter	Symbol	Example
Requests per day (average)	\(D\)	5,000,000
Input tokens per request (average)	\(L_{\text{in}}\)	512
Output tokens per request (average)	\(L_{\text{out}}\)	256
Peak factor (peak hour vs average ratio)	\(k_{\text{peak}}\)	1.8
Hours per day with significant traffic	\(H\)	16

Gpu on lo0 — Tech Blog

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

TL;DR

The analogy: the restaurant that switches off the kitchen

How Knative scales

KPA versus HPA

Why an LLM cold start is not starting a pod

The techniques that shorten the wait

Knative and KServe: the two modes

The economics

Operational traps and honest scepticism

For an inference factory

See also

Sources

Model chain of trust (4/4): who serves the model and which machine you trust

TL;DR

The analogy: the biosafety laboratory

Part A — Workload identity

Six services talking to each other and none of them knows who the other is

SPIFFE: the standard

SPIRE: how that document is issued

Federation between trust domains

From identity to effective authorisation

AI agents and MCP: the missing piece

Part B — Isolation of the execution environment

Confidential Containers: the TEE

The hardware: CPU and GPU

The overhead numbers, with source and with judgement

When each rung is worth it

Operational traps

Closing the series: the complete chain

Mapping the series to ENS, ISO/IEC 42001 and the EU AI Act

For an inference factory

See also

Sources

Volcano and Kueue: gang scheduling, queues and GPU quotas for distributed workloads on Kubernetes

TL;DR

The analogy

The problem neither solves by default: the kube-scheduler

Volcano: the batch scheduler

What it is and what it replaces

Gang scheduling via PodGroup

Queue: queues with quotas and priority

Scheduler plugins: DRF, binpack, topology-aware

Network and NUMA topology (v1.11+)

GPU virtualization in v1.11+

What it replaces or adds to the default scheduler

Kueue: the queue and quota manager

What it is and what it does NOT do

The four core objects

Fair sharing and preemption

Gang semantics in Kueue: all-or-nothing with ready Pods

Topology-Aware Scheduling (TAS)

Framework integrations

The key distinction: Volcano places, Kueue admits

How they coexist: the production pattern

The third way: the sig-scheduler coscheduling plugin

Complete YAML examples

Volcano: Queue + PodGroup + VolcanoJob

Kueue: ResourceFlavor + ClusterQueue + LocalQueue + annotated Job

Kueue + Volcano together: PyTorchJob with Kueue quota and Volcano gang placement

For LLM workloads: training, fine-tuning and batch inference

Distributed multi-GPU training and fine-tuning

Batch inference and evaluations (evals)

Multi-tenant GPU quota and chargeback

Diagram: the flow of a training job with Kueue + Volcano

Full comparison table

Operational pitfalls and honest scepticism

1. Deadlock from a badly configured gang

2. Quota versus real capacity: the silent drift

3. GPU resource naming: MIG, time-slicing and ResourceFlavor

4. Preemption in production: the evicted workload loses progress

5. Volcano as the sole scheduler versus coexistence with the kube-scheduler

6. Real operational complexity in 2026

See also

Sources

From the SLO to the number of GPUs: how to size and justify the investment in inference hardware

TL;DR

1. The sizing chain: from demand to the number of GPUs

1.1 Step 1 — Characterise the demand